Detect Reliability - 30-Day Review

Perfect. So, uh, today's agenda — this is officially the 30-day post-incident review for the Detect outage that happened March 10th through the 18th. We want to cover what happened, where we landed with the fixes, and then Diana, I know you have some customer impact stuff you want to walk through.

Yeah, exactly. And I want to make sure we have time at the end to talk about messaging, because I'm still getting questions from a few accounts and I need to know what I can and can't say.

Sure, yeah. So the short version — the event processing pipeline had a single point of failure in the event ingestion layer. What happened was we had a cascading failure that started around 11 PM on the 10th and by about 5 AM on the 11th we had essentially zero threat monitoring visibility for customers on the Detect module. Six hours of complete blindness.

And just to be clear for this review, this wasn't a security breach, right? Like the data itself was fine, it was just the monitoring layer that went down.

Correct, yes. No data loss, no unauthorized access. The pipeline that processes incoming threat events just stopped processing. So events were queuing but nothing was being analyzed or alerted on.

Right, and that distinction actually matters a lot for how some of our customers interpreted it initially. A few of them — I'll get into this later — a few of them thought their environments had been compromised because they stopped seeing alerts.

We can talk about that. Let me just finish the technical recap and then Diana you can take us through the customer side. So, um, root cause confirmed as the single ingestion node — no redundancy, no failover. When that node got overloaded due to a spike in event volume from one of our larger tenants, it effectively took down the whole pipeline.

So we shipped two things. First was redundant processing nodes — we now have three ingestion nodes running in parallel with automatic failover. If one goes down, the other two absorb the load. Second thing was implementing the circuit breaker pattern, which basically means if we detect that a downstream component is failing, the system stops trying to push work to it and routes around it instead of just hammering it until everything collapses.

The redundant nodes were deployed March 24th. Circuit breaker pattern was fully validated and in production as of, uh, March 29th I believe.

And we've had no recurrence since then, right? Like the 30-day window has been clean?

Yeah, clean. We had one minor hiccup on March 31st where latency spiked briefly but the circuit breaker caught it and we never lost visibility. Customers didn't see anything.

Yeah, I'd appreciate that going forward. Even if it's just a Slack message saying hey, we had a blip, circuit breaker fired, all good. Just so I'm not getting caught flat-footed if a customer asks.

Noted, that's a fair ask. We'll add that to the on-call runbook — any circuit breaker activation gets a heads up to Customer Success.

Thank you. Okay so — customer impact. Let me pull up my notes here. So we had... forty-seven accounts on Detect that were affected. Of those, eleven filed formal support tickets during the outage window. Three accounts requested executive briefings afterward.

One was Meridian Financial — they're a large account, been with us about two years. Their VP of IT Security was pretty unhappy. They had an internal compliance audit scheduled for March 15th and the fact that they couldn't show continuous monitoring coverage for those six hours was a problem for them. Like a real problem, not just a perception problem.

We provided them with a formal incident report and a letter from our CTO explaining the nature of the outage and the remediation steps. Their auditors apparently accepted it as a documented exception. So it's closed but it left a bad taste.

They're up in August. I'm watching it closely. They haven't said anything explicitly threatening but I'm not comfortable with where the relationship is right now. We need a good next few months.

Is there anything engineering can do to help with that? Like, would it help if we could offer them some kind of enhanced monitoring visibility or a preview of upcoming Detect improvements?

Actually, yeah, that might be worth exploring. We're planning to roll out the expanded event telemetry dashboard in Q2 anyway — could we get Meridian into a beta or an early access program?

I'd have to check with product on timeline but I don't see why not. Chris, thoughts?

I think it's doable. The dashboard work is about 60% done. We could have something meaningful to show by mid-May. I'd rather not promise anything before I talk to Priya about the scope but conceptually I'm supportive.

Okay, let's flag that as a follow-up action. Who owns that conversation with Priya?

I'll take it. I'll ping her today and loop you in on the thread, Diana.

Great. So the third executive account — that was actually a smaller one, Treswick Healthcare. They were more worried about the regulatory angle, specifically HIPAA. They use Detect as part of their security monitoring for covered systems. Same situation — the gap in monitoring coverage was a documentation headache for them.

Did we pull the Comply team into that conversation at all? Because, um, actually — today's kind of a relevant day for this. Comply v2 just went GA this morning, right?

Ha, yeah it did. And actually that's kind of relevant because Comply v2 has the on-demand HIPAA reporting. If Treswick was running Comply they could pull an audit trail report that would show, you know, here's exactly what happened and here's the documented gap. That's actually more defensible than us writing a manual letter.

Are they on Comply currently?

I don't know off the top of my head. I'd have to check. But if they're not, this might be a moment to have that conversation with them.

Yeah, I'll look into their contract. That's actually a good angle — turning a service recovery situation into a broader platform conversation. I like that.

Okay, so to wrap up on the action items — Tyler owns the technical documentation for the 30-day review, Chris is going to talk to Priya about the Meridian early access thing, and Diana you're going to check on Treswick's Comply status. Anything else we're missing?

I want to make sure we have a finalized customer-facing summary of the incident and the resolution. Something I can point people to if they ask. Right now I'm working from my own notes and the internal doc, but I'd like something I can actually share.

Tyler, can you take a pass at drafting that? You've got the technical detail. Keep it non-technical enough for a customer to read but specific enough that it's credible.

Yeah, I can do that. I'll have a draft to you both by end of week. Diana, do you want to review before it goes anywhere?

Definitely. Send it to me first, I'll give you feedback on the tone and then we can finalize. I don't need it to go through legal unless Chris thinks it needs to.

Given that we already sent the formal letter to Meridian I think a general summary doc is probably fine without legal review, but maybe just a quick sanity check with Jared in Legal just to be safe. It's like a 15-minute thing.

Fair enough. I'll add that to the list. Alright, I think we've covered the main bases. Is there anything else before we wrap?

We're working on an improved alerting runbook for that. Should have it done by end of April. Alright, I'll send the meeting notes out this afternoon. Thanks everyone.