Protect Performance - Scalability Concerns

Okay good, so — let's just dive in because I don't want this to run long. The reason I called this is, uh, we've been getting some pretty alarming signals from the Protect module over the last couple of weeks and I think we need to talk through what's actually happening before this becomes a customer-facing issue.

Sure, yeah. So we ran a simulated backup workload at roughly 2x our current P95 customer scale, and the results were... not great. Throughput started degrading at around the 60% load mark, and by the time we hit full simulated load we were seeing backup job queue times spike to like — I think the worst number was 47 minutes for a job that should complete in under five.

That's... I mean that's genuinely bad. Like that's not a tuning problem, that's a fundamental architecture concern.

Right, which is sort of what I've been trying to say. The scheduler — and look, I don't want to throw anyone under the bus here — but the job scheduler isn't built to handle concurrent dispatch at this scale. It was designed for, I don't know, a much smaller customer footprint than where we're heading.

Okay and this is exactly what I documented in QA-1847 back in, uh, late February. The concurrency ceiling on the scheduler queue was a known risk item. I flagged it as high severity.

Nina, I know you flagged it. It was on the backlog. We were dealing with — look, we had the Detect outage in March, that pulled a lot of engineering bandwidth, and some things got deprioritized. That's just the reality.

So I've been doing some digging since Tyler shared the numbers yesterday, and I think there are actually two separate issues tangled together here. One is the scheduler concurrency problem which Nina flagged, and the other is — and this one I actually didn't expect — the snapshot deduplication layer is doing a lot of synchronous I/O that it really shouldn't be. Like it's blocking on disk reads mid-job.

Honestly? It's not a quick fix. The scheduler wasn't designed to be horizontally scalable. To do this properly we'd probably need to refactor the dispatch layer, introduce some kind of distributed queue — I keep thinking something like a partitioned queue model would actually solve this — but we're talking weeks of work. Plural. Not two weeks, more like... six to eight minimum if we want to do it right and not create new problems.

Six to eight weeks. Okay. And we're heading into what's supposed to be a growth quarter for Protect. Sales has been pushing it pretty hard post the Comply v2 launch — they want to bundle them together.

Which is great for the business but if we scale the customer base and the infrastructure can't handle it, we're going to have outages. Real ones. Customer-impacting ones.

And that's the last thing we need right now. After March, after the Detect thing — honestly our credibility with some of these enterprise customers is already fragile. Another major incident would be... it would not be good.

Yeah, I think about that a lot. The Detect outage — the six hours of no visibility — that shook some people. I know we fixed it, the redundant nodes, the circuit breaker stuff, all of that. But trust takes a while to rebuild and piling a Protect failure on top of that is not something I want to explain to leadership.

There might be some things we can do at the config level. Like we could tighten the job concurrency limits per tenant — basically throttle how many concurrent backup jobs any single customer can run — which buys us some headroom on the queue. It's not a fix, it's a band-aid, but it could prevent the worst-case blowups.

I agree with Sofia. A graceful degradation we control is better than an uncontrolled failure. We implement the throttle as a temporary measure, we communicate it internally, and we stay off the critical path while we do the real fix.

Yeah, actually I think the dedup thing is more isolated. If we move the deduplication check to an async worker and just do a quick hash lookup first with a fallback... I mean I'd need to prototype it but I think that's a one to two week effort. Maybe less.

Okay, that's at least something. That's one concrete thing we can actually move on quickly.

Can I ask something that might be uncomfortable? Do we need to be having a conversation about whether sales should be slowing down Protect commitments right now? Like, should we be having a conversation with them about customer onboarding pace?

Ugh. I mean — yes, probably. That's going to be a fun conversation. Sales is riding high on the Comply v2 momentum and they're not going to love being told to pump the brakes on Protect.

I know, but better to have that conversation now than to have it after a customer's backups fail during a ransomware event or something. Like, that's the scenario we're trying to avoid.

You're right. Okay. I'll set up time with the sales lead this week. I'll frame it as a scaling preparation conversation rather than a stop-selling conversation. But the message will basically be: let's be thoughtful about onboarding pace for the next six to eight weeks.

That's probably the right framing. And in parallel, Tyler and I can start sketching out the scheduler architecture? Like even just getting the design doc started so we know what we're actually committing to?

Yeah, let's do that. I can pull together the load test traces and annotate them so you've got the full picture of where things are falling apart.

Good. Nina, can you put together a summary of the QA findings — the existing ticket plus anything new from the test results — so I have something to bring to leadership? I need to brief them this week and I want the data to speak for itself.

Yeah, I can have that to you by tomorrow morning. And I'll include the timeline of when things were flagged because I think that's... I think that context matters.

Understood. Okay — I think we have a rough path forward. It's not a great situation, I won't pretend it is, but at least we're not walking into it blind anymore. Let's regroup end of week and see where we are on the dedup fix and the design doc. Sound good?