Comply v2 - GA Deployment Plan

Ha, no worries at all. Okay so — April 7th is six days away, which honestly feels very real now, and I wanted to get the four of us together to go through the GA deployment plan for Comply v2 and make sure we're all aligned on the rollout sequence.

Perfect. So the plan right now is to do the actual infrastructure changes on the night of April 6th — like a late evening deploy, probably around 10 PM Eastern — and then flip the feature flag to GA on the morning of the 7th. Ravi, does that sequence still work for you on the infra side?

Yeah, it does. I've already got the Terraform configs updated for the new reporting service nodes, and I ran a full dry-run in staging yesterday and it looked clean. I'm feeling good about the 10 PM window. We'll have, um, about a three hour buffer before any early morning traffic picks up.

Mostly smooth, honestly. There was one thing — the new SOC 2 report generation was taking like 8 seconds on first call, which is a little longer than I'd expect, but I think that's just cold start on the Lambda, not a real issue. I'll add a warm-up ping to the pre-deploy checklist.

Yeah so, I finished the review on the HIPAA and ISO 27001 report generators this morning and honestly the code is in really good shape. Priya did a great job on those. My only thing — and this is more of a minor nit — is the PCI DSS report has some hardcoded version strings that I'd like to see pulled into config, but it's not a blocker by any means.

Definitely, yeah, I already filed it. It can live as a v2.0.1 cleanup item. I wouldn't block launch over it.

Okay so, good news overall. We finished the full regression suite yesterday and everything passed. The on-demand report generation — all four frameworks, SOC 2, PCI DSS, HIPAA, ISO 27001 — tested across the three tenant sizes we defined, small, medium, enterprise, and all of them are generating correctly. The PDF rendering looks great, the data mapping to controls is accurate, we verified that manually.

Ha, right? So the one area I'd flag — not necessarily a blocker but something I want us to have eyes on post-launch — is the concurrent report generation. When we hammered it with 50 simultaneous requests in the load test, we did see some queue latency creep up. Like, reports were still completing, but the p99 latency was around 22 seconds, which is above our SLO target of 15.

Okay, 22 seconds at 50 concurrent — do we think that's a realistic scenario for day one? I mean, we're not flipping this on for all customers simultaneously, right?

No, exactly. We're doing a phased flag rollout — tier one customers first, which is maybe, what, a few hundred orgs, and then expanding over the following week. So I think the risk of hitting 50 concurrent on day one is pretty low.

Right, and honestly I agree it's low risk for launch. I just wanted it on the radar so we're not surprised if we see it in the MetricFlow dashboards. I'll set up a specific alert for queue depth on the reporting service so we can catch it early if it starts trending up.

Absolutely, we can knock that out tomorrow morning. I'll ping you Nina.

Perfect. And actually, since we're on the topic of observability — and this is maybe a bit of a sore subject but I think it's worth saying out loud — after the Detect outage in March, I want to make sure we're not shipping Comply v2 with any single points of failure in the pipeline. Ravi, can you talk through the redundancy story?

Yeah, no, totally fair to bring up, and honestly the Detect situation kind of informed a lot of how we architected the reporting pipeline. So the event ingestion layer has redundant processing nodes — we're running three — and we've implemented the circuit breaker pattern on the report request handler so if one downstream service starts degrading, it fails gracefully instead of cascading. It's a very different story from what happened with Detect.

Same. Okay, good. Let's talk about rollback — Ravi, what's the rollback plan if something goes sideways on the night of the 6th or morning of the 7th?

So rollback is clean. The feature flag means we can instantly revert to v1 behavior without redeploying anything — just flip the flag back. The infrastructure changes are additive, not replacing anything, so the v1 reporting service stays running in parallel until we're confident. We can decommission that after, uh, I was thinking two weeks post-GA.

Yeah, two weeks is good. Most of the edge cases we'd be worried about would show up in the first week honestly. I'm comfortable with that.

Awesome. Okay, I think we're in really solid shape here. I'll update the runbook today with the alert config action item and the warm-up ping note, and I'll send it around for a final review by EOD Thursday. Does anyone have anything else they want to flag before we wrap up?