Support Case #3536 - Stratos Cloud LogVault Integration Timeout

Okay, sure. I appreciate that. So — look, I'll just get into it. We've been running Aegis Detect integrated with our LogVault environment, right, and starting around — I want to say April 18th, maybe 19th — we started seeing consistent timeout errors when Detect tries to push event data to our LogVault HTTP Event Collector endpoint. It's not intermittent, it's basically every... every batch that goes through.

Okay, so consistent timeouts hitting the HEC endpoint. And when you say every batch — are these on a scheduled interval or is this continuous streaming from Detect?

We have it set to push every five minutes. And it was working fine, totally fine, for months before this. Nothing changed on our end — we didn't update LogVault, we didn't change the HEC token, nothing. So whatever broke, it broke on your side.

-1Damien insists nothing changed on their end and blames Aegis for the break. · service reliability

I hear you, and I want to be careful not to make assumptions before we dig into the logs, but I do want to look at this thoroughly. Can I ask — are you seeing any error codes coming back, or is it just a straight connection timeout with no response from the HEC endpoint?

Okay so LogVault HEC is healthy, other sources pushing fine, only Aegis Detect is timing out. David, I'm wondering if this could be related to the event processing changes that were pushed in the pipeline update earlier this month.

Yeah, I was thinking the same thing. Damien, I want to ask — do you happen to know your Aegis Detect tenant version? It should be visible in the platform under Settings and then About.

Uh, hold on. Let me — yeah, give me a second, I'm pulling it up now. Okay it says... Detect pipeline version 4.1.2, event processor build April 8th.

Yes, that's correct. So Damien, I don't know if you were made aware — in mid-March we had a significant incident with the Aegis Detect event processing pipeline. There was a cascading failure that caused monitoring gaps, and we pushed a remediation update that included redundant processing nodes and a circuit breaker pattern in the ingestion layer.

Oh, I am very aware of that outage. Yeah, we were one of the customers affected. Six hours of no visibility — that was a really bad week for us. We had to explain to our own leadership why we had blind spots in our threat monitoring. That was not a fun conversation.

-1Damien describes the March six-hour outage as a bad week and an unpleasant conversation with leadership. · service reliability

Yeah, we — we really do understand that, and I'm sorry again for the disruption that caused. What I'm wondering now though is whether the circuit breaker implementation in that fix might be contributing to what you're seeing with LogVault. If the outbound connector to your HEC endpoint is getting tripped by the circuit breaker under certain load conditions, that could explain the timeout pattern.

I want to be clear that I'm not confirming that yet, we need to look at the logs before I say anything definitively. But it's a hypothesis worth investigating. David, can you pull the outbound connector logs for Stratos on the Detect side?

Already on it. Damien, while I'm doing that — can you tell me roughly what volume of events Detect is processing in your environment? Like ballpark events per minute during peak hours?

Uh, we're a pretty active environment. I'd say during peak — probably somewhere between eight and twelve thousand events per minute. We're a technology company, we've got a lot of infrastructure generating signal.

Okay, yeah that's — that's actually useful context. So I'm looking at the connector logs now and... okay, yeah. I'm seeing the circuit breaker state flipping to open on the LogVault HEC connector. It looks like it's tripping because the payload size on the five-minute batch is exceeding the default threshold that was set in the 4.1.2 build.

So the circuit breaker that was added to protect the pipeline — it has a payload threshold, and at your event volume, the five-minute batch is generating a payload that's hitting that limit. When the circuit breaker trips, it stops the outbound push to LogVault and the connection just drops, which is why you're seeing the timeout with no response code.

So it's stopping itself from sending data to our SIEM. That means we've had gaps in our LogVault data since the 18th? That's — that's four days of incomplete data. Do you understand what that means for us from a security monitoring standpoint?

-1Damien highlights four days of incomplete data and presses the security impact. · service reliability

I do, and I want to be transparent with you — yes, the events during the tripped periods would not have been forwarded to LogVault. They are retained within Aegis Detect itself, so the data isn't lost, but it hasn't been pushed to your HEC endpoint during those windows.

Okay so the data exists on your side. Can we backfill it? Can you push the missed events retroactively to our LogVault instance?

That's actually something we can do, yes. We have a manual replay function for the outbound connector. What I'd want to do first though is fix the underlying threshold issue so we're not in the same situation after the backfill.

The configuration change itself is pretty quick on our end, maybe fifteen minutes to apply and validate. There'd be a brief pause in forwarding during the switch — we're talking under a minute typically. And then we'd do the backfill right after to make sure you have everything.

Okay. Do the streaming mode. And I want a written summary of what happened here — the root cause, the timeline, what data was affected, and what was done to fix it. I need to take that to my leadership and frankly I'm going to need to share it with our security team so they can assess any gaps in coverage.

Absolutely, Damien. I'll personally make sure a detailed incident summary is drafted and sent to you — I'd say end of business tomorrow at the latest. And I want to flag that given this is the second significant disruption to your Detect environment this year, I'd like to schedule a call with you and our customer success team to review your overall configuration and make sure we're set up in a way that's more resilient for your environment going forward.

Yeah, that conversation needs to happen. I'll be honest with you — my team has been asking questions about whether Aegis Detect is the right platform for us. We're a fast-growing company, we can't be absorbing incidents like this. So that conversation with customer success, fine, but it better be substantive.

-2Damien says his team is questioning whether Aegis Detect is the right platform and that they cannot absorb incidents like this. · service reliability

Understood, and I appreciate you being direct with us about that. That feedback is important and I won't minimize it. Let's get the immediate fix applied today, get you the backfill, and then we'll make sure the follow-up conversation is meaningful. David, can you go ahead and initiate the streaming mode switch for Stratos now?

Yeah, doing it now. Damien, you might see a very brief pause in the LogVault feed — like I said, under a minute — and then it should resume and you'll see the stream mode active. I'll confirm with you on the call when it's validated.

Given the volume we're talking about, probably one to two hours for the full four-day backfill to complete. It runs as a background job. I'll send you a confirmation email when it kicks off and another when it completes.