Support Case #9856 - Silverline Brands Backup Window Exceeded

That's the short version, yeah. I mean, look, this has been going on for about two weeks now and it's — it's becoming a real problem for us, especially with our retail operations. We're a 24/7 environment basically, and when backups are running into business hours it's causing performance degradation on our production systems.

-1Reports backups running into business hours causing production performance issues. · service reliability

Yeah I mean, I submitted the case on — I think it was the 4th? And honestly I expected a faster response, I'll be honest with you. We're paying for premium support.

-1Expresses disappointment with slow response despite premium support. · support experience

You're right, and I — I want to own that, the initial response time on this case wasn't where it should've been and I'm sorry about that. I want to make sure we make good use of this time right now though. Can you walk me through what you're actually seeing? Like, what was the original backup window you had configured?

So we had it set to run from 1 AM to 5 AM, that's our four-hour window, which historically has been plenty. But now these jobs are running until — I've seen it go to 9, 9:30 in the morning. That's four to five hours of overage and that's right in the middle of when our warehouse and e-commerce teams are ramping up.

-1Describes four to five hours of overage disrupting warehouse and e-commerce teams. · service reliability

Okay, and are all of your backup jobs affected or is it specific workloads — like, is it just certain VMs, certain datasets, or is this across the board?

That's a good question. It seems like it's — it's pretty broad, but the worst offenders are our SQL Server instances. We've got three pretty beefy SQL servers running our e-commerce database and our inventory management system, and those are the ones really blowing past the window. Some of the smaller file server backups are also slow but not as dramatic.

Got it, so SQL workloads are the primary pain point. And are these incremental backups that are running nightly or are you doing full backups every night?

Right, yeah, incrementals ballooning like that is definitely a red flag. A few things I want to check — have you seen any significant growth in your data change rate? Like, is there any reason your daily change set might have gotten larger around that time, end of month processing, new application rollouts, anything like that?

Yeah, no, you're right, a quarterly process shouldn't account for that magnitude of slowdown. I want to pull up your backup job logs from the Aegis Protect console, do you mind if I take a minute to look at those?

Okay, so I'm looking at your environment here... and yeah, I can see the job durations, oof — yeah that's significant. Your SQL backup for, looks like it's the instance labeled SQLPROD01, that one's averaging about eight hours over the last ten days. That's uh, that's not good.

Right, exactly! And I keep getting told by my team to just expand the window but that's not a real fix, I shouldn't have to carve out more production hours for backups that used to run in under two hours.

-1Rejects expanding window as a real fix. · service reliability

No, absolutely not, that's a band-aid, not a solution. Let me look at the throughput metrics here... okay so I'm seeing the backup throughput dropped pretty substantially — it looks like you were averaging around 800 megabytes per second before March 27th and now you're sitting around 180 to 220. That's a massive drop in throughput.

I hear you, and I want to be transparent — so looking at our platform logs, around March 10th through the 18th we had a significant incident with Aegis Detect, our threat monitoring pipeline, and as part of the remediation work we pushed some backend infrastructure updates. Now, Protect and Detect run on separate systems, but some of the underlying network routing configurations were touched as part of that work, and I'm wondering if — and I need to confirm this with our infrastructure team — if some of those changes affected the data transfer paths for backup jobs in your region.

Wait, so a problem with a completely different module potentially broke our backups and nobody flagged this to customers? That's — I mean, I'm sorry David but that's pretty alarming to me.

-1Alarmed that a different module's incident may have affected backups without customer notification. · support experience

I understand your frustration, Dominic, and I'm not going to spin this — if that's what happened, it shouldn't have happened that way and customers should have been proactively notified. I'm still confirming the root cause but I want to be upfront with you about what I'm seeing rather than tell you nothing.

Okay so here's what I can do right now — I can escalate this internally to our Protect infrastructure team with the throughput data I'm looking at, and I also want to check your backup agent version because I'm seeing you're running 4.1.2, and we released 4.2.0 about three weeks ago which had some improvements to the data transfer compression algorithms. That could help even before we nail down the root cause.

That's a fair question — 4.2.0 was staged as an opt-in update rather than automatic because of some compatibility flags for SQL Server environments, which is ironic given your situation. But let me get that queued up for your environment, I'll push it out as a manual deployment through the console and you should see it applied within the hour if you can confirm your agents are reachable.

They should be, yeah. And that update — is that going to fix the throughput issue or is that just going to help a little bit?

Honest answer? It should help, probably meaningfully, but I don't want to promise you it's the full fix until I understand whether there's a network routing issue on our backend. I'm going to get the infra team on this today — I'm flagging it as a P1 on my end — and I want to schedule a follow-up call with you tomorrow morning, before your backup window starts, so we can walk through whatever findings we have.

None taken, I'll make that happen. I'll have a Protect platform engineer on the call. Can we say 10 AM your time? And what timezone are you in?

Perfect, I'll send a calendar invite to the email on your account within the next 30 minutes. And Dominic, one more thing — is there anything I can set up as a temporary workaround for tonight? For example, I can look at adjusting the backup job priority settings and throttling some of the lower-priority file server jobs so the SQL workloads get more of the available bandwidth during the window.

Yeah, yeah, let's do that. The SQL servers are the priority, the file servers we can live with being a little slower if it means the SQL jobs finish closer to on time.

Okay, I'm going to make those priority adjustments now in your policy configuration, and combined with the agent update that should — hopefully — get tonight's run into a better place. I'll also set up an alert so if the SQL job duration exceeds six hours I get paged directly, and I'll check in on it tomorrow morning before our call.

Alright. I mean, look, I'm still frustrated that we're two weeks into this and I'm just now getting real traction, but — I do appreciate you being straight with me about what you're seeing and not just giving me the runaround. Let's see what tomorrow brings.

-1Still frustrated after two weeks but acknowledges transparency. · support experience