Support Case #7615 - Crestline Wealth Group Policy Sync Delay

I completely understand and I apologize for the delay in getting you on a live call. I do want to dig into this with you today and get it resolved. So the notes in the case mention a policy sync delay in Aegis Identity — can you walk me through what you're seeing?

Sure. So basically, we push policy updates through Identity — role assignments, access group changes — and they're just... not propagating. Or they are, but it's taking, like, four, five hours sometimes. And in a financial services environment that is not acceptable. We had a situation last Thursday where we provisioned a new analyst and she couldn't access core systems until the next morning.

-1Customer describes four-to-five-hour sync delays and an analyst unable to access systems until the next morning. · product capability

It's both, honestly. But the provisioning one is the one that's really biting us. Role-based policy updates seem to eventually get there, but it's slow. The provisioning — that's where it's the worst.

-1Customer says the provisioning delay is 'really biting us'. · product capability

I mean, we started noticing it, I want to say, around mid-March? Maybe a little before? But we thought it was just, you know, a temporary thing. It wasn't until it kept happening that we actually opened a ticket.

Mid-March, okay. I do want to flag something — and I want to be transparent with you about this — we did have a significant incident with Aegis Detect in that timeframe, March 10th through the 18th, that was a pipeline-level issue. Now that was Detect, not Identity, but I want to check whether there were any downstream effects on shared services. Can you tell me — are you running Detect alongside Identity?

Yeah, we have Detect and Identity, we don't have Protect or Comply. Well — actually we're supposed to be getting Comply soon but that's a whole other conversation.

While you're looking at that — I just want to say, like, this is impacting us more than I think the ticket description conveys. We're a wealth management firm. We have compliance requirements. We have auditors. If someone has access they shouldn't have, or doesn't have access they should have, that's not just an inconvenience, that's a potential audit finding.

-1Customer stresses compliance and audit risk from access delays. · product capability

No, you're absolutely right and I take that seriously. Financial services environments have zero tolerance for this kind of lag and I'm not going to minimize it.

I appreciate that. It just — it feels like we report these things and then it kind of goes into a black hole, you know? I submitted the ticket on the 22nd and I got one automated response and then nothing until you called today.

-1Customer says the ticket went into a black hole with no human response for five days. · support experience

That's fair feedback and I'll make sure that's flagged internally. You should not have had a five-day gap with no human response on a Priority 2 case. That's on us. Okay, so I'm looking at your sync logs now and... yeah, I'm seeing what you're describing. There are queue depth spikes — significant ones — on your Identity sync worker. They're clustering around, it looks like, every few hours.

So is that a known bug or what? Like what's causing it?

So, I'm looking at your configuration and I think I can see what's happening. Your sync worker is configured with a pretty aggressive polling interval — looks like it's set to sixty seconds — but your directory size, you've got a pretty large AD environment, right? How many users roughly?

We're around 800 users, give or take.

Okay so 800 users, and you're on a shared sync cluster tier — I can see that here — and what's happening is the polling cycle is completing but the write queue is backing up because the worker isn't getting enough processing headroom between cycles. It's essentially falling behind itself. Now whether that's purely a configuration issue or whether there's an underlying bug causing the worker to be slower than it should be, I want to be honest, I'm not a hundred percent sure yet.

Well, I mean — we haven't changed our configuration. This was working fine before mid-March. So something changed on your end.

-1Customer notes no configuration change on their end and insists something changed on the vendor side. · product capability

That's a fair point and I don't want to dispute that. Something may have shifted in how the shared cluster is handling workloads, possibly related to the remediation work we did after the Detect incident. I want to get our backend engineering team to look at this because if the circuit breaker changes we made affected cluster resource allocation, that could be contributing to what you're seeing.

I understand why that's frustrating. I'm not going to make excuses. What I can do right now — there are a couple of things. First, I can escalate this to a P1 and get engineering eyes on it today, not tomorrow, today. Second, I can make a configuration change on your sync worker right now — I can move you off the shared cluster tier to a dedicated worker, which should immediately reduce the queue backup. It won't fix the root cause but it should get you back to normal sync times while we investigate.

Okay. Yeah, let's do that. How long does the dedicated worker change take?

It's usually — it's a backend change, so I need to submit a provisioning request, it typically takes about 30 to 45 minutes to propagate. You might see a brief sync pause of like two or three minutes during the cutover but it shouldn't be disruptive.

Nothing on your end. I'll initiate it right after this call and I'll send you an email confirmation when it's done. I'd ask you to monitor sync times over the next couple of hours and let me know if you're still seeing delays above, let's say, ten minutes — because on a dedicated worker at your directory size, you should be well under that.

That's the right question. I'm going to escalate this case with my notes and the log data I'm pulling right now. Engineering's SLA for a P1 investigation is 48 hours to root cause determination. I can't promise a fix in 48 hours but I can promise you'll have a definitive answer on what caused it and a remediation plan. And I will personally follow up with you — not an automated email — on Wednesday.

Please do. And Derek — I know you mentioned Comply is coming up for you. Once this is stabilized I'd actually recommend making sure your Identity configuration is reviewed before that goes live, just to make sure the integrations are set up cleanly. But that's a conversation for after we get this sorted.

Yeah, yeah that makes sense. One thing at a time. I just — look, I like the platform overall, I really do, but when stuff like this happens and it takes days to even get someone on the phone, it makes me nervous about renewal conversations.

-1Customer likes the platform but says support delays make him nervous about renewal conversations. · support experience

I hear you, and that's feedback I'm going to make sure gets to the right people. Your account team should be aware of this experience regardless of the technical outcome. I'll flag it.

Okay. Alright. Yeah, let's go ahead and do the dedicated worker thing and I'll watch the sync times this afternoon.

Perfect. I'll get that initiated right now. You'll get an email from me within the hour with confirmation and I'll include a direct line for this case so you don't have to go through the queue again. And Wednesday at the latest you'll hear from me with an engineering update.

Alright. Thanks Sarah. I appreciate you actually digging into it.

+1Customer appreciates Sarah actually digging into the issue. · support experience