Observations

What went wrong, and what to do about it

Real breaches, each put through the same four questions: one control that worked, one that did not, one number worth watching, and what should have been done instead. Every entry cites a primary source, because the point of this section is that you can take it to a meeting.

NO. 08/Government / Water utilities/11 September 2026

The same red team against two SOCs, and the alert queue that swallowed one

CISA Advisory AA26-237A

CISA ran two red team assessments at the same time, using similar tradecraft against a government services organisation and a water utility. Both ended the same way: full domain compromise, sensitive business systems reached, cloud resources reached. One of them caught the initial access within minutes and quarantined the hosts, which forced the red team into an assume breach model to carry on. The other never detected anything at all. The two organisations were running comparable tooling, and that is what makes this advisory worth the time.

What worked

Tuning. That is the whole finding, and it is unusually well controlled for a public document, because the same activity was run against both networks. At Organization B it produced a medium-severity alert that somebody triaged, and three workstations came off the network. At Organization A it produced alerts that nobody picked out of the volume. The alert text was "An executable file loaded an unexpected DLL file". No signature, no threat intelligence, no name for the thing. It stood out because Organization B had a baseline and had filtered the routine traffic away from it, so an anomaly could still look like one. B also let its defenders act: they triaged, found the root cause, and took the fix to engineering themselves rather than waiting to be told they were allowed to.

What failed

Organization A, at every layer above the sensor. Its tools were never tuned, so routine business activity and false positives ran at a volume the SOC could not work through, and the intrusion sat inside that volume looking like everything else. Then the organisation finished the job: several SOCs that did not talk to each other, no shared visibility with the system owners, and staff who did not know what they were authorised to do on which segment of the network. With no escalation procedure to fall back on, they defaulted to wait and see. Nobody in that chain did anything unreasonable, which is the uncomfortable part.

Organization B was better at detection and no better at hardening. Its Machine Account Quota was set to 1,000 for every domain user, and a service account held AllExtendedRights over a domain controller, which is a DCSync and then the whole domain. Detection bought time; it did not close the path.

In the cloud they failed identically. Both granted excessive permissions to applications, neither applied Conditional Access to workload identities, and neither had a process for revoking access and refresh tokens after a compromise. That last gap is the one to sit with: it means an actor you have evicted from the network still has the tenant.

The number

20 minutes

the slowest of Organization B's three workstation isolations, against no response at all from Organization A to the same red team activity.

What to do instead

Tune what you already own before you buy anything else. Two organisations, the same tradecraft, opposite outcomes, and the variable was a baseline rather than a product. Go and count what share of last week's alerts your analysts closed without investigating. That share is your actual detection coverage, and it is usually nothing like the coverage on the licence.

Then give your defenders the authority to act on what they find. Organization A's analysts were not short of alerts, they were short of permission, and "wait and see" is what a team defaults to when nobody has written down who may pull a host off the network at two in the morning. Write that down.

Then fix the two cloud gaps both of them shared, because those are the ones that survive eviction. Apply Conditional Access to workload identities and not only to people, and write the token revocation procedure before you need it: refresh tokens outlive the containment you are proud of, and nobody improvises that step mid-incident. While you are there, go looking for long-lived static IAM keys set never to expire. Organization A's were sitting in configuration files in users' home directories, which is where yours will be.

Sources

Named for the doctrine distinction between a lesson observed and a lesson learned: a lesson is not learned until behaviour has changed and somebody has checked. Pretending otherwise is how the same finding turns up in three consecutive post-incident reports.