Observations

What went wrong, and what to do about it

Real breaches, each put through the same four questions: one control that worked, one that did not, one number worth watching, and what should have been done instead. Every entry cites a primary source, because the point of this section is that you can take it to a meeting.

NO. 33/Healthcare/15 October 2026

Antivirus in monitor mode, and eight weeks inside the national health network

PwC, 'Conti cyber attack on the HSE: Independent Post Incident Review', commissioned by the HSE Board, 3 December 2021

Not yet reviewed
This one was written here from the source and has not yet been signed off, so it carries no byline. Every factual claim in it comes from the document linked below and can be checked against it. The judgements are ours, and until somebody has been back through them they are marked as unverified rather than presented as settled.

On 18 March 2021 a workstation at Ireland's Health Service Executive was infected when its user opened a malicious Excel file attached to a phishing email sent on 16 March. The attacker operated in the environment over an eight week period, compromising highly privileged accounts, servers, and six voluntary hospitals and one statutory hospital, before executing Conti ransomware at about 01:00 on 14 May 2021. The HSE switched off all its IT systems and disconnected the National Healthcare Network from the internet, and recovery efforts continued for over four months.

What worked

The Department of Health acted on an alert from its security provider on 13 May, contacted the NCSC and had endpoint detection and response tooling installed, which blocked the ransomware across the vast majority of its infrastructure. Hospital A brought in an incident response provider on 12 May, reset passwords for 4,500 accounts and changed firewall configuration. The HSE's shutdown on 14 May removed the attacker's access. PwC also lists as mitigating factors the relative simplicity of the attack and the attacker's release of a decryption key on 20 May.

What failed

On 31 March the HSE's antivirus detected Cobalt Strike and Mimikatz on the first infected workstation, but it was set to monitor mode and did not block them; that workstation had not had antivirus signatures updated for over a year. On 13 May the HSE's security provider emailed the Security Operations team about unhandled threat events since 7 May on at least 16 systems, and the team asked the Server team to restart servers. When Hospital A asked about two IP addresses, the HSE incorrectly concluded that the suspicious activity originated from Hospital A. Hospital C was advised that its risk was low because antivirus had remediated its Cobalt Strike alerts, and did not start an investigation. PwC found there was no security monitoring capability able to effectively detect, investigate and respond to alerts across the HSE's IT environment.

The number

8 weeks

the attacker operated inside the environment, through several detections, before the ransomware was executed.

What to do instead

Run endpoint protection in blocking mode, keep signatures current, and make a named team answerable for every unhandled detection within a fixed time. A Cobalt Strike or Mimikatz detection is an incident by definition, even when the antivirus reports it cleaned; restarting the server is not a response. When a connected organisation reports suspicious traffic from your addresses, examine your own hosts first. Build a monitored security operations capability with authority across the whole network, take offline backups on a fixed schedule, and keep a prioritised list of clinical systems ready for recovery.

NO. 11/Cross-sector/12 September 2026

Detection was not the limiting factor. Response time was.

Darktrace, Hiding in Plain Sight: Uncovering a Multi-Stage Ransomware Attack Through Behavioral Detection

Compromised VPN credentials put an operator on the network. Over the next three days they scanned east to west across a broad range of ports, triggered Active Directory replication consistent with a credential dump, ran command and control over SSL, pushed data out to Wasabi cloud storage, and encrypted thousands of files. The customer had behavioural detection running and it saw all of it, stage by stage, while the intrusion was still reversible. Three days is a long time to be inside a network that is watching you.

What worked

The detection, and Darktrace is unusually blunt about which half of the problem it solved: "detection was not the limiting factor; response time was." The scanning was flagged. The replication anomaly was flagged. The command and control was flagged. More than 30 rare outbound connections to a Wasabi endpoint were flagged before a single file was encrypted, along with close to 3GB going to one external address and just under 4GB to another. None of that needed a signature or prior knowledge of the tooling, which is the case for behavioural detection made about as well as a vendor can make it.

What failed

Everything downstream of the alert. Autonomous Response was licensed and switched on in this environment, and the report says it "was not fully configured across the impacted devices, allowing the attack to progress to ransomware deployment." So the control that would have cut the sessions was bought, enabled, and inert on the machines that mattered. Nobody acted by hand either; the three days are the proof of that.

The uncomfortable reading is that this customer spent their detection budget correctly and finished in the same place as somebody who had spent nothing. Darktrace says as much in the counterfactual: "Prompt investigation and containment of devices exhibiting anomalous behavior could have prevented lateral movement, data exfiltration, and ultimately ransomware deployment." Every one of those stages was on a screen when it happened.

The number

3 days

from the first use of the compromised VPN credentials to encryption, with alerts raised at every stage in between and none of them acted on.

What to do instead

Go and check which of your automated response actions are actually armed, on which machines. Not whether the product is licensed, and not whether the feature is on in the console, but whether it will act tonight on the hosts that decide the outcome: domain controllers, file servers, the VPN concentrator, the jump boxes. A response capability configured across most of an estate is one the attacker routes around, and the part that was missed is never random. Ask for the coverage list rather than the coverage number.

Then write down who may cut a session, and when, and without whose permission. This intrusion ran for three days through alerts that a person could have acted on, and the reason teams default to watching is almost never that they did not believe the alert. It is that nobody has told them in writing that they are allowed to pull a host off the network at two in the morning and be wrong about it.

Then take the exfiltration signal seriously on its own. Rare-destination cloud storage is the most useful single rule here: more than 30 connections to a Wasabi endpoint from a host that had never spoken to it, carrying gigabytes, in an environment where nothing legitimate uses that provider. If your egress alerting is tuned on total volume rather than on destination rarity, that traffic looks like a backup job.

And rotate the VPN credentials and check what else they open. This started with valid credentials on remote access, which is the same first step as most of what lands in this section.

NO. 08/Government / Water utilities/11 September 2026

The same red team against two SOCs, and the alert queue that swallowed one

CISA Advisory AA26-237A

CISA ran two red team assessments at the same time, using similar tradecraft against a government services organisation and a water utility. Both ended the same way: full domain compromise, sensitive business systems reached, cloud resources reached. One of them caught the initial access within minutes and quarantined the hosts, which forced the red team into an assume breach model to carry on. The other never detected anything at all. The two organisations were running comparable tooling, and that is what makes this advisory worth the time.

What worked

Tuning. That is the whole finding, and it is unusually well controlled for a public document, because the same activity was run against both networks. At Organization B it produced a medium-severity alert that somebody triaged, and three workstations came off the network. At Organization A it produced alerts that nobody picked out of the volume. The alert text was "An executable file loaded an unexpected DLL file". No signature, no threat intelligence, no name for the thing. It stood out because Organization B had a baseline and had filtered the routine traffic away from it, so an anomaly could still look like one. B also let its defenders act: they triaged, found the root cause, and took the fix to engineering themselves rather than waiting to be told they were allowed to.

What failed

Organization A, at every layer above the sensor. Its tools were never tuned, so routine business activity and false positives ran at a volume the SOC could not work through, and the intrusion sat inside that volume looking like everything else. Then the organisation finished the job: several SOCs that did not talk to each other, no shared visibility with the system owners, and staff who did not know what they were authorised to do on which segment of the network. With no escalation procedure to fall back on, they defaulted to wait and see. Nobody in that chain did anything unreasonable, which is the uncomfortable part.

Organization B was better at detection and no better at hardening. Its Machine Account Quota was set to 1,000 for every domain user, and a service account held AllExtendedRights over a domain controller, which is a DCSync and then the whole domain. Detection bought time; it did not close the path.

In the cloud they failed identically. Both granted excessive permissions to applications, neither applied Conditional Access to workload identities, and neither had a process for revoking access and refresh tokens after a compromise. That last gap is the one to sit with: it means an actor you have evicted from the network still has the tenant.

The number

20 minutes

the slowest of Organization B's three workstation isolations, against no response at all from Organization A to the same red team activity.

What to do instead

Tune what you already own before you buy anything else. Two organisations, the same tradecraft, opposite outcomes, and the variable was a baseline rather than a product. Go and count what share of last week's alerts your analysts closed without investigating. That share is your actual detection coverage, and it is usually nothing like the coverage on the licence.

Then give your defenders the authority to act on what they find. Organization A's analysts were not short of alerts, they were short of permission, and "wait and see" is what a team defaults to when nobody has written down who may pull a host off the network at two in the morning. Write that down.

Then fix the two cloud gaps both of them shared, because those are the ones that survive eviction. Apply Conditional Access to workload identities and not only to people, and write the token revocation procedure before you need it: refresh tokens outlive the containment you are proud of, and nobody improvises that step mid-incident. While you are there, go looking for long-lived static IAM keys set never to expire. Organization A's were sitting in configuration files in users' home directories, which is where yours will be.

Sources

Named for the doctrine distinction between a lesson observed and a lesson learned: a lesson is not learned until behaviour has changed and somebody has checked. Pretending otherwise is how the same finding turns up in three consecutive post-incident reports.