Real breaches, each put through the same four questions: one control that worked, one that did not, one number worth watching, and what should have been done instead. Every entry cites a primary source, because the point of this section is that you can take it to a meeting.
NO. 30/Technology/3 October 2026
One engineer's laptop, and a 2FA-backed session that reached production secrets
CircleCI, 'CircleCI Jan 4, 2023 security incident report' (circleci.com/blog, dated 12 January 2023)
Not yet reviewed This one was written here from the source and has not yet been signed off, so it carries no byline. Every factual claim in it comes from the document linked below and can be checked against it. The judgements are ours, and until somebody has been back through them they are marked as unverified rather than presented as settled.
Malware on a CircleCI engineer's laptop, compromised on 16 December 2022, stole a valid, 2FA-backed SSO session. The engineer's regular duties included generating production access tokens, so the intruder reached a subset of production systems, carried out reconnaissance on 19 December and on 22 December exfiltrated customer environment variables, tokens and keys. CircleCI learned of the intrusion after a customer reported suspicious GitHub OAuth activity on 29 December, and told all customers to rotate their secrets on 4 January 2023.
What worked
A customer's report on 29 December started the investigation, and by 4 January CircleCI had established the scope and the entry path. The report says the company's authentication, security and tracing tools allowed it to diagnose and remediate the issue comprehensively. CircleCI shut down the compromised employee's access, cut production access to a very small group, rotated all potentially exposed production hosts, and revoked or rotated Project API tokens, Personal API tokens, Bitbucket tokens and GitHub OAuth tokens on customers' behalf.
What failed
The malware was not detected by CircleCI's antivirus software. Because the theft was of a session cookie, the second factor had already been satisfied and the intruder could impersonate the employee from a remote location. The exfiltrated data was encrypted at rest, but the intruder extracted encryption keys from a running process. The first warning came from a customer on 29 December, after the last recorded unauthorised activity in production on 22 December. Rotating every customer's GitHub OAuth tokens began on 31 December and was not complete until 7 January.
The number
Fewer than 5
customers had reported unauthorised access to third-party systems when the report was published, yet every secret stored on the platform had to be treated as exposed.
What to do instead
Assume a completed MFA challenge can be stolen as a session cookie and design for it: short session lifetimes, device-bound sessions, and step-up authentication before production access or token generation. One engineer's routine privileges should not include minting production access tokens without a second control. Encryption at rest gives little protection when the keys sit in a process an intruder can read, so hold keys where each use is separately authorised and logged. If you store secrets with a CI provider, prefer short-lived OIDC tokens, restrict inbound IP ranges, and rehearse a full secret rotation before you need one.
Four credentials believed unused, and the Atlassian server they opened
Cloudflare blog, 'Thanksgiving 2023 security incident', 1 February 2024
Not yet reviewed This one was written here from the source and has not yet been signed off, so it carries no byline. Every factual claim in it comes from the document linked below and can be checked against it. The judgements are ours, and until somebody has been back through them they are marked as unverified rather than presented as settled.
From 14 November 2023 a threat actor used one access token and three service account credentials, taken in the October 2023 Okta compromise and never rotated, to probe Cloudflare and then reach its self-hosted Atlassian server. The actor accessed Confluence and Jira, created an Atlassian user account, installed the Sliver framework through the ScriptRunner for Jira plugin, and viewed 120 code repositories, 76 of which Cloudflare treats as exfiltrated. Cloudflare detected the actor on 23 November and terminated all access on 24 November.
What worked
On 23 November the actor added the Smartsheet service account to an administrator group at 15:58, an automated alert reached the security team at 16:00, and the SOC deactivated the account at 16:35. The actor's attempts on Cloudflare's Okta instance and the Cloudflare Dashboard were denied, and the attempt to reach a console server for the not yet in production São Paulo data centre failed. Cloudflare credits access controls, firewall rules and hard security keys enforced through its own Zero Trust tools for limiting lateral movement, and says no customer data or systems were affected.
What failed
Out of thousands of credentials leaked in the Okta compromise, one service token and three service accounts were not rotated because they were mistakenly believed to be unused. The Moveworks service token granted remote access into the Atlassian system, and the Smartsheet service account held administrative access to Jira, which let the actor create a lasting user account and install Sliver. The actor first gained access to Atlassian services on 15 November and was not noticed until the administrator group change on 23 November. A non-enforced ACL is what allowed the attempt on the São Paulo console server to be made at all. The clean-up, named Code Red, ran from 27 November to 5 January and included rotating more than 5,000 production credentials and forensic triage of 4,893 systems.
The number
4 credentials
out of thousands leaked in the Okta compromise were left unrotated, because they were believed to be unused.
What to do instead
After a supplier breach, rotate from an inventory and not from memory: a credential counts as unused only when logs prove it, and then it should be deleted rather than left in place. Do not give SaaS integration accounts administrative rights over the systems they plug into. Alert on changes to administrator groups, since that is the detection that fired here. Hold staging and pre-production equipment to the same enforced ACLs as production. Expect the clean-up to cost far more than the rotation you skipped.
One service account password in a personal Google profile, and 134 customers' support files
Okta Security, 'Unauthorized Access to Okta's Support Case Management System: Root Cause and Remediation', 3 November 2023
Not yet reviewed This one was written here from the source and has not yet been signed off, so it carries no byline. Every factual claim in it comes from the document linked below and can be checked against it. The judgements are ours, and until somebody has been back through them they are marked as unverified rather than presented as settled.
From 28 September to 17 October 2023 a threat actor used a service account to reach files in Okta's customer support system associated with 134 customers. Some were HAR files containing session tokens, which the actor used to hijack the Okta sessions of 5 customers. Okta found that an employee had signed in to a personal Google profile in Chrome on an Okta-managed laptop, and the service account's username and password had been saved into that personal Google account, which Okta considers the most likely route of exposure.
What worked
Customers raised the alarm: 1Password reported suspicious activity to Okta Support on 29 September and BeyondTrust on 2 October. The turning point was an IP address supplied by BeyondTrust on 13 October. With it, Okta identified the service account on 16 October, disabled it and terminated its sessions on 17 October, and revoked the session tokens embedded in the HAR files.
What failed
For 14 days of active investigation Okta did not identify the suspicious downloads in its own logs. Its analysts looked at access to support cases, but the actor went straight to the Files tab of the support system, which generates an entirely different log event with a different record ID. The service account's credential was stored in the support system itself and carried permission to view and update customer support cases. Blocking personal Google profiles in Chrome on Okta-managed laptops was a remediation applied only afterwards. A gap in the support system logs also hid the final hours of the actor's access, so further downloaded files were only found on 19 October.
The number
14 days
of active investigation in which Okta did not identify the suspicious downloads in its own logs.
What to do instead
Block personal browser profiles and password sync on managed devices through enterprise browser policy, rather than trusting staff not to sign in. Keep service account secrets in a vault, scope them narrowly, and alert on any use from an unfamiliar network. Before an incident, list every log event type your SaaS vendors emit for file access and test that your hunt queries catch each route to the same data. Treat HAR files as credentials: strip session tokens before they are uploaded and keep token lifetimes short. When two customers report the same oddity within days, assume the common factor is you.
Named for the doctrine distinction between a lesson observed and a lesson learned: a lesson is not learned until behaviour has changed and somebody has checked. Pretending otherwise is how the same finding turns up in three consecutive post-incident reports.
