One engineer's laptop, and a 2FA-backed session that reached production secrets
Not yet reviewed
This one was written here from the source and has not yet been signed off, so it carries no byline. Every factual claim in it comes from the document linked below and can be checked against it. The judgements are ours, and until somebody has been back through them they are marked as unverified rather than presented as settled.
Malware on a CircleCI engineer's laptop, compromised on 16 December 2022, stole a valid, 2FA-backed SSO session. The engineer's regular duties included generating production access tokens, so the intruder reached a subset of production systems, carried out reconnaissance on 19 December and on 22 December exfiltrated customer environment variables, tokens and keys. CircleCI learned of the intrusion after a customer reported suspicious GitHub OAuth activity on 29 December, and told all customers to rotate their secrets on 4 January 2023.
What worked
A customer's report on 29 December started the investigation, and by 4 January CircleCI had established the scope and the entry path. The report says the company's authentication, security and tracing tools allowed it to diagnose and remediate the issue comprehensively. CircleCI shut down the compromised employee's access, cut production access to a very small group, rotated all potentially exposed production hosts, and revoked or rotated Project API tokens, Personal API tokens, Bitbucket tokens and GitHub OAuth tokens on customers' behalf.
What failed
The malware was not detected by CircleCI's antivirus software. Because the theft was of a session cookie, the second factor had already been satisfied and the intruder could impersonate the employee from a remote location. The exfiltrated data was encrypted at rest, but the intruder extracted encryption keys from a running process. The first warning came from a customer on 29 December, after the last recorded unauthorised activity in production on 22 December. Rotating every customer's GitHub OAuth tokens began on 31 December and was not complete until 7 January.
The number
Fewer than 5
customers had reported unauthorised access to third-party systems when the report was published, yet every secret stored on the platform had to be treated as exposed.
What to do instead
Assume a completed MFA challenge can be stolen as a session cookie and design for it: short session lifetimes, device-bound sessions, and step-up authentication before production access or token generation. One engineer's routine privileges should not include minting production access tokens without a second control. Encryption at rest gives little protection when the keys sit in a process an intruder can read, so hold keys where each use is separately authorised and logged. If you store secrets with a CI provider, prefer short-lived OIDC tokens, restrict inbound IP ranges, and rehearse a full secret rotation before you need one.