Observations

What went wrong, and what to do about it

Real breaches, each put through the same four questions: one control that worked, one that did not, one number worth watching, and what should have been done instead. Every entry cites a primary source, because the point of this section is that you can take it to a meeting.

NO. 18/SaaS / Technology/12 September 2026

Industrial-scale extraction, through the API you sell

CISA Advisory AA26-251A, China-Based Artificial Intelligence Companies Conducting Industrial-Scale Distillation Campaigns Against U.S. AI Companies

CISA says China-based AI companies have been systematically extracting the capabilities of US frontier models by querying them at scale, taking "billions of tokens across millions of exchanges/requests" from variants of Claude, GPT, Gemini and Grok since at least late 2024. The route is not an intrusion. It is the product’s own inference API, reached through accounts "not registered to legitimate users" and through premium subscriptions bought in bulk and shared across teams of developers, with automated failover between pathways whenever one gets blocked. Individual campaigns run from days to months, at query volumes in the thousands to millions per domain.

What worked

The advisory does not name a control that held. It does record that blocking happened and was routed around, which is a finding in itself: somebody was detecting this and acting on it, and the answer was an automatic switch to another pathway.

What failed

Account identity first. The accounts were not registered to real people, so there was nothing to revoke that could not simply be created again, and any verification weak enough to automate sets the effective rate limit to infinity.

Then the commercial relationship, which is harder. The premium subscriptions were bought legitimately and then shared across teams, and no technical control sees that, because from the platform’s side it is one paying customer using an allowance they paid for. The prohibition lives in the terms of use, which is a contract rather than a control.

The part worth sitting with, for anybody running an API product, is that this is not an attack in the sense the rest of this section means. Every request was well formed, authenticated and paid for. What made it an incident was volume, pattern and intent, none of which is visible in a single call.

The number

billions

of tokens taken across millions of requests from US frontier models since at least late 2024, in campaigns running days to months at thousands to millions of queries per domain.

What to do instead

If you sell inference, your abuse detection has to work on the shape of a customer rather than the shape of a request. CISA’s first recommendation is monitoring of anomalous prompts, accounts, networks and behaviours together, and the word carrying that sentence is "together". A campaign like this looks ordinary at every individual layer and only appears when you join them.

Then make the block cost something. The most practical detail in the advisory is that blocking triggered automated failover between pathways, which means a block that is instant and unambiguous is a block that trains the client. Responses that degrade rather than refuse, and that cannot be attributed to one request, are much harder to build a failover around.

Then treat account registration as a security boundary and not a signup funnel. Fraudulent accounts were the way in, and everything downstream inherits whatever that check was worth.

And for everyone who does not sell an API, the transferable part is smaller and more useful. This is a product being used exactly as designed, at a volume and for a purpose nobody intended, by a customer who paid. Nothing in a normal security stack points at that. Work out who in your organisation would notice, and whether they have the data to.

Sources

Named for the doctrine distinction between a lesson observed and a lesson learned: a lesson is not learned until behaviour has changed and somebody has checked. Pretending otherwise is how the same finding turns up in three consecutive post-incident reports.