What breaks first when a machine learning system fails: the model, or the plumbing around it? On 18 November 2025 Cloudflare gave the industry a very clean answer. The model never ran.
The trigger was a permissions change on a ClickHouse database cluster. That change altered query semantics, the query that builds the Bot Management feature file started returning duplicate rows, the file roughly doubled in size, and it blew past a hard-coded 200-feature memory preallocation limit inside the Rust proxy that fronts Cloudflare’s network. The proxy panicked, and not on one machine. On every machine that picked up the file.
Cloudflare published a detailed timestamped postmortem the same day, and it is worth reading in full. What follows is the part I find most useful for anyone running ML in production. This was a config file that got bigger than a buffer someone sized years ago.
Detection took twelve minutes, diagnosis took an hour and forty-five
The outage started at 11:20 UTC. First visible errors appeared at 11:28 UTC and Cloudflare declared an incident at 11:32 UTC, so detection was fast: twelve minutes from onset to a declared incident. Diagnosis was not. Bot Management was identified as the root cause at 13:05 UTC, roughly one hour and forty-five minutes after the errors started.
Why the gap? The failure looked like something else entirely. Per Ars Technica’s coverage and Radware’s write-up, the initial traffic pattern resembled a massive DDoS, so the incident was first read as attack traffic. Cloudflare later explicitly ruled out a cyberattack.
There was a second reason the signal was confusing. The FL2 proxy re-reads the Bot Management feature file on a refresh cycle of roughly every five minutes. Depending on which nodes had regenerated the file and which had not, machines flipped between working and panicking. Error rates oscillated in waves rather than flatlining, which is exactly the shape you would expect from intermittent attack traffic and exactly the wrong shape for pointing an on-call engineer at a config artifact. The Roundz deep dive and a later Gremlin analysis both land on the same reading; Gremlin estimates roughly 90 minutes for the corrupted file to saturate the fleet and produce sustained high 5xx rates.
Once the cause was known, recovery was mechanical. Cloudflare rolled back to the known-good feature file at 13:37 UTC, core traffic was restored at 14:30 UTC, and all systems were fully operational at 17:06 UTC. That is five hours and forty-six minutes from onset to full recovery, three hours and ten minutes to core traffic. Cloudflare’s own postmortem calls it the company’s worst outage since 2019.
Degraded services included Workers KV, Access, Turnstile, the Cloudflare Dashboard and Email Security, all of which traverse the same edge path. That is the blast radius of putting a bot score on every request: the feature file becomes a hard dependency for anything that goes through the proxy, including the dashboard you would use to fix things.
The Rust module panicked where it could have served stale features
The technical failure is small enough to describe in one sentence. The Bot Management module preallocated memory for a fixed number of features, the file arrived with more, and the Rust code panicked instead of truncating, skipping, or falling back to a cached last-known-good file.
There was no bounds check on the feature count and no circuit breaker around the module. That is the entire difference between a six-hour global outage and a line in a log file saying “feature file rejected, serving with stale features.”
I want to be fair to whoever wrote that limit. A hard-coded 200 is a perfectly reasonable engineering decision at the time you make it. You know the feature set, you know it grows slowly, you preallocate to avoid per-request allocation on the hottest path in the company. The problem is that the constant became a liveness condition for the whole network, and nothing between the ClickHouse query and the proxy asked whether the file it was handing over looked like the file from five minutes ago.
Note also where the change originated. Nobody deployed new proxy code, nobody shipped a new model, nobody changed the bot detection logic. Someone changed permissions on a database cluster. In most organisations that action is not classified as a production change to the traffic path at all, which is precisely why it did not get canary treatment.
Cloudflare’s fix list says nothing about the model
Cloudflare’s stated remediation covers four things: tighter validation on the file-generation pipeline, rejecting files that exceed expected size or structure, slower fleet-wide propagation of new files, and fail-safes so one bad configuration cannot take down core traffic. They also committed to expanded canarying for database and configuration changes affecting production traffic, plus faster detection of abnormal file growth.
Read that list again and notice what is missing. Nothing about the model, nothing about bot detection accuracy, nothing about ClickHouse. Every item is about the artifact pipeline between a data store and a runtime.
My take: most teams running ML in production treat model artifacts and feature files as data, and they treat data as something that flows rather than something that deploys. That is the category error. **A feature file that every request depends on is a deployment artifact with the same blast radius as a binary**, and it deserves the same controls: schema validation at generation time, size and cardinality bounds checked at load time, staged rollout, and an explicit last-known-good to fall back to. Cloudflare had the model governance. What failed was the file governance.
Four things I would actually check this week if I ran a system with a similar shape:
- Find every hard-coded capacity constant on your hot path and ask what happens when the input exceeds it. Not “will it exceed it”, it will eventually. Ask whether the code panics, truncates, or degrades. If the answer is panic, and the artifact is fleet-wide, you have Cloudflare’s bug.
- Validate artifacts where they are produced, not only where they are consumed. Cloudflare’s fix rejects files that exceed expected size or structure. Row count against the previous file is the cheapest possible signal, and a doubling would have caught this one.
- Treat schema and permission changes in upstream data stores as production changes to every downstream consumer. The ClickHouse permissions change was invisible as a risk because the ownership boundary made it look like a database task.
- Make stale-but-working the default failure mode. If the new file fails validation, keep the old one and page someone. Serving yesterday’s bot scores is a worse product for a few hours. Serving 5xx is no product at all.
The refresh cycle deserves its own note. Five minutes is a sensible cadence for keeping a bot model current. It also means that once a bad file exists, you have five minutes of margin before it is everywhere, and the oscillation it produces will actively mislead your responders. Faster propagation is not automatically better, and Cloudflare’s remediation explicitly slows it down.
One thing I cannot judge from the outside: whether the 1h45m diagnosis time was mostly the DDoS misread or mostly the difficulty of correlating the oscillation with the refresh cycle. The postmortem gives timestamps but not the internal reasoning trail. My read is that the misdiagnosis cost the most, because a DDoS hypothesis sends you to traffic analysis and mitigation tooling, which is a long detour from “diff the config file against the one from an hour ago.”
There is a broader pattern worth naming. Cloudflare is unusually disciplined about the ML layer itself, and they have been public about internal AI infrastructure, including open-sourcing the AI workspace their whole workforce uses, built around agents that start with zero access. Careful engineering at the model and access layers did not help here, because the fragile part sat below both: a generated file, a fixed-size buffer, and no contract between them.
Six hours, one query, two hundred features.
