What happens to a safety filter when the model runs on a laptop with no network connection? One European investigator told Bloomberg the set of image-model variants under surveillance went from fewer than 12 two years ago to more than 500.
Bloomberg published an investigation on October 6, 2026 (updated October 9) based on interviews with more than a dozen child-safety experts and law-enforcement officials. It found that locally run open-weight image models (Stable Diffusion, FLUX, Qwen, Hunyuan) now appear routinely in child-exploitation cases, because offline generation skips hosted scanning, hash-matching and abuse reporting. Tracked variants went from fewer than 12 to more than 500 in two years.
Bloomberg names four model families in live cases
The reporting is specific in a way most AI-safety coverage never manages. Bloomberg names four model families that investigators said appear in active cases: Stable Diffusion from Stability AI, FLUX from Black Forest Labs in Germany, Qwen from Alibaba, and Hunyuan from Tencent. Bloomberg also reported that five of the seven most “violative” open-source models in comparative testing came from Chinese companies.
That last point will get the most political attention and deserves the most caution. Bloomberg does not, in the material available to me, describe the testing methodology behind that comparison, and “most violative” is a ranking rather than a measurement of real-world harm volume. I would read it as a signal about where safety post-training is weakest. It is not a scoreboard of who caused what.
The volume numbers come from outside Bloomberg and point the same direction. The Internet Watch Foundation assessed 8,029 AI-generated images and videos as realistic child sexual abuse material in 2025, up 14% year over year. Within that, AI-generated motion video went from 13 items in 2024 to 3,443 in 2025. On the US side, NCMEC’s CyberTipline received more than 400,000 reports in 2025 with a generative-AI nexus, of which over 182,000 involved possessing, generating or attempting to generate GAI CSAM.
The jump from 13 to 3,443 in video is the figure I keep coming back to. Still images were already a solved generation problem. Motion was not, and the gap closed inside one calendar year. Whatever enforcement posture was calibrated against 2024 volumes is now calibrated against nothing.
Offline inference passes through none of the detection chokepoints
Every detection layer the industry built between 2020 and 2024 sits at a chokepoint that offline inference never touches. Hash-matching (PhotoDNA and successors) works on files that cross a platform. Prompt-level classifiers work on API calls. Output scanning works when the output lands on someone’s server. Abuse reporting works when there is a vendor with logs.
Run a quantised checkpoint locally and none of that exists. There is no request to classify, no file to hash on upload, no account to suspend, no provider to send a CyberTipline report. The only artefacts left are the ones that reach distribution, which is to say the ones investigators find after the harm.
The jump from under 12 to over 500 is not 500 new foundation models. It is the fine-tune and merge ecosystem: LoRAs, community merges, re-releases with safety layers stripped. Each one is a separate thing to characterise, test and recognise in evidence, and they appear far faster than any lab can red-team them. That changes the enforcement economics. A base model release is a discrete event a regulator can attach obligations to. A derivative released by an anonymous account on a model-sharing site at 2am is not. The surveillance burden scales with the number of variants, and the number of variants scales with how easy fine-tuning has become.
Cinder’s checkpoint testing of FLUX.2 cut vulnerabilities 77% to 98%
The most load-bearing new evidence in this story did not come from Bloomberg. It came from a vendor case study, and it is useful anyway.
Black Forest Labs commissioned the online-safety firm Cinder to red-team five FLUX.2 open-weight releases at early, intermediate and final checkpoints before launch. Cinder ran nearly 4,000 adversarial prompts across text-to-image and image-to-image, covering CSAM and non-consensual intimate imagery. The prompt set included disguised prompts, composition from individually harmless components, “undressing” real people and de-aging real people. Black Forest Labs says it filters nude and pornographic material before training and works with the IWF to remove known CSAM.
Two results matter. Post-training safety measures cut vulnerabilities by 77% to 98% against earlier FLUX.2 checkpoints. And Black Forest Labs reported FLUX.2 had over 10x fewer vulnerabilities than other popular open-weight models from Alibaba, Tencent and ByteDance.
Take that 10x peer comparison with appropriate scepticism. It is a vendor-commissioned test of competitors, published by the vendor, with no independent replication. The checkpoint-over-checkpoint number is different. That comparison stays inside one model family, measured by the same firm with the same prompt set, which makes it the cleanest signal in the dataset. It says the gap between a model that has had safety post-training and one that has not is roughly an order of magnitude on this specific prompt battery.
The Cinder data cuts against both camps. It undermines “we just released the weights, what happens next is not our problem”, because a lab can evidently measure its own contribution to the harm surface before shipping. It equally undermines “open weights must be banned”, because a 77% to 98% reduction is a real engineering result and not a fig leaf. My expectation is that Brussels reads the first half and ignores the second, because the first half is legible to lawyers and the second half requires reading a methodology appendix.
Auditability is the weakest defence of open weights here
The standard defence of open weights runs on auditability, reproducibility and no vendor lock-in. I have made versions of that argument myself and I still think it holds for most model categories, including the ones I covered in the piece on open-source AI revenue reality. It holds less well here, for structural reasons rather than moral ones.
Auditability is a property you exercise on a model you control. The moment the weights are downloadable, the relevant actor is whoever fine-tunes away the safety behaviour, and the 500-variant number suggests that is a large and productive population. **Open weights give the defender inspection rights and the attacker modification rights, and modification is cheaper than inspection.**
My take: the honest position for anyone shipping open weights in 2027 is not that releasing is safe. It is that releasing carries a measurable cost, that the cost drops substantially with pre-release work, and that the lab should publish what it did and what the residual number was. Black Forest Labs has now set a floor for what that disclosure looks like. Labs that release without an equivalent will have a harder time explaining why.
There is also a jurisdictional asymmetry worth naming. Black Forest Labs is German, operating inside the EU AI Act’s enforcement reach. Alibaba and Tencent are not. If five of the seven most violative models in Bloomberg’s cited testing came from Chinese companies, European regulation lands hardest on the vendor that commissioned a red team and lightest on the vendors that did not. I do not have a clean answer to that. Distribution-side obligations on model-hosting platforms are the only lever I see that bites regardless of where the weights were trained, and those obligations are politically harder than obligations on labs.
Four things to settle before a regulator asks
If you ship anything that generates or accepts images in the EU, the regulatory conversation around this reporting will reach your product before it reaches the labs.
Inventory which weights are actually in your stack
Not the base model you licensed, the specific checkpoint and any community fine-tunes or LoRAs someone added. If a model in production came from a sharing site rather than a vendor release page, you cannot currently answer where its safety post-training went. Find those first.
Ask vendors for checkpoint-level red-team data, not a policy page
Cinder’s methodology (adversarial prompts at early, intermediate and final checkpoints, across text-to-image and image-to-image) is now a published reference point. “We filter training data” is a weaker answer than it was on October 5, 2026.
Keep server-side scanning even for locally run inference
You cannot scan what runs on a user’s machine, but you can scan what comes back to you: uploads, shared galleries, support attachments. Every detection mechanism described above still works at your perimeter. It is the only chokepoint you own.
Write down your weight-release position before someone asks for it
If you open-weight anything image-capable, decide now what pre-release testing you will do and what you will publish. Deciding this during a regulatory inquiry is worse.
For media and production teams, the model-selection calculus changes. I went through the licensing side of this in the ranking of AI video models an EU studio can legally ship, and the Bloomberg reporting adds a second axis: whether the vendor can show you what it tested before release.
Where I expect this to go by mid-2027
I expect pre-release safety evaluation to become a de facto condition of open-weight distribution in the EU within 12 months, enforced less by statute than by platform policy. The faster path runs through model-hosting platforms adding evaluation requirements to listing terms, because that moves at commercial speed rather than legislative speed.
I also expect at least one more vendor to publish Cinder-style checkpoint data in the next two quarters, because Black Forest Labs just made it cheap to look responsible and expensive to stay silent.
Less confident, and I will flag it as such: I think the variant-tracking number keeps growing faster than any law enforcement capability to characterise individual variants, which pushes investigators toward output-side detection and away from model attribution. If that happens, the policy debate about which lab released what matters less to actual case work than it currently appears, even as it matters more politically.
One thing I cannot assess from the available data: whether the IWF’s 14% year-over-year increase in assessed AI CSAM reflects more material or better detection. Both stories fit the number. The jump from 13 to 3,443 videos is harder to explain as a detection artefact, which is why I weight it more heavily.
The defensible position on open weights is no longer “the weights are public, so the harm is downstream”. It is “here is what we tested at each checkpoint, here is the reduction we measured, here is the residual”. If you ship or depend on open-weight image models in the EU, that is the document you need, and you need it before a regulator asks. Book a call →
