Signal, Noise and Safeguarding; How to Operationalise Predictive CSAM Detection

TL;DR: Hash matching asks whether content has been seen before. Predictive CSAM detection asks how likely it is to be abuse, and lets you find material no list contains. That power comes with an operational decision every platform must own: where to set the action threshold. Here is how we set ours, and what we have learned.

As regulators move towards mandating proactive detection of child abuse material, platforms face an operational leap: from hash matching to predictive detection. These are fundamentally different capabilities.

Hash matching answers a yes-or-no question: has this piece of content been seen and catalogued before? A predictive model asks a different one: how likely is it that this content depicts child abuse risk? It transcends the ‘yes or no’ paradigm and fragility of exact hash matching and allows you to find newly generated and modified content; vastly extending the range of detection, and protection, but also, naturally, raising operational considerations of how to operate at scale and calibrate for confidence and noise. 

I have written about Athena and the case for proactive detection in previous pieces. Now, as we move into our first year of running Athena, and providing it to some of the world’s largest platforms, I want to share how we approach those operational choices: keeping human review manageable while maximising detection and the protection of children. 

Strong figures, from a different environment 

At the heart of Athena is the Vigil AI CAID Classifier, developed by Roke for national security and law enforcement. With the permission of the UK Home Office, it is trained on data from the UK’s Child Abuse Image Database, on material graded to the UK Sentencing Council’s severity categories and verified independently by multiple trained officers. Roke’s benchmarking puts precision (ability to detect whilst limiting false alarms) and recall (ability to find all matching content) in the mid-90s simultaneously.

In our next blog in this series, Roke will explain how they built the model and the ethical protections they deployed. Those standards, and the efficacy they achieved, are why we sought the partnership: to bring the detection available to policing into the online safety space and extend the safeguarding of children globally. 

But deploying outside of law enforcement meant understanding how accuracy figures established for digital forensic triage would translate to a world online where content is overwhelmingly benign, volume is enormous, and CSAM, whilst far too common in absolute terms, is vanishingly rare as a proportion of what a classifier sees. We knew the performance figures would look different in an online safety context, and we also knew that to operationalise anything at internet scale, the balance of signal to noise is critical.

Optimising that balance is something we work on every day, across every risk we cover. So we have spent the past months observing, analysing and working with our partners to build a view of Vigil AI as a child safety capability sitting inside a live Trust and Safety operation, and to reach our own optimum deployment.

Operational deployment

Athena is a service we provide to global online platforms, and one we use ourselves in our risk intelligence and child safety work. Like most platforms, we are limited in how we can test the model ourselves: because CSAM is illegal content, we are not permitted to store the true positives we would need to run through the classifier. So we use two complementary data sets. Partners help us run independent tests for recall (how much CSAM the model catches) and our own internet traffic tells us about noise (how often a benign image is flagged for human review).

This composite approach is a practical response to working with restricted data sets, and still gives us actionable information on performance. 

To answer the recall question, Roke, the owners of the model, worked with the Internet Watch Foundation, who are lawfully able to hold confirmed CSAM, for an independent check on how much of the content reported to them the classifier could detect. At an operating point of 0.80 (80% confidence score), the model identified 71% of the material the IWF holds. Each detection came from the image itself, with no hash lookup involved. That is far beyond what manual review could reach at this volume, and as our VP of Human Intelligence Jon Best will set out in a later post, an important move to safeguard our analysts.

That 71% is not in conflict with the mid-90s benchmark Roke measured in the law enforcement context. They measure different data at different operating points: the benchmark reflects forensic material, while 71% is recall on IWF material at a threshold that keeps noise manageable at our volumes. It is a setting, not the model’s ceiling.

Together these two views give us a signal-to-noise picture: how much is correctly detected at each confidence score, and how much review that score generates. To give a sense of the practicalities, in our last review period, working at a 0.8 (80%) probability threshold, we scored 2.8M images. Fewer than 1 in 2,000 (1,332) went to specialist review, so over 99.9% of images never needed a human to see them. Within that set we confirmed and reported 4 cases of CSAM that matched no hash and would otherwise have gone undetected. As with any detection system we can’t see everything we didn’t flag, which is why we triangulate with the independent IWF recall test above.

As we look to find the best operating point we can see that each of those items received a high confidence score, so, we can consider raising our action threshold to reduce noise, whilst still finding harm. These are the operational decisions we, and our partners are making quarter to quarter to find harm at scale. The appropriate threshold for the data we see and our volumes will not be the same for another platform, for different data, and different volumes. A

platform safeguarding hundreds of millions of images a day will feel noise acutely, while a low volume platform may be able to review more. Importantly, with this type of analysis, as is now the regulatory expectation (see Ofcom’s Risk Assessment Guidance June 2026), platforms can understand and evidence their risk, and they can explain how they have incorporated and calibrated predictive classification as part of their detection suite to proactively pinpoint and mitigate harm in that context.

A snapshot of our operational data: the view that helps us to set our action threshold, adjusting the balance of signal to noise to respond to natural changes in performance as internet traffic changes.

Combining signals for enhanced effectiveness

It never feels comfortable making a decision that reduces recall. Athena is not our only route, or any platforms’, to finding and mitigating harm to children, but image detection is a powerful, proactive tool and we want to maximise it however we can. So on top of the regular analysis of action thresholds, we have been working out how to reduce false positives without giving up any recall at all by understanding where the model’s edges are.

In our operational flows, we now label any suspected CSAM that turns out not to be CSAM to understand: what’s the nature of the image, and what are its technical characteristics? We have been running long enough to have built a statistically significant set, and the mis-classifications are not random. There are clear patterns, and patterns are something we can act upon.

• Technical properties. Some cluster around the image file itself: resolution, compression,
• aspect ratio. We can check these before the classifier is called, so the largest single driver of our false positives can be routed differently at no cost to detection.
• Content. Others cluster around the scene and setting. These are the model’s edges: the visual constructs where its confidence is least reliable, particularly as online content keeps evolving beyond the training set. They are exactly what targeted retraining can address, and we share these insights with Roke for that purpose (see below). 

This lets us add decisions to our action thresholds that take into account the wider risk context. The same score can be actioned differently in different situations. Consider a low confidence score, perhaps 0.7, for an image posted by an established account, on a well managed platform, to a public feed. We consider this differently to an image with the same score delivered in a higher-risk environment; maybe an account with no age-assurance, high message velocity to young users and repeated attempts to move conversations off-platform, the pattern we document in our Weaponised Loneliness briefing on Com networks. 

We can vary the operating point by the risk already attached to the user and the platform: being conservative where volume is enormous and the risk profile is low, and adopting a more assertive escalation posture where clear additional risk indicators exist. By combining signals in this way, we are finding we can lift precision substantially without raising the action threshold and compromising recall.

Signal sharing and consistent improvement

The thematic findings from our false positive analysis go back to Roke, the model’s owners, for targeted retraining, so the model edges we observe in live traffic inform the next version of the classifier. We offer our customers the option to share their insights too, so together we can improve the protections for children online. Roke’s access to CAID means they can act on this information; our exposure to platform traffic means we can tell them where to look. 

Each update improves detection online and strengthens the capability in the hands of law enforcement at the same time. The work is ongoing, and the figures here measure one version of a model, against one population of data at one moment. Our continued focus as partners is to keep observing, keep refining our action thresholds, and our understanding of performance drift to feed improvements and strengthen protections

Moving forward, effectively, transparently

We are often asked, ‘what’s the accuracy?’ and whilst we know it’s strong, there is no single figure that translates from one place to another. Regulators are reaching a similar view. In May 2026, Ofcom published advice on how technology for detecting child sexual abuse material should be assessed before it could be required of a service under the Online Safety Act.

They avoid setting a single precision or recall expectation and speak instead to judging the evidence and being transparent; reporting false positive rates and choosing methods appropriate to the context of the data on which the capability is used. An action threshold is a decision each platform makes, based on their data, their risk, and their capacity, rather than it being a fixed property of the capability.

Ultimately, regulators are not looking for perfection, they are looking for explanation about what has been implemented, how it is performing and how it is being improved. Athena enables platforms to discharge this obligation seamlessly.

Proactive, predictive CSAM detection is a meaningful addition to our operations, giving us new reach to safeguard children and users. With the right analysis and implementation, such models offer a highly effective way to make real progress against material that was until very recently invisible, and that behind those numbers are children who would not otherwise have been found. 

For a more detailed explanation of how defensible predictive CSAM detection is implemented in practice, see the Athena service overview.

About the author: Frances McAuley is Director of Product in Resolver’s Trust & Safety Division, where she leads the development of intelligence‑led capabilities supporting regulated platforms across online safety, compliance, and enforcement.