Stop Making People Look First; the Human Cost of CSAM Moderation
I spent seventeen years in child safeguarding before I came to Resolver, and there’s a problem that has always stuck with me: someone always has to look first. Before a hash gets added to a database, before an offender gets identified, before a single victim gets safeguarded, a person sits down and sees something they can never unsee. I did it. I asked analysts to do it. Most of my career since has been spent trying to minimise the impact of it.
CSAM has never been a solved problem. It has only ever been a measured one. In 2025, the Internet Watch Foundation confirmed 311,610 criminal abuse reports. NCMEC processed 21.3 million reports containing 61.8 million individual files. Year after year those numbers climb, no matter what technology, legislation or public pressure we throw at them.
Then generative AI arrived, turning an exponential rise into a vertical wall.
NCMEC’s AI-linked reports jumped from 4,700 in 2023 to 1.5 million in 2025. A model fine-tuned on twenty photos of a real child can now produce a convincing new image of that child in fifteen minutes.
None of that material exists in any database. None of it existed twelve months ago. That is why legacy defences are failing.
The defence that requires a victim
Hash matching works by fingerprinting content a human has already inspected and confirmed as abuse. Every hash in the NCMEC or IWF database exists because a person looked at that exact file first, decided it was CSAM, and logged it so the next copy could be caught automatically. I watched the UK’s Child Abuse Image Database do exactly that, and it was a genuine breakthrough. It just has one flaw baked into its design: it can only ever know what it has already seen.
Everything new slides straight past it, and thanks to synthetic generation, “new” is now most of what arrives. INHOPE’s hotlines saw suspected CSAM records jump 450% in a single year, from 1.05 million to 4.78 million, with synthetic images topping the list. What’s caught is the tip of the iceberg. What’s underneath it is everything else, and it’s growing faster than any team of humans can look at.
The human and financial bill
That leaves one thing standing between the unknown and the known: a person, looking at each file, deciding what it is before anyone can act on it. I know what that costs, because I’ve watched it happen to good people. In one court-ordered assessment of 144 content moderators, medical professionals diagnosed 81% with severe PTSD. That’s not a statistic to me. That’s committed people on the front line putting themselves in harm’s way.
It costs the victims too, every single time. Survivors tell us the ongoing spread of their abuse material is worse than the abuse itself. Three-quarters live in constant fear of being recognised. 95% of those who are recognised suffer further harassment as a direct result. Every manual review is one more person opening that file. One more moment where the worst day of someone’s life gets relived by a stranger, and by the analyst forced to confirm what it is. There is no version of “toughen up our people” that makes this acceptable. We shouldn’t be building resilience to something we can prevent.
For Trust & Safety leaders, this human reality is closely tied to operational sustainability too. One major platform paid $52 million to settle claims from 10,000 moderators over inadequate exposure protections. The International Labour Organisation now recognises PTSD as an occupational disease. As regulatory frameworks and occupational health standards evolve, relying on manual review as a primary filter creates unsustainable operational risk. Modern Duty of Care requires platforms to minimise human exposure wherever technology can safely step in.
The solution: automated pre-exposure triage
Wellness programs, counselling, and rotation schedules are vital support mechanisms, but they do not fix an underlying architectural bottleneck. True protection comes from stopping exposure before it happens.
Athena changes the system design entirely.
Trained on law-enforcement data from the UK’s Child Abuse Image Database using Roke’s Vigil AI Classifier, Resolver’s CSAM detection service, Athena, analyses visual characteristics at the point of upload. In under 150 milliseconds, Athena flags illegal content, known or unknown, real or synthetic, before it reaches a human queue, operating under a strict zero-image-retention policy that purges media immediately after analysis. Human review doesn’t disappear. It just stops being the front line. Analysts move from finding out what a file is to confirming it, on the small number of cases that still genuinely need a person’s judgement.
I didn’t get into this work to protect systems. I got into it to protect children, and the people we ask to stand between them and the worst of humanity. Success in this fight shouldn’t be measured by how well we help someone recover from what they’ve seen. It should be measured by how rarely we ever ask them to look.
Close your platform’s pre-exposure gap
Manual review shouldn’t be your platform’s first line of defence. Discover how Resolver’s Athena clears moderation bottlenecks and protects your operational team.
For a more detailed explanation of how defensible CSAM detection is implemented in practice, see the Athena service overview.
About the author: Jon Best is Vice President of Human Intelligence at Resolver, formerly Crisp. Before joining Resolver, he spent over 16 years with West Yorkshire Police, specialising in child safeguarding, intelligence, and complex investigations. Jon now leads efforts to advance technology and human expertise to better protect victims and prevent exploitation in the digital age.
