OpenAI Launches New Image Classifier to Combat CSAM on ChatGPT

OpenAI Launches New Image Classifier to Combat CSAM on ChatGPT

New AI Image Classifier Can Detect Previously Unseen Abuse

One of the central features is OpenAI’s new CSAM Image Classifier, designed to identify images that may contain child sexual abuse material, including content created or modified using generative AI.

Unlike systems that primarily identify previously known images through matching techniques, the new classifier analyses visual signals to help identify potentially new or previously unseen material.

It generates risk scores that help trained reviewers prioritise images requiring further investigation.

OpenAI says the classifier was developed with assistance from NCMEC, which operates the CyberTipline for reporting online child exploitation.

The tool is being offered at no cost through the OpenAI API to vetted organisations, including eligible child-safety groups and technology partners. It is not a general-purpose image checker available to every ChatGPT user. <Cite refs={[“turn668395view0″,”turn668395view1”]}/>

Early Testing Shows Reduced Review Work

OpenAI has shared encouraging results from its early evaluations.

According to the company’s testing:

  • The classifier could flag 48% fewer images while identifying roughly the same number of abusive images as its existing tool.
  • Images flagged by the new classifier were approximately twice as likely to contain abuse.
  • In testing by one early partner, 72% of high-confidence flags were confirmed by human reviewers as previously unseen CSAM.
  • Another partner reported that one in four flagged cases resulted in a ban.

These results suggest that the system could help safety teams spend more time reviewing potentially harmful material instead of investigating large numbers of false positives.

However, OpenAI emphasises that the classifier produces risk signals, not definitive legal or factual conclusions. Human review remains important. <Cite refs={[“turn668395view0″,”turn668395view1”]}/>

ChatGPT Will Show Warning Screens for Harmful Requests

OpenAI is also introducing new warning screens designed to interrupt attempts to find or generate content that sexualises children.

When ChatGPT detects such a request, it can:

  • Display a warning explaining the harm associated with sexualising children, including through AI-generated imagery.
  • Stop the conversation and remove it from the user’s accessible chat history.
  • Temporarily suspend the user’s ChatGPT access for 24 hours.
  • Direct the user towards specialist support resources intended to help stop harmful behaviour.

Repeated violations may result in longer restrictions, permanent account closure or reports to authorities, where appropriate.

The approach was developed with input from organisations including Protect Children and the Lucy Faithfull Foundation, along with academic researchers. <Cite refs={[“turn668395view0″,”turn668395view2”]}/>

New Child-Safety Guidance for Developers

OpenAI has also published practical guidance for developers building applications with its AI tools.

The guidance covers several important areas:

  • Preventing the creation and distribution of abusive content.
  • Detecting potentially harmful images and other material.
  • Providing reporting options for users.
  • Reviewing suspected abuse through appropriate safety processes.
  • Identifying repeat offenders and responding to misuse.
  • Connecting with established child-protection reporting systems.

Developers can combine the new classifier with existing moderation systems, known-image matching tools and trained human review.

OpenAI says the guidance was developed with input from organisations including the Tech Coalition, Thorn, NCMEC and the Internet Watch Foundation. <Cite refs={[“turn668395view0″,”turn146375search9”]}/>

AI-Powered Tools for Law Enforcement

The announcement also includes additional support for specialist law-enforcement teams investigating child exploitation and abuse.

OpenAI plans to provide authorised agencies with AI-powered investigative tools, dedicated plugins and training.

The initial tool is designed to help investigators:

  • Review lengthy chat records and legal-process returns.
  • Identify possible connections between individuals and information.
  • Organise large volumes of investigative material.
  • Create timelines linked to the original source material.

Access will depend on agency authorisation and programme requirements. The tools are intended to support investigators rather than replace professional judgement or established investigative procedures. <Cite refs={[“turn668395view0”]}/>

Why AI-Generated Abuse Is a Growing Safety Challenge

Generative AI has made it easier to create realistic images and other digital content. While these technologies have legitimate uses, they can also be misused to create fabricated or manipulated imagery that sexualises children.

Such material can cause real harm even when it is artificially generated and does not depict an identifiable real child.

Traditional image-matching systems may struggle with newly created or substantially altered images because they have no previously known version to match.

This is why OpenAI is combining several approaches, including image classification, existing-content matching, behavioural monitoring, moderation systems and human review.

The broader objective is to make it harder to create, distribute and access abusive material through AI platforms. <Cite refs={[“turn668395view1″,”turn668395view0”]}/>

What Does This Mean for ChatGPT Users?

For ordinary users, the announcement means stronger safeguards against attempts to use ChatGPT for child exploitation.

OpenAI’s Help Center says these warning and suspension interventions are gradually rolling out worldwide. They apply to personal ChatGPT accounts using Chat and Work modes on the web and mobile, including certain unsigned-in conversations.

The Help Center currently excludes Business, Enterprise, Edu and Healthcare workspaces, as well as voice mode, Codex and ChatGPT desktop apps from these specific interventions. Availability may vary by country.

Users who believe a suspension was issued incorrectly can follow the appeal link in their suspension notification email. <Cite refs={[“turn668395view2”]}/>

A Wider Push for Online Child Protection

OpenAI’s latest announcement highlights the importance of cooperation between technology companies, child-protection organisations, developers and law-enforcement agencies.

The new image classifier, warning screens and investigative resources are intended to strengthen different stages of the response, from detecting potentially abusive material to preventing harmful behaviour and supporting investigations.

As AI-generated content becomes more sophisticated, technology companies will face increasing pressure to ensure that their products cannot easily be exploited to harm children.

The effectiveness of these measures will depend on continued testing, responsible deployment, accurate detection and cooperation across the wider online safety ecosystem.

0
Would love your thoughts, please comment.x
()
x