OpenAI Warns Courts May Have No Reliable Way to Spot AI Deepfake Evidence

AI-generated deepfakes could make it harder for US courts to establish whether audio and video evidence is genuine, creating a growing challenge for judges who must decide what material can be trusted in a trial.

Representatives from OpenAI and Meta raised the concern at a 15 October meeting of the US Judicial Conference's Advisory Committee on Evidence Rules in Boston. Their warning comes as the judiciary considers whether existing authentication standards can cope with increasingly convincing fabricated recordings.

However, the immediate scale of the problem remains unclear. A Federal Judicial Center survey found that only 15 of the 931 judges who responded had encountered a case in which a party challenged evidence on the grounds that it was a deepfake.

That gap between predicted risks and reported cases puts the judiciary in a difficult position: changing evidence rules too soon could create unnecessary burdens, while waiting too long could leave courts struggling to assess sophisticated fabrications.

Why AI Deepfakes Are Becoming Harder to Authenticate

OpenAI policy counsel Shira Anderson warned the committee that advances in generative AI could undermine efforts to identify fabricated images. 'AI images are only getting better, and I do not think that there will be a scientific way going forward to detect them,' she said.

Devin Norder, Meta's director of product management, also told the panel that media produced by recent advanced AI models could be indistinguishable to humans from authentic material. He said creating such content had become relatively inexpensive and quick.

The challenge extends beyond spotting visual glitches or unnatural speech. A fabricated recording may appear credible even when it misrepresents a conversation or event. Lawyers may need to establish where a file came from, how it was produced, and whether it has been altered before a court can assess its authenticity.

Watermarking and other verification tools may help, but they do not offer a universal solution. Anderson said companies use different systems, making it difficult to reliably detect every watermark or verify all visual content. She also cautioned that such measures would not be 'bulletproof'.

For judges, the practical concern is that a recording appearing genuine is not necessarily authentic. Equally, uncertainty about a file does not automatically prove it has been manipulated.

Federal Judges Weigh Changes to Evidence Rules

The committee has been examining deepfake authentication since 2023. Its October agenda says members have prepared, but not finally approved, a proposed amendment known as Rule 901(c). Under the proposal, a party challenging evidence as a deepfake would first need to provide enough evidence to support that claim. If that threshold were met, the party seeking to introduce the material would then have to show that it was more likely than not authentic.

The proposal remains under consideration and is not an adopted change to federal evidence rules. US District Judge Jesse Furman, who chairs the committee, acknowledged that predictions about deepfakes in court had not yet been matched by a large number of reported disputes.

He appointed a subcommittee to examine the issue further. Judge Valerie Caproni, another committee member, cautioned against rushing into an AI-specific rule without stronger evidence that deepfakes were becoming a widespread courtroom problem.

Existing Safeguards Face a Test as AI Advances

The debate is not about whether courts currently have any way to challenge evidence. Federal Rule of Evidence 901 already requires parties to provide sufficient support for a finding that evidence is what they claim it to be.

That framework allows judges to consider challenges to recordings and other material. The unresolved question is whether traditional authentication procedures will remain effective when AI-generated fabrications become more convincing and easier to produce.

Caproni argued that courts have long dealt with people presenting false evidence, and that existing laws already prohibit submitting fraudulent material. Her position highlights the risk of creating new procedures before the scale of the problem is clear.

Yet the low number of reported challenges does not establish that the risk is negligible. It may reflect the current stage of the technology, the difficulty of recognising manipulated evidence, or the limited number of cases in which authenticity has been formally disputed.

The committee must therefore balance two competing concerns: protecting legitimate evidence from unnecessary challenges while ensuring that fabricated material cannot gain credibility simply because it looks or sounds real. For now, the proposed rule remains under review.