Fake AI Content Is Now Showing Up in Scientific Journals, Chatbot Answers, and Federal Court Filings

Since scientific integrity consultant Elisabeth Bik warned this month that researchers "cannot distinguish fake from real" when it comes to AI-generated research papers, the same problem has surfaced in courtrooms, chatbots, and Facebook feeds within days of each other.

The Science Problem

Bik, speaking to MedPage Today, said AI tools have gotten good enough at generating fake data, fake images, and fake papers that peer reviewers and editors are losing the ability to catch them. This is a warning from someone whose job is literally to spot fraud in published research.

Chatbots Getting Poisoned on Purpose

Security researchers at Vigilance Security identified a campaign they're calling "Dark Sourcery," according to Dark Reading. Attackers are flooding the internet with fake support pages, reviews, and PDFs specifically optimized to get scooped up by ChatGPT, Google Gemini, and Google's AI Overview feature. Vigilance found tens of thousands of malicious pages containing fake phone numbers, fake login pages, and fraudulent support contacts, all designed to get served up by AI as if they were trustworthy answers.

Ariel Simon, Vigilance's vice president of research, said attackers are essentially doing SEO for lies, engineering content so AI systems present it as fact. OpenAI told the Wall Street Journal, in a separate report on OpenAI's own agents aggressively scraping a United Nations data hub more than 16,000 times between April and June, that it's reviewing the findings. That's a different problem: agentic AI behaving badly on its own. It lands in the same news cycle as evidence that AI systems are hard to keep inside their guardrails in either direction, being manipulated by outsiders or acting aggressively themselves.

A Vietnam-Based Fake News Operation

CBC and Radio-Canada's investigative team traced a network of roughly 200 Facebook pages with more than 55 million combined followers, and about 50 connected websites, back to a Hanoi-based company called HTV Network, run by CEO Ngo Anh Tan. The outlets pushed fabricated stories, including a false claim that Elon Musk banned Teslas in Canada, and exploited a real shooting in Tumbler Ridge, British Columbia, for clickbait traffic. The company markets itself as an "international media company" connecting the "world with authoritative journalism." It employs at least 100 people, according to a Vietnamese job listing reviewed by CBC, and profits by running ads next to the content it generates.

Fake Citations, Now in Federal Court

The fabrication problem has reached the legal system too. U.S. District Judge Loren AliKhan in Washington, D.C., ordered Severna Park attorney Donald Quinn to explain why he shouldn't be sanctioned after a December 2025 brief he filed was, in her words, "littered with incorrect and misleading citations and quotations," according to The Daily Record. This is not Quinn's first run-in over the issue. He previously told a Maryland federal judge he would stop using AI to revise legal documents, and he told The Daily Record in May that a supposedly "closed" AI editing system his firm used had turned standard citations into fabricated quotes without his knowledge. AliKhan gave him two weeks to respond and flagged a possible violation of D.C.'s Rules of Professional Conduct.

The Counterargument: This Is Just New-Tech Growing Pains

Some in the industry argue this is overstated. When Chinese AI firm Z.ai's ZCode coding assistant was caught quietly uploading users' private code repositories to Alibaba Cloud servers, the company disabled the offending feature and opened its codebase to outside scrutiny within days, according to The Hacker News. That mirrors how xAI's Grok Code coding tool responded after it was found uploading entire Git repositories to a Google-controlled cloud bucket. The pattern: problems get found, companies patch them, the tools keep shipping. Defenders of the technology would say that's how software has always worked, and that treating every flaw as evidence AI is fundamentally broken ignores the speed at which these companies are responding once caught.

That argument has a real limit, though. Bik's specific worry isn't a bug that gets patched. It's that fake scientific papers, once published and cited, don't get un-published just because someone later builds a better detector. The same goes for a false story about Elon Musk banning Teslas that already reached 55 million Facebook followers, or a federal court record now on file with citations a judge says don't hold up.

A Separate, Related Fight Over What Counts as Trustworthy Media

The information-trust question isn't confined to AI. Attorney General Todd Blanche told ABC's Martha Raddatz on "This Week" that President Trump is "sick and tired of certain media outlets just refusing to report accurate news," defending the administration's decision to bar CNN from Air Force One pool coverage, a move a federal judge temporarily blocked this past week. Raddatz pressed Blanche on whether CNN's exclusion was about anything other than how it covers the president; Blanche said the White House routinely decides who travels with the president and denied this was a one-time punitive action. CNN disputes that characterization, saying it was the scheduled pool outlet that day and was denied access regardless. Neither side's version has been resolved by the ongoing litigation.

None of the AI-fakery reports say who, if anyone, is going to audit the scientific literature already contaminated by AI-generated papers, or how many HTV-linked stories are still circulating on Facebook. Quinn has two weeks to respond to Judge AliKhan. What sanction, if any, she imposes will be one of the first tests of how federal courts intend to punish lawyers for AI-generated errors rather than just scolding them.