AI Stealing Work From Scientists

PopSci, AI-generated nonsense is leaking into scientific journals, displacing human-generated nonsense.

In February, an absurd, AI-generated rat penis somehow snuck its way into a since retracted Frontiers in Cell and Developmental Biology article. Now that odd travesty seems like it may just be a particularly loud example of a more persistent problem brewing in scientific literature. Journals are currently at a crossroads on how best to respond to researchers using popular but factually questionable generative AI tools to help draft manuscripts or produce images. Detecting evidence of AI use isn’t always easy, but a new report from 404 Media this week shows what appears to be dozens of partially AI-generated published articles hiding in plain sight. The dead give away? Commonly uttered, computer generated jargon.

404 Media searched the AI-generated phrase “As of my last knowledge update” into Google Scholar’s public database and reportedly found 115 different articles that appeared to have relied on copy and pasted AI model outputs. That string of words are one of many turns of phrase often churned out by large language models like OpenAI’s ChatGPT. In this case, the “knowledge update” refers to the period when a model’s reference data was updated. Chat. Other common generative-AI phrases include “As an AI language model” and “regenerate response.” Outside of academic literature, these AI artifacts have appeared scattered in Amazon product reviews, and across social media platforms.

AI generated scientist

Several of the papers cited by 404 Media appeared to copy the AI text directly into peer-reviewed papers purporting to explain complex research topics like quantum entanglement and the performance of lithium metal batteries. Other examples of journal articles appearing to include the common generative AI phrase “I don’t have access to real-time data” were also shared on X, formerly Twitter, over the weekend. At least some of the examples reviewed by PopSci did appear to be in relation to research into AI models. The AI utterances, in other words, were part of the subject material in those instances.

Though several of these phrases appeared in reputable, well-known journals, 404 Media claims the majority of the examples it found stemmed from small, so-called “paper mills” that specialize in rapidly publishing papers, often for a fee and without scientific scrutiny or scrupulous peer review.. Researchers have claimed the proliferation of these paper mills has contributed to an increase in bogus or plagiarized academic findings in recent years.

Having been on both sides of the peer review process, both author and reviewer, I have to admit it would be pretty easy to sneak AI generated text into a paper, and goodness knows it might help improve the readability of a lot of the stuff. If the scientist who used the AI text actually checked to make sure it got its facts right (given AI's propensity to hallucinate facts), I don't see a problem with it. However, leaving in the characteristic AI generated text suggest it often isn't. The article also highlighted a point I hadn't considered:

Those who support permitting AI-generated text in some instances say it can help non-native speakers express themselves more clearly and potentially lower language barriers. Others argue the tools, if used responsibly, could speed up publication times and increase overall efficiency. 

I had occasion to review several papers from non-native speakers, and it was as painful chore. I tried very hard to overlook, and even fix the glitches in the language, and focus on the science.

Still, the best use for AI is pictures.  

The Wombat has Rule 5 Sunday: Blast From The Past up and ready for your digital pleasure.