Is AI a new research scientist? Human-led research not that

A comprehensive study examining the capabilities of artificial intelligence in academic research, researchers at the University of Florida found that AI can be valuable assistants, but not at the point of replacing human scientists in many important fields.

The study, detailed in the paper entitled “AI and the Emergence of The Cyborg Behavioral Scientist,” tested how well a common generative AI model, including Openai’s ChatGPT, Microsoft’s Copilot, and Google’s Gemini, can handle the stages of the research process. This study is published in the Journal of Consumer Psychology.

The team has led these AI systems to six stages of academic research. Start ideas, literature reviews, research design, record results, expand research and final manuscript production, and limit human intervention.

What they discovered was a mixed bag of ability and limitations. It was probably good news for research scientists wondering whether AI would do the job.

“The broad horror surrounding these AIS is the ability to deprive human labor,” explained Jeff Tomino, assistant professor of marketing at the University of Florida, Warrington University.

“We’ve seen that in general these AIs can provide some kind of support, but their value stops as support. These tools can make a ton of footnotes. However, researchers still have a significant place, act as AI directors and critics, and are not equal partners.”

Specifically, researchers have discovered that AI is a useful tool for idea processes and research design, including methods and stimulus design substances. However, during the literature review, results analysis and manuscript production stages, they struggled to generate valuable production volumes and required substantial monitoring.

Based on their findings, the University of Florida team advises researchers to maintain high skepticism about AI output, treating them as starting points that require human verification and improvement. For journals, researchers are highly encouraged to consider policies that call AI support for research papers and to largely prohibit the use of AI in the research review process.

Their work focused primarily on whether AI could do the job of academic researchers, but the team leading the study encourages researchers to reflect on whether AI should play their role.

“As researchers, we take great pride in the work we do,” Tomino said. “The specific steps that bring us joy (and anxiety) as researchers are likely to be as diverse as the research they are used. As these AI tools evolve, it’s up to the individual researchers to decide which steps in the research process they want to become cyborg behavior researchers and simply want to keep humans.”