Reviewing the scientific literature is an important part of advancing a research field. A review of the scientific literature provides the current state of the coalition through a comprehensive analysis of existing research and identifies knowledge gaps on which future research may focus. But writing a good review article is great.
Researchers comb through many academic papers. Researchers should select studies that are not outdated, avoiding recency bias. Then comes the intensive work of assessing research quality, extracting relevant data from successful studies, analyzing data to glean insights, and writing a compelling story that summarizes the past while looking to the future. Continue. Research synthesis is a research field in itself, and even good scientists may not be able to write a good literature review.
Enter artificial intelligence. As in many industries, a number of startups are emerging to use AI to speed up, simplify, and revolutionize the scientific literature review process. Many of these startups are positioning themselves as academic research-focused AI search engines, each with differentiated product features and target audiences.
Elicit encourages search users to “analyze research papers at superhuman speed” and highlights Elicit’s use by professional researchers at institutions such as Google, NASA, and the World Bank. Scite says it has built the largest citation database by continuously monitoring 200 million scholarly sources and offers “smart citations” that categorize points into supporting and controlling evidence. Consensus features a homepage demo that appears to be aimed at helping laypeople gain a more solid understanding of specific questions, describing the product as “Google Scholar meets ChatGPT” and summarizing key points. We offer a consensus meter. These are just a few of many.
But can AI replace high-quality systematic scientific literature reviews?
Research synthesis experts now tend to agree that these AI models excel at performing qualitative analysis, or creating narrative summaries of scientific literature. What’s not so good is the more complex quantitative layer to make the review truly systematic. This quantitative synthesis typically involves statistical methods such as meta-analysis, which analyzes numerical data across multiple studies to draw more robust conclusions.
“AI models are nearly 100 percent as good as humans at summarizing key points and writing fluid arguments,” said Co-Founder of the American Research Association Synthetic and Integrated Methods Center (MOSAIC) , says Joshua Polanin. “But we’re still less than 20% ahead in terms of quantitative synthesis,” he says. “A real meta-analysis follows a rigorous process in how it searches for studies and quantifies the results. These numbers form the basis for evidence-based conclusions. By the time AI can do that, has not yet been achieved.”
Quantification problems
Polanin explains that the process of quantification can be difficult even for trained professionals. Both humans and AI are usually able to read research and summarize the main points. That is, study A found an effect, or study B found no effect. The difficult part is setting a numerical value for the range of effect. Additionally, there are often different ways to measure effectiveness, and researchers need to identify study and measurement designs that align with the assumptions of their research questions.
Polanin said the model must first identify and extract relevant data and then provide subtle instructions on how to compare and analyze it. “Even human experts try to make decisions in advance, but sometimes they end up having to change their minds on the fly,” he says. “That’s not what computers are good at.”
Given the arrogance seen around AI and within startup culture, one might expect the companies building these AI models to protest Polanin’s assessment. But you won’t get the following counterargument from Consensus co-founder Eric Olson: “Honestly, I couldn’t agree more,” he says.
Consensus, Polanin noted, is intentionally “at a higher level than other tools, giving people the foundational knowledge for quick insights,” Olson added. He considers the typical user to be a graduate student, someone with an intermediate knowledge base and striving to become an expert. Consensus may be just one tool among many for true experts in the field, or it may be a tool for science, such as the European consensus users who keep abreast of research on rare genetic diseases in children. It can also be useful for non-users to stay informed. “He had spent hundreds of hours on Google Scholar as a non-researcher. He told us he had been dreaming of something like this for 10 years, and it changed his life. . I use it every day now,” says Olson.
The team at Elicit targets a different type of ideal customer. “Someone working in industry in a research and development context, perhaps in a biomedical company, trying to decide whether to move forward with the development of a new medical intervention,” says James Brady. , Head of Engineering.
With these high-stakes users in mind, Elicit clearly shows users’ causal claims and the evidence to support them. This tool breaks down the complex task of literature review into manageable parts that humans can understand, and also provides more transparency than the average chatbot. Researchers can see how the AI model arrived at the answer and check it against the source.
The future of scientific review tools
Brady agrees that current AI models do not provide a complete Cochrane-style systematic review, but says this is not a fundamental technical limitation. Rather, it’s a matter of future advancements in AI and better and faster engineering. “In principle, I don’t think there’s anything our brains can do that computers can’t do,” Brady says. “That also applies to the systematic review process.”
Roman Lukyanenko, a professor at the University of Virginia who specializes in research methods, agrees that developing methods that support an early and rapid process of gathering better answers should be a major focus going forward. . He also points out that while the current model tends to prioritize freely accessible journal articles, there is a lot of high-quality research behind paywalls. Still, he’s bullish about the future.
“We believe that AI is revolutionary in this field on so many levels,” Lukyanenko said. He co-authored a pre-ChatGPT 2022 study on AI and literature review with Gerrit Wagner and Guy Paré, which went viral. “We all have an avalanche of information, but human biology limits what we can do with it. These tools have great potential.”
Scientific advances often come from interdisciplinary approaches, he says, and this is where AI’s potential may be greatest. “There’s a term called ‘Renaissance Man,’ and I like to think of ‘Renaissance AI,’ which is something that can access most of our knowledge and make connections,” Lukyanenko said. says. “We should be committed to making serendipitous, unexpected, and far-flung discoveries across disciplines.”
From an article on your site
Related articles on the web
