Large language models (LLMs) trained on huge amounts of text data recognize patterns in language and predict new information. A new study from researchers at University College London (UCL) shows that LLMs are better at predicting the outcomes of neuroscience experiments than human experts.
The researchers describe the methods and results of this study in a paper published in Nature Human Behaviour.
The researchers created a tool called BrainBench to test this ability. BrainBench involves showing both real and fake versions of neuroscience research summaries to LLMs and human experts. The real summaries come from actual studies, detailing the research background, methods, and results. The fake summaries keep the background and methods the same but change the results to something that could be believable but wrong.
Here's what the researchers found: When asked to pick which summary was real, the LLMs got it right about 81% of the time, while human experts only managed about 63%. Even when only the most knowledgeable scientists were considered, their accuracy was still lower at 66% compared to the LLMs. Plus, the LLMs were more accurate when they felt more certain about their choices, measured by a perplexity score that shows how surprising the text is to the model.
The researchers went a step further by customizing an LLM, which they called BrainGPT, specifically for neuroscience. This version did even better, achieving an accuracy of 86% compared to the general LLM's 83%.
Why does this work?
This finding suggests that LLMs could be very useful for science. Instead of just looking up known facts, these models can predict new research outcomes by recognizing patterns in existing scientific literature. This could speed up scientific discovery by helping researchers design experiments that are more likely to yield meaningful results.
Researchers Ken Luo explains that this is really about using AI to predict future scientific developments. This could change how experiments are planned, making them more efficient and insightful. Research leader Bradley Love notes that this also raises questions about how novel scientific research really is if AI can predict outcomes so well, suggesting that perhaps scientists need to be more innovative.