The UK's research funding landscape is evolving, and with it, a bold experiment is underway. UK Research and Innovation (UKRI), the nation's primary funding body, is exploring the potential of AI to revolutionize grant peer review.
With an annual research funding allocation of over £8 billion, UKRI faces a unique challenge. Despite a decrease in funded research grants over the past seven years, applications have surged by a staggering 80%. This has led the agency to seek innovative solutions to streamline its peer review process.
Enter Mike Thelwall, a data scientist from the University of Sheffield, leading a research team funded by the UK Metascience Unit. Their mission? To investigate the role of generative AI in grant peer review. Starting in October, the team will delve into the data underlying up to 2000 grant proposals, aiming to determine if AI can predict peer reviewer scores and recommendations accurately.
But here's where it gets controversial... Thelwall's team will have access to confidential grant proposals, both funded and rejected, to train large language models (LLMs). The goal is to see if these models can predict the scores and decisions made by human reviewers. While the team will know the outcomes, they won't disclose them to the LLMs, hoping to find a way to speed up the grant review process.
Thelwall has prior experience in this field, having worked on a project exploring AI's role in refereeing articles for the UK's Research Excellence Framework. In December 2022, his team's data suggested that AI systems generated identical scores to human reviewers 72% of the time, falling short of the 95% accuracy benchmark Thelwall believes is necessary.
And this is the part most people miss... Mohammad Hosseini, an AI ethics researcher at Northwestern University, raises valid concerns. He questions whether LLMs can create novel ideas, suggesting they may struggle to detect truly creative concepts due to their training on existing data. In grant proposals, ideas are shared with potential, unlike manuscripts that report past events.
Another challenge arises when funders use LLMs without transparency. Researchers may rebel if they don't know the criteria used to feed the AI. However, if funders are open about the process, grant applicants might manipulate their writing to please the AI, leading to a different set of issues.
So, how might UKRI implement generative AI? Thelwall suggests it could be used in tiebreaker situations or as an additional reviewer. It could also assist in a fast-track desk-reject option, reducing the burden on human experts. Thelwall cites the example of the la Caixa Foundation in Barcelona, where AI-assisted grant peer review is being piloted. While it saves some reviewer time, around 90% of applications still go through full peer review with three experts.
The future of grant peer review is uncertain, but one thing is clear: the potential for AI to transform this process is immense. With further research and development, AI could revolutionize how we evaluate and fund research, but it's a delicate balance. As we navigate this exciting yet uncharted territory, what are your thoughts? Do you think AI can truly revolutionize grant peer review, or are there ethical and practical hurdles that might hinder its progress? We'd love to hear your opinions in the comments below!
