Exploring the potential benefits and challenges of artificial intelligence for research funding organisations: a scoping review

Eligibility criteria

There is no universally accepted definition of ‘artificial intelligence’, so for the purpose of this review it was important to clarify what we meant by ‘AI’. Below are some previous explanations of AI that provided context for the review, including how we developed the eligibility criteria.

Context

The context included UK, European and international settings (e.g., sectors, fields or disciplines) (based on the inclusion criteria) and were included to gain as much information as possible on potentially relevant AI applications.

Participants

Research funding and performing organisations, private sector business and enterprise, and public administrations were included as part of the AI application context.

Inclusion criteria

Articles were included if they were written in the English language (or could be transcribed to English), available in full text, and focused on the application and implementation of AI (rather than the development of algorithms). To be relevant to RFOs, these AI applications had to apply primarily to text data¹⁶ (rather than images) used for/in organisational processes in an institutional, enterprise or business context (comprising specific human tasks and operational processes). We defined ‘organisational processes’ to include (but not be limited to): research management and administration; business intelligence; compliance and risk management; data management and analytics; and decision-making.

With regards to capturing relevant AI solutions:

Exclusion criteria

We excluded any articles that were retracted or provided limited information on AI applications, as defined in the inclusion criteria. News announcements related to company partnerships or the marketing of new products were excluded, as well as articles on AI applications not relevant to an organisational context (e.g., military or medical AI applications). The following is a non-exhaustive list of AI applications that were excluded from the scoping review: Drone technology; 3D printing; environmental and agricultural (e.g., fire safety, water reserves, farming, drainage, space); satellite data and weather monitoring; facial and voice recognition; metaverse and mobile networking (5G and 6G); estimation and stimulation modelling (e.g., urban flood, mining, energy grids); medical education; healthcare intervention; medical devices and diagnostics.

Source types

All article types and study designs were included in the review to maximise coverage due to the diverse nature of the use and adoption of AI applications across different settings and sectors (which could be relevant to RFOs). We therefore included:

Academic type articles: Peer reviewed journal articles; commentaries; editorials; opinion letters; perspectives; and preprints

Grey literature: Policy documents; reports; blogs; educational articles; guidance; and newsletters (e.g., AI News, AI Magazine).

Study designs: Randomised controlled trials; non-randomised controlled trials; before and after studies and interrupted time-series studies; analytical observational studies (including prospective and retrospective cohort studies); case-control studies; analytical cross-sectional studies; descriptive observational study designs (including case series); individual case reports; and descriptive cross-sectional studies. Qualitative studies were also considered that focused on qualitative data including, but not limited to, phenomenology, grounded theory, ethnography, qualitative description, action research and feminist research. In addition, systematic reviews that met the inclusion criteria were considered, depending on the research question.

Search strategy

Searches were conducted on 1 February 2024, using Web of Science and ProQuest databases to search for published journal articles, and Overton and Google Scholar were used to search for grey literature and government or RFO documents. The search strategy was iteratively developed and adapted to each database, with keywords and search terms identified from pilot searches in Web of Science, and manual searches in Connected Papers (https://www.connectedpapers.com/), to inform the final search strategy. The final search strategy was reviewed and approved by a librarian information specialist at the University of Southampton.

Searches were not limited by study type or design, source of publication, country, or language of publication. Initially, no limits were placed on year of publication; however, during initial screening of the title and abstract, it was noted that those published before 2022 were consistently out of scope and did not meet inclusion criteria for the current review (e.g., articles were too conceptual or focused on algorithm development only). Subsequent searches were limited to articles published in the last three years only (2022-2024) and this was reflected in the search strategy and screening process in accordance with PRISMA (https://www.prisma-statement.org). Articles deemed relevant by title and abstract were then screened by reading the full texts of the articles.

Manual searches were also conducted on 13 websites of research funding and professional organisations (e.g., research councils, charities and professional/regulatory bodies) to identify announcements, funding opportunities, or other activities relevant to AI. This was done by typing the name of each organisation into the Google search engine and navigating to relevant webpages. The organisations were chosen by purposive sampling, following discussions with National Institute for Health and Care Research (NIHR) Coordinating Centre staff, and aimed to be representative of positions, practices and developments in big data and AI across the research funding landscape.

Data extraction

The articles identified through the databases and manual searches were exported with citation, titles, and abstracts into Endnote 20 (Clarivate, UK). Duplicates were removed using the EndNote duplication function and then manually checked for consistency. To test eligibility criteria and ascertain agreement, 20 articles were assessed in a pilot screen by two authors (ABJ and KC). All remaining articles were screened by title and abstract against the inclusion criteria. Relevant articles were then checked for availability of full texts and the full texts were then screened by both authors to check for eligibility. Any disagreements between authors regarding the decision to include or exclude were resolved through discussion until consensus was reached.

After the final screening, ABJ and KC extracted data from the article’s full texts into a Microsoft Excel data extraction spreadsheet developed by the authors specifically for this review and was piloted to make sure the relevant information was extracted to address the three research questions and the classification framework. The spreadsheet included the following sections for extraction:

Figure 1. Classification framework: Utility, potential, considerations and risks of AI for research funding organisations.

Where applicable, further exclusions were made during the full data extraction stage, for example, where the same article was included multiple times due to different source types (i.e., duplication of information). In such instances, a peer reviewed journal article took precedence (and was included) over a preprint manuscript and a conference proceeding, and a preprint manuscript took precedence over a conference proceeding. For peer reviewed articles, if the same article was published in two journals, only one was included. Any articles written in languages other than English, or lacking an English translation, were excluded from full extraction.

All extracted data were used to inform summary tables characterising the included articles and drawing out the key concepts to address the three research questions (refer to extended data for Tables 2,3, and 4). As this was a scoping review, we did not conduct a risk of bias or quality assessment, and all evidence was mapped or categorised to either pre-defined terms and concepts or to those developed and agreed by the authors during the data extraction stage. The results of the searches and study inclusion process are reported in full, in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-analysis extension for scoping reviews (PRISMA-ScR) (see Figure 2).

Figure 2. PRISMA flow diagram of the results of the searches.

Extraction from the research funding or professional organisations was extracted in a separate Microsoft Excel spreadsheet containing: the RFO or professional body name; links to the relevant webpages; the date the information was accessed; and a summary of the relevant AI information (e.g., funding announcements, public statements, training activities, and guidance or initiatives).

Characteristics of the included articles

The characteristics of the included articles are provided in Table 1. All articles were reviewed against the classification framework^6,7 and demonstrated inclusion of evidence on the 1) utility and potential of AI (n=112), or the 2) considerations and risks of AI (n=105). More than three quarters of the articles covered both areas of the framework (n=94).

All 122 articles were published between 2022 and 2024, with the majority published in 2023 (n=68). Almost half of the articles originated from Europe (n=54), with the remaining originating from the Americas (n=49), Asia (n=25), Australasia (n=7), the Middle East (n=4), and Africa (n=1). There were 85 peer-reviewed journal articles, with more than half reporting on original research (n=47), and a further 25 articles were perspectives, and 13 were reviews.

The literature on AI covered a wide range of sector and organisational settings (e.g., IT and software engineering, research and industry, healthcare, finance), and most of the reviewed literature was in the academic and big data/AI development space, with applications focused to data science, IT and software engineering (n=40) and/or research and industry (n=31). In 16 articles, the information covered was found to be relevant to multiple sectors or organisation types.

The evidence also covered a breadth of organisational AI application areas, pointing to numerous processes that could benefit from integrating AI systems. The most covered area was data management and security (n=91), followed by data analytics and visualisation (n=35), and decision-making and strategy (n=35). The least covered area was operations management and maintenance (n=5), which could be due to the evidence being more specialised to industry settings (e.g., manufacturing, production) and therefore having limited relevance to RFOs.

Summarising the evidence

The evidence from the 122 included articles was summarised using the classification framework that was developed from existing AI/ML literature (see Figure 1).^6,7 The summaries from the included articles were used to draw out key statements, which were then categorised into common themes and assessed for frequency (number of references contributing to each statement).

Table 2 presents the statements, frequencies and evidence sources regarding the utility and potential of AI, and Table 3 presents the statements, frequencies and evidence sources regarding the consideration and risks of AI. Interestingly, whilst mapping the evidence against the classification framework, an additional area around the importance of sharing efforts in AI, whereby articles focused on the value of collaborative working and partnerships between private sector companies (e.g., AI vendors), research institutions, and public administrations (e.g., including funding organisations) in the AI space. On this basis, key statements derived from article summaries promoting shared AI efforts across communities of interest were also extracted and presented in Table 4 and Figure 3, along with the frequencies and evidence sources.

Figure 3. Mapping the evidence: utility, potential, considerations, risks and shared AI efforts for research funding organisations.

The following sections provide a summary of the evidence captured from the included articles based on the review’s three research questions. The quality or assessment of the evidence was not explored as part of the review, and the key statements are not presented in order of priority.

Utility and potential of AI for research funding organisations

Of the included articles, 112 described the utility and benefits (e.g., technological, functional, operational, and user focused) of implementing AI to support organisational, research or administrative processes. These articles either described the generic AI functions or systems (e.g., automation, generative AI, natural language processing) or pointed to specific tools (e.g., models, proprietary software or web platforms) that were either commercially distributed and ready to use (including open-source solutions) or remain in development (e.g., experimental ML algorithms).

It was clear from the evidence that understanding the utility and applicability of different AI technologies is complex, particularly due to the diversity of use across sectors and the different stages of digital transformation.^49–54 However, the evidence did provide several opportunities for RFOs to consider, particularly around the application, capability and implementation of AI. (see Table 2) The central focus was on AI/ML-powered tools and platforms, illustrating the transformative impact on data-driven organisational and/or administrative processes to enhance efficiency, innovation and decision-making.

Advanced data management and analytics: The evidence offered several opportunities for using AI/ML platforms to enhance organisations’ abilities to manage and analyse large datasets, providing valuable insights for ways to improve customer relationships and operational efficiency.^{26,49,55–57} AI-enhanced master data management and decision support systems also support data quality and can inform decision-making for stakeholders and facilitate long-term data management efficiency. For example, AI Ignite goes beyond building AI infrastructure, offering faster insights and operational efficiency, and empowering organisations to utilise the full potential of AI.^58,59 While, AI-driven systems like Darktrace have also shown promising steps to enhance cybersecurity by preventing cyberattacks and are in turn driving the development of more robust public policies to govern AI cybersecurity.⁶⁰ Data mining algorithms and artificial neural networks are other AI solutions that can also inform organisations’ business intelligence models, and these work by enabling functions such as, data warehouse (e.g., Snowflake), data extraction (e.g., ETL) and report generation.^61–64

Administrative task automation: Several ways of how AI can streamline public administration by automating services and processes, enhancing efficiency, and increasing public confidence was noted from the evidence.^{26,28,42,54,65} For example, in accounting and law, AI has been used to automate repetitive tasks and enhance decision-making through predictive analytics and real-time data transparency, and therefore improving risk assessment and audit quality.^1,63,66–68 Conversational AI meanwhile showed to enhance efficiency and customer satisfaction by automating customer-staff communications.^53,69 The evidence found that AI automation is predicted to save considerable time (40-50%) for administrative and analyst users, with this figure reportedly increasing to 80% as AI improves with more data. An Accenture analysis also determined that 40% of working hours across industries could be automated or augmented by AI adoption.^70,71

Predictive and performance analysis: The evidence offered several ways on how AI-driven automation and predictive analytics can optimise resource-intensive activities and support sustainability by enabling remote work and real-time performance analysis.^3,52,71,72 This is particularly relevant for RFOs who are keen to enhance project management, as AI has the potential to predict costs and risks, therefore harnessing productivity and complex data problems. In Human Resources (HR) practices, neural networks, NLP, and gamification tools could be leveraged to enhance recruitment, training, and performance, while ‘Empathetic AI’ can support leadership and HR processes through automation, improving employee support and organisational efficiency.^71–77 Other examples were shown in accounting, where predictive audit support enhanced risk assessment and audit quality, and in legal environments AI was shown to improve decision-making through real-time monitoring and data transparency ( Table 5).^1,78–81

Considerations and risks of AI for research funding organisations

Of the included articles, 105 described the considerations and risks (e.g., technological, operational, ethical and user focused) associated with ensuring organisational AI ‘readiness’ to assess capabilities and optimise organisation’s data infrastructure and management systems in preparation for AI/ML deployment (see Table 3). The evidence strongly indicated several organisational challenges that must be addressed for any business transformation or research activity involving AI/ML adoption to be credible. In particular, introducing or experimenting with AI technologies has raised concerns around data quality, issues with how data is shared, managed, and modelled for ML, and whether research and funding environments generally have the space, capacity, capability, and infrastructure to support and potentially mainstream the adoption of AI/ML for efficiency benefits.^{46,50,82–84}

Data storage and governance: From the evidence, several organisations were found to be implementing cloud and edge computing solutions to improve data architecture (e.g., how data is integrated, stored, and governed), manage AI/ML workloads, and cope with increasing data volumes and data types (e.g., text vs image, structured vs unstructured data).^{61,68,83,85–89} Cloud-based solutions like data lakehouse⁶¹ and data mesh⁸⁸ can notably enable scalable and efficient data management, streamline data flows, and provide real-time insights for enhanced business performance. Examples of data warehousing solutions were also provided in the evidence (e.g., Hadoop, Microsoft Azure, and Apache Spark) and can reportedly support big data management and decision-making by offering comprehensive governance frameworks that enhance data quality, efficiency, and security.^87,90 Cybersecurity was also highlighted in the evidence, particularly around how federal agencies and organisations are adopting secure cloud fabrics to ensure data governance, seamless connectivity, and secure access to data lakes.^83,91 Other potential opportunities to enhance big data management and security included using distributed ledger technology like blockchain to securely manage digital assets (e.g., Syneos Health and startups like Granica) with comprehensive data management strategies for organisations adopting LLMs.^91,92

Frameworks, models, and metrics: The evidence suggested that integrating standards, governance and ethics into workflows and AI models could alleviate the burden on innovation, and allow organisations to effectively utilise, regulate and govern AI.^{16,51,64,93–96} A key consideration is having clear AI definitions to implement the necessary legal frameworks; developing fairness metrics for models; mitigating algorithmic bias; and implementing learning frameworks (e.g., IBM AI Fairness 360⁹⁷). Several frameworks and models were found in the evidence to regulate and verify the adoption and implementation of AI, such as the AI Ethics Orchestration and Automated Response (EOAR) framework,⁹⁸ to ensure regular audits and testing of models; using platforms such as Open Scale to define, monitor, and tune the ethical behaviour of IBM AI models⁹⁸; the ‘Digital-Era Governance’ model for analysing the impact of digitalisation on administrations and⁹⁹; the International Science Council’s adaptive analytical framework (derived from the OECD AI Classification framework) which offers a comprehensive checklist for stakeholders to address AI impacts.¹⁰⁰ Other national-level efforts in this area included the UK Department for Science Innovation and Technology’s proposed regulatory framework for AI¹¹ to ensure responsible innovation and research-driven initiatives in the US and Europe (e.g., HPC-FAIR, AI4GOV project) aimed at redefining and applying FAIR principles to maximise the impact of data investments and monitoring AI for bias or non-compliance.^101,102

Enhancing big data analytics: Several cloud service providers were suggested such as AWS, Azure, and Google Cloud Platform (GCP) to enhance data management and AI-powered analytics in the financial sector, offering scalable, flexible solutions for handling large data volumes, enabling real-time decision-making, and improving operational efficiency.⁸⁹ In addition, edge computing can pinpoint and locally process the data necessary for analyses, providing scalability, real-time data analysis, interoperability and confidentially for emerging ‘Internet of Things (IoT)’ (e.g., sensor and ICT-based) solutions.⁵⁷ Developments in Blockchain technology were also shown to assist AI algorithms in managing big data in healthcare, and can help monitor data and assess the integrity of analytical outputs.¹⁰³

Data standards and research tools: The evidence reported on the development and use of data standards for AI-driven research, as well as data-centric initiatives to promote AI data quality, facilitate big data management in research, and ensure that data is more findable, accessible, interoperable and reusable (FAIR) across organisations, institutions and platforms.^{50,93–96,101,104} For example, the FAIR4HEP collaboration promotes FAIR data principles for the reuse of experimental data within and between institutions⁹³; the Research Description Framework (RDF) provides semantic web standards for consistently describing, integrating and exchanging metadata⁹⁵; and a proposed computational framework for quantifying the FAIRness of AI models and adopting AI-ready datasets has been developed to enable AI-driven scientific discovery.⁹⁶ From a business perspective, knowledge graphs and semantic web standards were reported to enhance AI implementations by automating workflows and facilitating complex queries¹⁰⁵ and semantic web standards also ensure data compliance with FAIR principles, enabling efficient data governance and retrieval.⁹⁵

Technical infrastructure: The evidence provided important considerations for organisations who wish to be ‘AI ready’ in their capabilities,^106–111 such as upgrading outdated infrastructure, maintenance and management systems^3,69; harmonising business processes; standardising and validating master data; and enhancing IT agility (e.g., by adopting Cloud) and ensuring, competitiveness, and resilience.⁶⁹ However, it was also evident that upgrading maintenance management systems requires adopting big data, cloud computing, and automation, managing vast data efficiently, addressing sustainability issues, and ensuring robust cybersecurity.³ Investment in data availability, quality and careful evaluation of cloud resources for model training are also crucial (e.g., for implementing generative AI models) and adopting cloud-based solutions in particular (e.g., Cloudera, Azure cognitive services) is advantageous as these support routine tasks; expedite report preparation; enable quicker strategic decisions; and promote data democratisation.⁶⁹

Guidance on ethical AI: One of the key considerations from the evidence was for the need for countries and organisations to adopt ethical guidelines around AI tools, and LLMs like ChatGPT in research.^{64,102,112–116} Several factors were mentioned around AI’s capacity to manage and analyse large data volumes (also enhancing data integrity), the need for human oversight to mitigate risks (e.g., reputational, legal, and regulatory)¹¹²; ensuring responsible use of AI in research funding and management³⁶; and ensuring the protection of democracies and the risks to human equality posed by big data models (e.g., bias against certain groups).^97,102 Regulation and accountability in big data is essential, and there is a need for public sector organisations to consider the historical barriers to innovation; embrace technological changes; leverage data to meet public needs; and replace outdated systems.²⁸ However, any integrated AI solutions must uphold existing rules regarding data integrity, ensure proportional and transparent data collection, and maintain confidentiality of sensitive data (e.g., patient, public, and research data).^4,51 Several regulatory efforts were reported, focusing on national AI model risk management frameworks, enhancing governance, and ensuring model lifecycle updates.¹¹³ For example, The EU guidelines on ethics in AI, emphasises the need to ensure AI systems meet criteria for transparency, privacy, and fairness⁴; the United Nations Educational, Scientific and Cultural Organization’s (UNESCO) recommend establishing a universal framework of ethical AI principles and actions¹¹⁵; the EU’s AI4GOV project is developing a reference framework for ethical and democratic AI¹⁰²; and the AIDE Project (Artificial Intelligence in Healthcare for All), a UK and Japan collaboration to identify and address public concerns regarding AI use in healthcare ( Table 6).¹¹⁷

Facilitating shared AI efforts

In addition to the above, the evidence also strongly indicated that facilitation of shared AI efforts through collaboration and partnerships is important in achieving AI readiness and successful implementation in the research sector. Fostering transparency and shared learning between the key stakeholders in research (e.g., researchers; RFOs; academic institutions; public and private organisations; public representatives) is crucial for a) understanding the real-world use for AI/ML in administrative research and research management practices, and b) enabling development of and access to solutions that can be transferable across several organisational and data management contexts.^{10,26,28,31,33,42,65} The statements below (see Table 4) provide a summary around the importance of an open culture and shared AI efforts between organisations and sectors who have a role to play in research.

Developing organisational AI strategy : From the evidence it was clear how understanding data value and trends is crucial for developing AI capability. Although understanding data requirements are closely tied to government policy (and initiatives), adherence to ethical standards, and availability of financial, knowledge and technical resources, there is a need for organisations to carefully consider AI solutions and develop data strategy plans that encompasses both AI and data management practices.^53,81 Investing in data strategies, with the significance of data literacy requirements, a key consideration for organisations is the potential to invest in collaborations and partnerships to facilitate AI implementation.^{26,27,53,54,70,71,73,74,83,87–89,99,110,111,115,118–120} Digitalisation strategy plans need to identify key performance indicators and effective adoption of ethical AI business practices, which in turn should involve having the right organisational structure; an interdisciplinary working culture; communication; management support; training; appropriate measurement and reporting mechanisms; and enforcement.^{9,27,43,54,70,71,75,82,86,96,110,111,118,121,122} The evidence also suggested that AI application management and AI-driven platforms need to facilitate stakeholder interactions and continuously innovate in rapidly changing environments.^44,60,92,113 LLMs in particular require a strategic roadmap for effective integration, with careful assessment of applications, techniques, and regulatory considerations. In financial management, the strategic roadmap should involve AI being implemented in stages and only follow once data management infrastructures have been upgraded.^50,83,106

Monitoring AI transparency: Strengthening accountability in AI (‘explainable AI’) requires human oversight and involvement throughout the model lifecycle, as well as engagement with internal stakeholders and AI users to avoid automation bias.¹²³ The development and adoption of explainable AI was mentioned across several organisational settings, with particular focus on the transparency and interpretability of AI technology.^{67,70,123–127} For companies developing powerful AI systems (e.g., OpenAI), this means taking appropriate steps to involve independent members to ensure democratic AI governance and to prevent misuse.¹²⁵ Organisations, particularly in healthcare or research settings, need to ensure well-documented datasets for all AI technologies; identify user groups at risk of disparate performance or harm; justify appropriate dataset use; evaluate performance across groups; report limitations or misuse; and develop risk mitigation plans throughout the AI technology lifecycle.^{26,42,52,76,87,91,113,125} The evidence also highlighted how model training and lifecycle development (e.g., for LLMs) need to explore incorporating FAIR data principles at every stage to generate FAIR-compliant datasets.^{50,55,68,93,96,98,101,102,104,119,123,127–129}

Increasing AI partnerships: Several initiatives and alliances were reported in the evidence, particularly around forming partnerships to ensure return on investment and sustainable social and economic growth.^{67,70,87,94,99,111,118,119} Public and private sector collaborations can offer comprehensive support solutions across various stages of AI implementation and co-develop AI systems that are intrinsically inclusive of human values and humans’ ability to create sufficient solutions that serve the public interest.^{9,17,26,28,42,44,51,65,78,105} Initiatives such as the Responsible AI UK (https://www.rai.ac.uk/) and the European Commission’s joint Research Centre’s Artificial Intelligence Watch (https://ai-watch.ec.europa.eu/index_en) were two examples seen in the evidence to enhance potential collaborative partnerships as well as strengthen data management and administrative practices in research.

Developing AI knowledge and skills: A key highlight from the evidence focused on staff training; staff development; career growth; and building talent that is adaptable to the rapidly changing technological advancements across and within different sectors and organisational settings.^50,70,72,74 The evidence suggested how organisations need to invest time and resources into addressing the skills gap in order to successfully implement AI and understand the level of support required for the adoption of AI.^50,53,73 Although there was some evidence around the implementation and adoption of AI to streamline processes and reduce bias, data literacy and data strategies are key components to gain ‘buy in’ from all levels and sections of an organisation.^{17,73,80,82,87,128} Upskilling and offering AI education will be required across whole organisations, especially as AI-driven scheduling software and intelligent assistants are increasingly being implemented to reduce management and administrative burdens.^{9,37,66,97,111} In educational settings, promoting knowledge around AI and adequate resources is required to support AI management and learning, facilitate staff willingness to adopt new digital skills and to reduce instances of unethical AI use that could have implications on academic integrity.^20,116 In more business focused settings, data leaders are already capitalising on AI, or expecting increased interest and investment in AI technologies, by upskilling their employees in AI/ML, and focusing more on improving data literacy and data readiness for AI in future data strategy plans ( Table 7).⁸¹

Evidence from funding and professional organisations in research

To gain further insight into the priorities and positions of key funding and professional organisations with respect to AI, we reviewed 13 webpages of research organisations. It was clear that several organisations are initiating funding opportunities that involve AI, and this was particularly evident from organisations focused on healthcare. For example, the UK Research and Innovation’s (UKRI) funding opportunities to accelerate new treatments and therapies using AI technologies, and the NIHR and Wellcome funding schemes to support healthcare research involving big data (https://www.nihr.ac.uk/explore-nihr/funding-programmes/ai-award.htm). Several key areas were noted from the review of a purposive sample of relevant organisations, which are reported below.

Ethical conduct: High on RFO’s agendas is ensuring responsible use and addressing the ethical challenges that AI presents, which had led some organisations such as the Canadian Institute of Health Research (CIHR) and UKRI councils to set up leadership groups and advisory committees in ethical AI/ML (e.g., Engineering and Physical Sciences Research Council (EPSRC), and the AHRC and Economic and Social Research Council (ESRC) have partnered to create a leadership team to drive the UK’s responsible and trustworthy (R&T) AI agenda). Other RFOs, like UKRI’s Arts and Humanities Research Council (AHRC) and the Academy of Medical Sciences (AMS), are exploring ways of including ethics in the research and development of AI technology, recognising the need for strengthening collaborations and partnerships with other organisations and technical experts to ensure that AI is used safely, effectively and ethically.

Security and confidentiality: Another area high on RFOs agenda focuses on security and confidentiality of personal data when using AI tools for peer review or grant applications. As a result, a joint statement from the UK’s Research Funders Policy Group, which include Wellcome and the NIHR, issued caution against the application of LLMs such as ChatGPT. The National Institutes of Health (NIH) have gone further by prohibiting peer reviewers from using any AI technologies for analysing or formulating their reports on grant applications and contract proposals (e.g., NLPs, LLMs and other generative AI tools). In a response to these concerns, several organisations such as the UKRI’s Challenge Fund are supporting service industries to use AI and data analytics in the security of their computer infrastructure.

Equitable AI: Preventing and mitigating biases were mentioned by several RFOs like the CIHR, who highlighted in a report from the AI for Public health Equity Workshop in Toronto planned efforts to promote equitable AI and equip public health sectors in Canada with new methods to advance health equality as a priority area (Public Health: IPPH Strategic Plan 2022-2026). The Gates Foundation are also focusing on access and equity in AI work and have established a task force to ensure AI usage is safe, ethical and equitable worldwide. Furthermore, the NIHR have announced their Artificial Intelligence and Racial and Ethnic Inequalities in Health and Social Care funding call to conduct work ensuring that AI research meets the needs of ethnic minorities (https://www.nihr.ac.uk/explore-nihr/funding-programmes/ai-award.htm).

AI knowledge, skills and resources: Although, several RFOs highlighted how they are assessing the need for staff to enhance their research skills through interrogation of AI, they are addressing this in different ways. For example, the NIHR ARC North Thames Academy have set up an AI in Systematic Reviews Masterclass for healthcare and government staff ( https://www.arc-nt.nihr.ac.uk/news-and-events/2024/feb-24/the-use-of-artificial-intelligence-ai-in-systematic-reviews- masterclass/), while the NIHR CRN have developed an AI e-learning course in partnership with Imperial College London (https://www.nihr.ac.uk/news/artificial-intelligence-e-learning-launched-for-researchers/31387).

Big data in research: The Association of Research Managers and Administrators (ARMA) first national survey of research offices predict that research management practices are poised for increased use of AI and big data in the near future. Similarly, the Cancer Research UK’s (CRUK) Grand Challenges Fund demonstrates that researchers are also looking to harness big data and AI for medical data analytics for early detection and treatment of cancers. On a larger scale, the EU has announced a Digital Strategy to establish standards in data and invest in AI, while major funders such as the NIH are establishing data science and AI programmes to design and develop intelligent data-driven AI-based medical screening tools.

Public scrutiny: Several organisations have examined the use of and implementation of AI to ensure that the public’s best interest are protected when using AI, while also considering (where applicable) regulations such as the EU Act (e.g., the European Commission AI Watch monitor the development, uptake and impact of AI across Europe through a regulatory framework based on human rights and fundamental values). Outside of the EU, the Gates Foundation are in efforts to make AI technology broadly accessible to all to transform how people communicate, work and learn, and the Joseph Rowntree Foundation are coordinating discussions around harnessing AI for public good. Other examples of promoting public scrutiny in AI include the Association of Medical Research Charities (AMRC) and their annual Benevolent AI award to help accelerate new treatment and therapies using AI, the CIHR have two initiatives through their Transforming Public Health IPPH Strategic Plan and AI Public Health Equity Workshop, and the NIHR’s plans to apply real-time operational modelling to maximise care pathways (https://www.uclhospitals.brc.nihr.ac.uk/criu/research-impact/artificial-intelligence-and-machine-learning).

Current and emerging use of AI

Adopting and successfully implementing AI is complex, particularly when comparisons are made between the private sector (the ‘facilitator’ of AI) and the public sector (seen as the ‘regulator’).^26,28,42,65 Added to the complexity is the need for organisations to assess the relative merit and effort required to implement AI technologies, which are particularly important in considering the readiness and capabilities of AI, and what is needed to address the unique challenges of AI adoption.^{9,28,51–53,65,81,87,111}

A large proportion of the evidence was relevant to the research sector, particularly around improving the quality of research data management (making it FAIR and verified for AI/ML training), offering important considerations for RFOs who collect research data to monitor and report on the impacts of funded research (for accountability and to secure further investments). RFOs manage high volumes of data both directly from awards and from other sources, and they engage in analytics to manage, monitor and evaluate their organisations funded portfolio. The evidence has shown the potential benefits and capability of AI, and how it could enhance efficiency and insights in these processes, particularly with ML. Standardising research data management and improving the quality of research data is needed to ensure accuracy of ML models applied to business and research processes, and ultimately the success of outcomes. However, it is also important for research in the wider context, based on reports of increasing research bureaucracy caused by lack of alignment in standards and data systems in digital research environments.^{12,13,50,86,91,93,94,96,101,105,131}

The focus of the available evidence on AI in research for academic use may be explained by the fact that this is where AI is mostly used due to free access tools such as ChatGPT,^{6,18,20,116,132} and subsequently raises several ethical concerns (e.g., data storage, data privacy and security, and the use of information). Although several efforts were noted from the literature^26,113,123 (e.g., ‘Explainable AI’ techniques such as LIME and SHAP) these may not necessarily anticipate all ethical and governance issues that may come up when AI is used more widely across public administrations (where accountability and public perception is more crucial).^{9,26,27,43,75,76,94,106,123} RFOs require high quality evidence to inform their own practices with regards to the adoption, implementation and capability of AI use, particularly when some funding organisations are taking a position on the use of AI during the funding application process (e.g., peer review and grant writing).^{10–12,24,32,36,96,133}

Potential benefits of AI for RFOs

To gain a comprehensive picture of the AI landscape, we considered existing AI applications, new developments in technologies and emerging uses for AI that could be applied to organisational contexts in research (e.g., research data management and administration). There was emphasis on exploring how AI could be leveraged to reduce research bureaucracy, particularly administrative burden for RFOs, based on widespread reports of increasing pressures in research to demonstrate impact and value of investment.^31,32 However, the increasing effort also extends to researchers, where most data management processes are manual and time-consuming.⁹² AI implementation could have the potential to reduce prompt-like queries, and develop Edge AI applications and NLP models for human-AI interaction to shift towards enhanced insight and reduction in burden for researchers.⁵³ Despite the advantages of AI applications though, they are not a replacement for human intelligence or critical thinking; they are tools to encourage more streamlined approaches to data management, research and educational processes.^53,78

It is also particularly challenging for RFOs, given the rapid speed of AI/ML development, to address issues around performance, fairness, bias (e.g., ‘AI hallucinations’ which is when AI models generate incorrect, misleading or provide false information) and maturity of AI technologies. There are increasing efforts focusing on the integration and capability of AI, such as the Responsible Artificial Intelligence UK (https://rai.ac.uk/) and the European Commission’s Joint Research Centre’s Artificial Intelligence Watch (https://ai-watch.ec.europa.eu/index_en), which could facilitate and foster greater reliability for the adoption and implementation of AI in research data management.^{27,28,42,43,65,82,94,134} Solutions are still needed to not only manage the current demand placed on RFOs, but also where AI/ML tools could serve organisations and the wider research community (e.g., by reducing duplication and facilitating monitoring, reporting and evaluations) (see Table 5 for a summary of the evidence on the key considerations for optimising AI implementation for RFOs).

Considerations and risks of AI for RFOs

Concerns over current data quality, data management practices, and the implications for AI/ML model accuracy and effectiveness, are common across sectors, organisations and disciplines.^97,101 The overall risk with using systems that are inaccurate, biased, or compromise human rights (e.g., privacy, accountability, security) is that users and leaders will lose trust in AI and not look to harness potentially useful technologies that could reduce the burden on innovation.¹²³ Although, the negative outcome can be avoided by ensuring human involvement throughout the AI model lifecycle, it is imperative for stakeholders and end-users to be actively involved in decision making throughout the initiation, and development of establishing systems for monitoring and evaluation purposes.⁹² (see Table 6 for a summary of the evidence on the key considerations to develop trust in AI implementation) There are also requirements for regulatory efforts to ensure safe and responsible AI innovation, particularly due to ethical and governance concerns around data integrity, data collection and confidentiality. Data initiatives and insight are constantly shifting (and at a rapid pace) and there is a clear drive for organisational and government strategies to engage in the long-term sustainability of AI technologies.^4,135–137 These investments in AI are predicted to be long-term and sustainability should not be overlooked by focusing on cost alone.⁸⁴ Moreover, integrating AI technologies requires new initiatives to modernise existing research data management practices; whilst that may come at a cost, it will inevitably produce greater efficiencies and optimise reduction in operational costs resulting in longer term efficiency gains for RFOs.

Shared AI efforts across sectors

There was a particular emphasis on the need to promote openness and transparency through communities of interest (e.g., different sectors, settings and organisations), particularly when there is a need to reduce the burden on innovation, and facilitate trust with AI technologies.^16,81,83 Facilitating and fostering transdisciplinary collaborative opportunities across diverse cultural domains could offer greater reliability of potential applications and promote a more collaborative, data-driven approach to further advance the adoption, capability and implementation of AI/ML technologies (from a collective viewpoint rather than from one organisation) (see Table 7 for a summary of the evidence on the key considerations around shared AI efforts). According to the International Science Council, there is an ontological gap between AI principles and their integration into regulatory and governance frameworks, requiring a systems approach that includes the scientific community and considers the broader implications of AI for science and society.^100,101 Effective regulation will necessitate risk-informed decision-making across multiple layers and stakeholders, including RFOs. The adaptive analytical framework could serve as an example for RFOs as they assess multiple layers of impact and are looking to develop a more systematic approach to monitoring and reporting outputs, outcomes and impacts.^100,101

With much of the literature in the data science space, RFOs may need to assess and consider how to upskill their workforce and seek collaborations with software companies and data science experts/ML researchers.^{9,27,54,70,71,75,86,110,111,118,121,127} Organisations could also engage with data scientists for AI expertise and to post-process their datasets, (clean, assemble and validate the data). Where AI is experimentally used on imperfect data, this reduces some processing effort and guides future data collection practices. These shared decision-making and learning practices across sectors could ultimately help foster and drive business performance and transformation, and by implementing strategies to enhance data management would be a competitive advantage in today’s data-driven world.^{26,28,42,65,91,93,96,105,106,127,136}

Study strengths and limitations

The main strength of the review was the extensive coverage of sectors across varied organisational settings and including a review of 13 research funding and professional organisations. Using a framework to map the evidence provided a clear interpretation of where and how AI has been and is being applied and adopted across a range of settings to determine the potential benefits and opportunities, and consideration and risks associated to implementing AI technology for RFOs. However, a limitation to the review was that scoping reviews are restricted to only mapping the evidence and do not assess the quality or risk of bias of the literature. Although we mapped the evidence by research methodology (e.g., original research, perspective, review) due to the wide scope of the review (not limited to a particular sector, discipline or field) it was clear how some research methodologies were not as developed or consistently defined as one would expect in healthcare or food and environmental settings. The review only covered a three-year period due to the rapid pace of evolving evidence (also noting that literature published prior to 2022 was more conceptual and focused on algorithm development only), which could also be seen as a limitation to the review. Although, there was some international coverage from the included articles, the majority were from Europe and Americas, which could have introduced regional biases. However, it could also be that there was a lack of evidence from these regions, given the coverage of the databases and grey literature searches.