The role of open research in improving the standards of evidence synthesis: current challenges and potential solutions in systematic reviews

Overview of systematic review workflow

SRs as well as MAs are characterized by an explicit and robust methodology which consists of several consecutive steps starting from framing the research question followed by identifying relevant work, appraising the quality of included studies, summarizing the evidence, and interpreting the results.¹⁶ However, researchers while conducting an SR, may experience many challenges despite their phenomenally simple workflow.¹⁶ FAIR principles refer to the fact that research data should display the following characteristics: data should be registered in a searchable source and assigned a unique identifier, be retrievable ideally through automated procedures, be available to be combined to maximize their value, and lastly be described in detail to facilitate their reuse (Figure 2).¹² The movement toward an open research culture that supports the FAIR principles in data sharing may have important implications in the conduct, methodology and reporting of SRs and other evidence synthesis studies.¹⁷ Below we discuss the potential obstacles that researchers may often encounter while conducting an SR and how they could overcome these barriers by using open research practices and open relevant tools

Figure 2. Findability, Accessibility, Interoperability and Reusability in evidence synthesis.

Challenges in evidence synthesis and open research solutions

Framing the research question

Conceptualizing an SR and formulating the research question is the most important step in evidence synthesis studies. A significant challenge that researchers face at this stage is to identify information regarding ongoing SRs addressing the same research question. Ioannides et al. have painted a picture in evidence synthesis research activity where there has been an explosion of SRs and MAs that are often redundant.¹⁴ Katsura et al. also suggest that multiple SRs on the same topic represent wasted efforts and they can be confusing and potentially misleading.^14,18

Interestingly, until 2011, SRs or meta-analyses were mostly available through search engines such as MEDLINE and EMBASE only after they have been completed and published.^14,19 Researchers and healthcare evidence users were facing challenges in finding ongoing SRs as only a limited number of organizations including Cochrane Collaboration contained information related to SRs that were in progress.²⁰ To overcome these challenges and promote FAIRness in evidence synthesis, the International Prospective Register of Systematic Reviews (PROSPERO) was launched in February 2011 as an open-access database where all health-related SRs are eligible for prospective registration.²¹ Additionally, the status of SRs is regularly updated and therefore researchers and healthcare professionals are able to find SRs which are in progress or are completed but not yet published.^19,21 Open-access findability of ongoing SRs provides researchers and commissioners with an important tool to avoid duplication and reduces research waste.²² It is worth highlighting that the ability to promote FAIRness of ongoing SRs through PROSPERO does not only benefit researchers but also is important for clinicians and other healthcare professionals to remain up to date with related ongoing developments in healthcare.²²

Despite, the benefits of findability through the PROSPERO open access database and the increase in its use over the years, still, only a fraction of SRs were noted to be registered.^14,23 Tawfik et al. by conducting a global survey of 270 authors, reported that only 50% of SRs were officially registered.²³ Interestingly, the most common cause was a lack of awareness of the importance of open-access prospective registration of SRs, indicating the need for academic education in open research culture.²³ To overcome this observation and encourage PROSPERO registration, the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement updated its criteria in 2020 for reporting SRs by moving the registration information to a separate checklist section to encourage open research practice within the global research community and promote FAIRness in SRs.²⁴

Identifying relevant work

To capture as many relevant citations as possible during an SR, a diligent and thorough search should be conducted.²⁵ An extensive literature search is achieved using multiple electronic databases to identify primary studies according to a priori specified inclusion and exclusion criteria.^2,25 Identifying relevant citations is a fundamental step for a reliable SR and includes the development of a meticulous Boolean strategy designed separately for each database. Subsequently, screening steps are performed based on the abstract and full text.² A survey by Major et al. showed that the design of a search strategy and the performance of an extended literature search were reported to be the most challenging aspects that researchers encountered during an SR.²⁶

The development of a thorough Boolean strategy algorithm including all the possible vocabulary and synonyms is a relatively complex and time-consuming process which could be overcome by implementing open research principles in SRs through the current available open research tools (PROSPERO).²⁷ A Boolean algorithm which is easily findable and accessible could be reused or modified by several other research teams who wish to conduct an SR on a different subject in the same research field to avoid duplication of research efforts. Interestingly, despite the presence of reporting guidelines such as PRISMA and the need for FAIRness and transparency in reporting the methodology of SRs, several studies have identified that in most SRs, a full Boolean search strategy was missing from the published articles.^28–30 To further promote open research practices in SRs, one of the alterations noted in the updated PRISMA 2020 guidelines is the modification of the search term which now recommends that full search strategies for all databases should be reported.²⁴ Additionally, the process of registration in the PROSPERO database requires the submission of the full search strategy; however, making this information publicly available prior to the publication is still optional.²¹ Recently, preliminary results from the REPRISE study by Page et al. showed that in a random SR sample less than 1% of the included studies reported an analytic search code which was used to generate results, highlighting that despite available open research tools and the revised PRISMA 2020 guidelines, further interventions are needed^31,32 The development of common rules across the databases regarding the Boolean strategy design or the presence of easily accessible instructions could potentially aid researchers in tackling the challenging task of analytic vocabulary code design.

The screening phase of an SR aims to select potentially eligible studies for further assessment based on the prespecified inclusion and exclusion criteria. According to Cochrane guidance, two or more reviewers should independently perform this step, first by screening the title and abstract followed by a full-text assessment of the studies initially included.²⁵ A common challenge that researchers face is the lack of consistent full-text availability leading them to proceed with individual communication with authors requesting full texts of the candidate manuscripts with variable responses.³³ Studies have shown that data requests are ignored in up to 41% of cases.^33,34 A potential solution to this challenge is proposed to be open-access publishing which is steadily increasing in popularity as articles that are freely accessible are more shared and ultimately benefit the distribution of knowledge.³⁵ However, open-access publishing is associated with increasing article processing charges that place an increased financial burden, especially on unfunded and early-career researchers.³⁶ Although journals and publishers should continue to encourage open-access publishing, we encourage them to consider alternate models of support to reduce the financial barriers that lesser-funded or unfunded researchers face.³⁶ Furthermore, given that institutional affiliation contact details frequently change in a researcher’s career, the availability of research websites such as ResearchGate or other social networking sites for researchers could facilitate communication across the research community and promote accessibility to full text or data of a study. In addition, we believe researchers should be entitled to free access to articles related to the SR they are conducting to promote knowledge and improve the scientific rigour of the study.

Data extraction and quality assessment of included studies

This step encompasses data collection from the included full texts and is accompanied by a quality assessment of the included studies.¹⁶ These steps not only require a considerable amount of time and effort but also generate a vast amount of valuable secondary data which could be used and built upon to generate further new knowledge.³⁷

Data extraction constitutes a significant portion of an SR. Making all data extracted during SRs openly available could reduce unnecessary duplication of effort from other researchers and promote a standardized format in which data are presented in SRs.^38,39 To increase SR data openness, the Systematic Review Data Repository (SRDR) was developed and launched in 2012 by the Brown University Evidence-based Practice Center and funded by Agency for Healthcare Quality and Research (AHRQ).³⁹ This open-access resource aims to serve as an archive for SRs as well as an open-access data extraction tool.³⁸ Saldanha et al. reported that since 2012, SRDR has gathered over 150 SRs and includes publicly available data from more than 15,000 studies covering a variety of health-related research subjects.³⁹ Cochrane collaboration has also made initiatives to support accessibility in SR data and is committed to being fully open access by 2025.⁴⁰

Quality assessment of included studies and risk of bias reporting are key features of a high-quality quantitative SR and one of the most challenging stages in evidence synthesis.⁴¹ Risk of bias assessment requires a thorough judgement of the included studies which may introduce an element of inter-rater variability meaning that the same data may be judged differently between research groups.⁴¹ For this challenge, there are several available standardized open tools such as the Cochrane tools (RoB1, RoB2, ROBINS-I), especially in health-related themed SRs which aid researchers in this task.⁴² Interestingly, although the risk of bias is considered essential according to the PRISMA guidelines, several studies assessing the reporting quality of published SRs showed that risk of bias reporting is performed in less than 50%, raising concerns regarding their quality.^43–45 A potential solution to this issue may be provided with the a priori registration of an SR in the PROSPERO open-access database which also requires a detailed study design, search strategy and protocol submission with the aim to minimize bias and promote high methodological standards and reproducibility.^19,21,22 Studies by Sideri et al. and Allers et al. reported that SRs which were officially registered in the PROSPERO database appeared to demonstrate higher standards in their methodology and reporting quality.^46,47 However, this improvement may not only be attributed to the early accessibility of the registered SRs’ design and protocols but also to the fact that authors may have invested more research time in a high-standard SR research design which may indirectly result in a more robust methodology.⁴⁶

Providing open access to all aspects of an SR including the risk of bias availability and raw extracted data could provide several benefits including a reduction in duplication of effort, improvement in the quality and efficiency of SRs as well as support for secondary analyses addressing additional research questions.^37,48

Barriers to implementing FAIR principles

Despite the undoubtedly significant numerous benefits of SR data sharing, several challenges exist in fully implementing FAIR principles related to motivational and ethical barriers.³⁷ Ethical concerns, including ownership of data, intellectual property of ideas, and recognition of data producers, are some of the barriers related to moral principles that may challenge data sharing and the widespread practice of FAIRness in research studies, including SRs.⁴⁹ Zuiderwijk et al. performed an SR to investigate what inhibits researchers from openly sharing their data. Their study identified trust as a powerful and impactful driver for FAIR data sharing especially in a pre-publication state.⁵⁰ Notably, Zuiderwijk et al. characterised fear over data control or recognition of researchers’ efforts on data generation as major trust inhibitors for open data sharing.⁵⁰ This barrier is widely recognized, and several organizations have highlighted the need for appropriate systems for recognition so that acknowledgements are made as needed.³⁷ For instance, Cochrane collaboration states that authors should “respect and acknowledge the source of the data”. Additionally, the SRDR has adopted data citation with digital object identifiers (DOIs).³⁷ Another practice is the registered reports publication process which is being implemented by an increasing number of journals and publishers where study proposals and protocols are peer-reviewed and pre-accepted before the actual research is undertaken and results are produced.⁵¹ Registered reports are an increasingly adopted publication practice as it envisages promoting transparency, reproducibility and openness in data sharing in line with the transparency and openness promotion guidelines.^51,52

Personal drivers and intrinsic motivations have been identified by Zuiderwijk et al. as important factors impacting openness in data sharing.⁵⁰ Scholars exhibit different character or belief-related traits in terms of being supportive or reserved in open data sharing.⁵⁰ In an effort towards behavioural and academic culture change, the role of education in an open research culture is of paramount importance.⁵³ Ioannidis et al. highlighted the need for a strong educational curriculum to equip researchers and meta-researchers with the knowledge of best scientific practices to promote FAIRness in meta-research.⁵³ Furthermore, Universities and other academic institutions may consider rewarding meta-researchers who make their analytic code and research data publicly available as proposed by the 6^th World Conference on Research Integrity and in line with the Hong Kong principles for assessing researchers to foster trustworthiness, rigour, and transparency.^31,54

Along with the various organizations which promote open research in evidence synthesis and the currently available open research tools, it is worth mentioning the key role that journal publishers and research funders play in enhancing data sharing and its reuse.³⁷ In 2016, the International Committee of Medical Journal Editors (ICMJE) proposed that data from published clinical investigative trials need to be shared to promote transparency and reusability.¹⁷ However, to date, no relevant policy exists for evidence synthesis research. Modification of the interface of the existing SR registries to include detailed protocols, search codes and raw data may further promote the reusability of SR/MA-generated data and improve the standards of evidence synthesis reporting.¹⁷ Additionally, an increasing number of publications, such as F1000 and Systematic Reviews, provide clearer requirements for datasets to be shared in accordance with the FAIR principles³⁷

Future directions: artificial intelligence and FAIRness in SRs

Undoubtedly, evidence synthesis research is growing exponentially, therefore the current robust but slow and resource-intensive process of the standard SRs is no longer sustainable.⁵⁵ Therefore, several methods and tools have been developed through the use of artificial intelligence (AI) in order to aid the automation of SRs.^55–57 AI includes machine learning (ML) which utilizes computer algorithms similar to logistic regression and natural language processing (NLP) which analyses a vast number of texts and extracts information.^56,58 Both ML and NPL are commonly implemented technologies used in the semi-automated conduction of SRs.⁵⁸ ML uses statistical predictive methods to calculate the likelihood that an article is relevant and is commonly used during the screening process whereas NPL analyses the semantic meaning and is used during the data extraction step of an SR.⁵⁹

Several automation tools have been developed that aim to execute the time-consuming and labour-intensive tasks of an SR such as search (e.g. Metta), article screening (e.g. Abstrackr), data extraction (e.g. ExaCT) and even automatic generation of PRISMA diagrams (e.g. PRISMA flow diagram generator).⁵⁵ The significant advance in AI software as well as the desire for openness and FAIRness in meta-research has led to the initiation of the International Collaboration for the Automation of SRs (ICASR).⁶⁰ In 2015, in the first ICASR meeting, a set of principles were established known as the Vienna principles which recommend the need for a collaborative multidisciplinary effort towards the automation of the evidence synthesis research process while highlighting that every automation technique should abide by the FAIR standards for open research and be freely available.⁶⁰

Despite the development of several automated platforms for SRs, the complete automation of the evidence synthesis process is not feasible at present, as no current AI tool can replace human judgement.⁵⁹ Additionally, the benefits of the implementation of AI methods in SRs remain unclear and AI platforms still need to undergo validation and refinement.⁶⁰ Nevertheless, AI tools are increasingly becoming incorporated into some of the steps of evidence synthesis until a fully automated process might be possible in the near future, (Figure 3). The conduction of evidence synthesis using AI in line with the FAIR principles and the utilization of computer science and health informatics may dramatically alter evidence synthesis research and eventually evidence-based medicine.^56,59,61

Figure 3. Automatable systematic review steps.

Sources Beller et al. and John Hopkins University of Medicine (https://browse.welch.jhmi.edu/sr-methods/sr-process). Created with Biorender.com (Academic license ID: KV24GJF9KY).

AI: Artificial intelligence, SRDR: Systematic Review Data Repository, SR: systematic review

Data Statement

No data are associated with this article