Open science interventions proposed or implemented to assess researcher impact: a scoping review

Eligibility criteria

Inclusion criteria

We included studies from the health sciences, biomedicine, psychology, and economics fields, as our impression is that open science developments are more prevalent in these disciplines, which are also closest to our knowledge and experience. We defined each discipline.⁶

We included studies that described any open science proposals or implemented open science practices by researchers, funders, academic institutions, societies (e.g., The Royal Society), to assess researchers and research impact, and focus on the following issues: researcher assessment (e.g., Declaration of Research Assessment – DORA), interventions (e.g., reporting guidelines^7,8), or incentives to increase the speed of access to new knowledge (e.g., preprints); the availability of all knowledge (e.g., registration; open access publication); improve the quality of published research (e.g., reporting guidelines^7,8); improve reproducible research (e.g., money back incentives⁹); and to increase sharing of data, code, and materials (e.g., open science badges^3,4).

Our focus was open science proposed or implemented signals/practices, as defined by the United Nations Educational, Scientific and Cultural Organization.¹⁰ Based on the Framework for Open and Reproducible Research Training (FORRT)¹¹ and the Transparency and Openness Promotion (TOP) guidelines,¹² open science topics include: reproducibility [crisis] and/or replication; design, methods, data [code] material transparency; registration and/or preregistration; publishing of research and publication models; conceptual and statistical knowledge; and academic life and culture.

Examples of studies that met our eligibility criteria include: studies evaluating the use of badges to increase data sharing^3,4; studies evaluation practices including pre-registration⁵; use of reporting guidelines^7,8 and change in editorial policy¹³ to improve reporting and compliance in published articles; and, new publication models¹⁴ or proposals of other models⁹ as solutions to the reproducibility crisis in research.

We included original research reported in English. We also included policy documents from funders (e.g., Wellcome Trust; National Institute of Health (NIH), Canadian Institutes of Health Research (CIHR)), academic institutions (e.g., Utrecht University) and researchers (e.g., open science Minerva^8,14). We included all article types, including but not limited to, conference proceedings, commentaries, and editorials.

Exclusion criteria

We excluded literature that was not published in English, and studies published before year 2000 due to logistics and resource constraints.

Search strategy

For our detailed search strategy please see Extended data.⁶³ An experienced medical information specialist (APA) developed and tested the search strategy in consultation with the project leads (DM, CA and MG). The search strategy was developed in OVID MEDLINE, and peer-reviewed by another librarian.¹⁵ We searched a range of databases to achieve wide cross-disciplinary coverage, which included: OVID MEDLINE, OVID EMBASE, OVID APA PsycINFO, EBSCO Cumulative Index to Nursing and Allied Health Literature (CINAHL), and EBSCO ECONLit. The search strategies were translated using each database platform’s command language, controlled vocabulary, and appropriate search fields. Medical subject headings (MeSH), EMTREE terms, American Psychological Association thesaurus terms, CINAHL headings and text words were used to search the concepts and synonyms of “open science”, “data sharing”, “tenure” and “academic promotion”. Language limits were applied to capture articles in English. We performed all searches on October 3, 2022.

We also searched the grey literature to help identify proposed and/or implemented interventions to promote open science as part of researcher assessment. The grey literature search was limited to sources familiar to the research team. Purposive searching ensured we had the resources to complete the project in a timely manner. Supplemental search sources were: Open Science Centre; Research on Research Institute; METRICS and METRICB (Berlin); RECOGNITION & REWARDS; Open Scholarship Knowledge Base; and DORA. We performed forward citation searching of the included studies using citationchaser,¹⁶ Scopus, and Web of Science Core Collection. Additionally, we reviewed the bibliography of included studies. All search strategies are reported using PRISMA-S.¹⁷

Screening and data extraction

Due to the volume of literature search results, bibliographic records were identified for screening using the Continuous Active Learning^® (CAL^®) tool, which uses supervised machine learning to rank titles and abstracts from most to least likely to be of interest. The CAL^® tool was chosen due to its prior use by some of our team members.¹⁸

The results of the database searches were imported into the CAL^® tool, without removing duplicates. The CAL^® tool learns from the results of manual screening by reviewers to identify and rank titles and abstracts most likely to meet the inclusion criteria.

For manual screening by reviewers, a screening form based on the review’s eligibility criteria was prepared to aid in making consistent judgements on article relevance. To calibrate, two reviewers screened the first 100 titles/abstracts using the CAL^® tool, independently. Discrepancies were identified. A third reviewer selected a pilot set of 10 titles/abstracts that were similar to the discrepant abstracts. The two reviewers screened the pilot set and discussed discrepancies. Subsequently, abstract screening was conducted using the CAL^® tool by one reviewer.

Several hundred titles/abstracts were screened until the yield of relevant records diminished. After screening 1000 records, we started to look ahead among the successive batches of 100 records that we screened. We stopped when the yield from the successive batches dipped below 5 (per 100) relevant records. After stopping, the results from the titles/abstracts screening were de-duplicated prior to full-text retrieval. For the screening questions, see Extended data.⁶³

We developed and piloted data extraction forms prior to data extraction. The extracted data included: (1) Corresponding author; (2) Discipline; (3) Publication year; (4) Journal or platform where the study was published; (5) The proposed and/or implemented interventions to assess researchers or research impact; (6) The stakeholders of the proposed/implemented intervention (e.g., academic institution, funder, publisher, and researcher); (7) Intervention type (single or multi-modal); and (8) Funding information of the study. Details of the forms used are provided in Extended data.⁶³

Data analysis

Our data analysis included both quantitative (i.e., frequencies and percentages) and qualitative (i.e., narrative interpretation) methods. Data collection was conducted by MG, DM, AA, and JYN. As an additional step, MG and AA discussed and verified 20 of the 27 full-text articles (open Science interventions proposed/implemented). Discussions were focused on how best to categorize whether a proposed intervention was single or multi-modal, and which stakeholders were targeted. Multi-modal interventions targeted several open science practices, although some recommendations can conceptually be linked to each other. Single-modal interventions, were those targeting a single open science practice. Stakeholders were those who were actors in conducting the proposed interventions. As a measure of impact, we entered the document object identifier of included each article in the BMJ Impact Analytics tool, to extract if the article was cited in policy documents.

We created a flow diagram to detail how articles captured in our search were screened. We also calculated the frequencies (and percentages), described narratively the characteristics of all data extracted from included documents, and present the results in summary tables. Our data extraction and presentation plan copied and pasted the text relevant information and present it from the included studies. This is noted with quotation marks at the beginning and end of each cut and paste. We reasoned this approach was more efficient and would more accurately report what the individual authors stated rather than any paraphrasing we did.

Deviations from study protocol

The original protocol indicated we would exclude case reports, case series, letters to editor, and narrative reviews. However, we did not exclude any records based on publication type.

Results of the search

The literature search retrieved 18,020 records (Figure 1). We screened 1512 titles and abstracts until the yield of relevant ones fell below 2 (per 375). Two hundred and eighty-nine records were considered eligible for full-text screening. After duplicates were removed in Endnote, 208 records remained for full-text screening. We identified 21 papers that potentially described any open science proposals or practices evaluated and/or implemented to assess researchers and research impact. One record was later excluded during the data extraction phase. A forward citation check of these 21 studies yielded 3901 records. Using search terms “research assessment” and “open science and research assessment”, resulted in 111 records and 19 relevant records for title and abstract screening and full text screening, respectively. Of these, 7 reports met our inclusion criteria. Finally, we included 27^19–45 studies in our scoping review.

Figure 1. Flow chart of search results.

Epidemiological characteristics

The 27 articles included for epidemiological data extraction were published between 2013 and 2022, with the number of studies describing open science practices for research assessment increasing gradually in more recent years (Table 1 and Table 2 – underlying data).⁶³ The corresponding authors were from 8 countries, with North America (e.g., USA and Canada) leading the vast majority of studies published (n = 12 and n = 6, respectively). Most of the publications were indicated as research culture, perspective, commentary, essay, proceedings of a workshop, research article, world view, opinion, research note, editorial, report, and research policy (n = 22). Only three records^19,31,33 in our scoping review were empirical research. The majority of the records did not report any funding information (n = 15), but of those that mentioned if they had received funding, seven records (47%) reported receiving funding, and similarly six records (53%) reported receiving no funding.

Open Science interventions proposed/implemented

The majority of the studies proposed recommendations for researcher assessment that were multi-modal (n = 20), although some recommendation can conceptually be linked to each other, targeting several open science practices. To this purpose, several proposals have been developed such as the San Francisco Declaration on Research Assessment (DORA),^40,46 The Leiden Manifesto for research metrics,⁴² The Metric Tide,⁴⁵ and the Hong Kong Principles for Assessing Researchers.³⁷ Some of the studies in our scoping review based their proposed recommendations on further evaluation or extension of previous initiatives. For example, Gagliardi et al.²⁵ identified and prioritized 10 measures (that were further grouped through consensus as five measures of high importance and five measures of moderate importance) for assessing research quality and impact (DORA-compliant measures). Hatch et al.,²⁷ formulated “practical guidance for research institutions looking to embrace reform … for driving institutional change at a meeting convened by DORA and the Howard Hughes Medical Institute”. While another study, Schmidt et al.,³⁴ discussed the implementation of DORA and anecdotally describes practices used for hiring junior faculty members in their department. Additionally, Hatch et al.³⁵ presented “a tool called SPACE that institutions can use to gauge and develop their ability to support new approaches to assessment, … built on design principles released by DORA, and a five-step approach to responsible evaluation called SCOPE⁴⁷”. The authors claim the tool “can be adapted to different institutional contexts, geographies, and stages of readiness for reform, thus enabling universities to take stock of the internal constraints and capabilities that are likely to impact their capacity to reform how they assess research and researchers”.³⁵ It proposes criteria such as, “diversifying career paths, focusing on research quality and academic leadership, and stimulating open science”. Understanding the relationship between SPACE and SCOPE is not inherently obvious.

Other multi-modal recommendations were included that did not mention any of the previously developed statements mentioned above. One study²⁰ proposed to ensure data sharing, registration/pre-registration, and to not assess researchers based on traditional metrics. Another study,²¹ with authors comprised of “The Committee on Developing a Toolkit for Fostering Open Science Practices”, drafted elements for a toolkit, discussing topics on communication of the benefit of open science, (e.g., compiling a database of research articles that describe the ways open science is beneficial, resources to signal an organization’s interest in open science activities). An anecdotal discussion that faculty promotion must assess reproducibility by new assessment practices, that ensure “researchers use proper experimental design and analysis”, and require institutions to incentivize data sharing and transparency, was discussed by another.²⁴ While another author²⁶ discussed several ways to reward research transparency during various stages of researcher’ career, encompassing the hiring process, promotion and tenure evaluations, and “society and organization honors and awards”.

Aiming to develop and evaluate “more appropriate, non-traditional indicators of research” that “facilitate changes in the evaluation practices for rewarding researchers”, Rice and colleagues³³ conducted a cross-sectional study “to determine the presence of a set of pre-specified traditional and non-traditional criteria used to assess scientists for promotion and tenure in faculties of biomedical sciences” among 170 randomly selected universities worldwide. Moher et al.,³⁶ “completed a selective literature review of 22 key documents critiquing the current incentive system” prior to convening a 22-member (e.g., academic leaders, funders, and scientists) expert panel workshop asking “how the authors perceived the problems of assessing science and scientists, the unintended consequences of maintaining the status quo for assessing scientists, and details of their proposed solutions. The resulting table was used as a seed for participant discussion”, which “resulted in six principles for assessing scientists and associated research and policy implications”.

Majority of included studies were in the field of health sciences (n = 10), followed by biomedicine/and or health sciences and multidisciplinary, (n = 7 and n = 6, respectively). One study³² proposed “open access for measuring research assessment in Earth and Natural Sciences”. Several studies targeted their initiatives mainly towards redesigning or modifying researcher assessments via new metrics, measures, methods, and incentives. One study¹⁹ designed a new metrics for evaluating researchers (e.g., ‘ε-index’), while another study²³ conceptualized “data-level metrics” “(DLMs) as indicators of scientific merit related to the production and (re-)use of datasets, … to capture and make data-sharing efforts visible”. Individualized research-philosophy statements, annotated curriculum vitae (CV), and “the use of a formal evaluative system that captures behaviors that support reproducibility” were also proposed,²² as a method to improve the transparency of scholarly reporting. The PQRST index – (research that is productive, high-quality, reproducible, shareable, and translatable) – is proposed by Ioannidis et al.²⁸ for appraising and rewarding research. Nicholson et al.³⁰ provided “a brief tutorial about impact metrics and how to use these metrics to document scientific impact, … detailing six steps to documenting the scientific and clinical impact of one’s research. Steps include: establishing an ORCID (Open Researcher and Contributor ID) account, creating research profiles on academic platforms, engaging in broad dissemination activities, harvesting bibliometric and altmetric data, adding bibliometric and altmetric information to one’s curriculum vitae, and submitting one’s promotion and tenure portfolio with confidence”. A discussion on how a researcher’s department has modified the incentive structure to support open science and replication work as well as innovation is provided by Lundwall et al.²⁹ The author describes the designing of “incentive changes to support faculty in both quantity (i.e., publication counts) and quality (i.e., … the journal impact factors, journal rejection rates, and newly added methods that support sound science) efforts … using a self-rating rubric … to guide faculty to succeed”. Strintzel et al.⁴⁴ proposed “ways of improving academic CVs for fairer research assessment”. Another study²⁹ proposed “a novel policy to provide incentives for open science … by offering open-source (OS)-endowed professorships”. Fischman et al.⁴¹ outlined “an alternative proposal in which trustworthiness and usability of research would complement traditional metrics of scholarly relevance”⁴¹ mainly in institutions.

The ramifications of research promotion and tenure scholarship ecosystem is of significant societal impact. The policy document by the European Commission⁴³ on the “evaluation of research careers acknowledging open science practices” also speaks to the recognition of international bodies in changing current practices. Moher et al.³⁸ highlights the importance of the proposed interventions for open sciences practices in relation to COVID-19.

The role of stakeholders is evident in a recent publication by Bonn et al.³⁹ captured the perspectives of research stakeholders regarding implementing changes over a five-year period. They translated their findings into four actions needed for fostering better research, with one of the action points relating to research assessment.

For the studies included in our scoping review, we extracted data on whether or not any of the proposed interventions were implemented (Table 3 – underlying data).⁶³ Most of the studies (n = 20) did not discuss implementation of their proposed intervention. The remainder of the studies (n = 7) discussed ‘near’ implementation strategies such as, changes in how junior faculty members are identified and recruited at the author’s department, piloting and an audit and feedback process for their proposed open science intervention, and discussions on initiatives of various stakeholders for more responsible assessment of higher education (scholarship) for the society. Of the 27 included articles in our scoping review, 10 were cited in policy documents, with The Leiden Manifesto being cited the most often (n = 104; Table 4 – underlying data).⁶³