Top health research funders’ guidance on selecting journals for funded research

Funders & open access

Most major health research funders mandate that funded research outputs be open access⁹. Open access mandates typically require researchers to ensure that research (and sometimes data) is published in an open access journal or is deposited in a publicly accessible digital repository (regardless of whether the publication was published in an open access journal), or both. Some journals may impose an embargo period only after which an article is made publicly available or can be archived in a repository (i.e., delayed-access journals). Many funders’ policies allow for such delays in open access to accommodate publishers’ preferences.

Open access policies are one way for funders to direct funded researchers towards publishing in credible journals abiding by established open access tenets¹⁰:

To facilitate researcher adherence with funder open access policies, many biomedical journals offering open access have agreements with the PubMed Central (PMC) repository to automatically deposit their published content on authors’ behalf¹¹. Additionally, researchers funded by the NIH and 13 partner funding organizations in the USA can upload funder-supported publications to PMC from journals without PMC agreements¹². Likewise, 29 funders from across Europe can submit funder-supported research to Europe PMC (which is mirrored in PMC)¹³. For some of these organizations, such as the NIH and Wellcome Trust, archiving in PMC or Europe PMC, respectively, is mandatory.

In a possible attempt to attract submissions, predatory journals appear to market themselves as ‘open access’^14,15. While research in them may indeed be free to access, discovery of their content in scientifically-curated databases is sparse and inconsistent^16–18. Predatory journal articles may haphazardly appear in search engines such as Google Scholar (which indexes anything that appears formatted as a scholarly article) or in PubMed (since it includes author-uploaded articles from PMC)¹⁹. Additionally, we do not know whether the contents of unindexed/unarchived journals will be perpetually available if a journal ceases to operate. Such preservation is typically achieved through journal/publisher agreements with digital preservation providers (e.g. Lots of Copies Keep Stuff Safe, LOCKSS). For journals indexed in Medline, for example, this is a prerequisite of indexing²⁰; PMC functions as a preservation service (i.e., has a remit to preserve content funded by public money)²¹. It is unknown whether predatory journals, not formally indexed in Medline, PMC, or other databases with similar requirements, have digital preservation arrangements.

Most researchers have a limited understanding of what open access means beyond making research free to read^22–25. Free use and unrestricted re-use of research is a fundamental component of open access, and licensing that permits this is a regular component of open access journals²⁶. Journals running nefarious and deceptive publishing operations have likely benefited from or exploited authors’ lack of awareness²⁷. Indeed, few predatory journals mention licensing for articles or provide information on the use and re-use of published research²⁶. Without explicit licensing for published articles, the legalities around distributing or building on research in predatory journals, for example, is uncertain. Whether researchers are deceived by predatory journals or are knowingly seeking easy and rapid publications in them (these journals tend to deliver quicker turnaround time than credible journals due to subpar or non-existent peer review^28,29), they are likely breaching the open access policies set by their funders.

In January 2017, the Bill & Melinda Gates Foundation implemented a policy mandating open access to research publications and data, without delay for all funded research³⁰. In February 2017, they initiated a one-year partnership pilot with the American Association for the Advancement of Science (AAAS) to enable Gates-funded research to be published as open access in five AAAS journals, including Science³¹. The Gates-AAAS partnership seemed to inspire several other influential journals (i.e., New England Journal of Medicine, Proceedings of the National Academy of Sciences) to introduce policies ensuring permanent open access for Gates-funded research³².

In January 2021, a number of international funders (including UK Research and Innovation, the Gates Foundation, Wellcome Trust, and the World Health Organization), led by Science Europe (a group representing funders across Europe), delivered a radical change to the way that funded research is published, via Plan S (coalition-s.org). Plan S, in part, requires research funders to mandate open access to funded research through publication in an open access journal or platform; requiring publications to be immediately available through an open access repository upon publication. To support this, agreed funders will pay the cost of article publishing charges (APCs) (up to a yet unannounced limit) to journals that are immediately and wholly open access (sometimes referred to as ‘gold’ open access).

Whether health research funding bodies, prior to Plan S, provide funded researchers with guidance or support towards selecting publishing journals in line with their policies and which facilitate proper (and permanent) access to research, and whether they monitor such policies, is unknown. Previous studies confirm that many non-commercial health research funders’ have policies requiring open access to completed research or results via publication or otherwise^33,34. Yet none seem to have assessed whether funders provide any specific information to researchers to facilitate their choice of publishing journal. For public or charitable funders, providing such guidance or support may be one way of ensuring responsible stewardship of public or donor funds. While research publication routes exist beyond scientific journals (e.g., preprint servers, repositories) the present project examines journals as the primary vehicles of research dissemination due to funders’ and academia’s reliance on them as a gauge of research impact/productivity. The current work will establish a pre-Plan S baseline of health research funders’ guidance on selecting journals in which to publish funded research.

The aim of this research is to describe the policies and recommendations of major health research funding bodies regarding suitable journals for funded research.

Data source

Global funding bodies with the largest documented health research expenditures were sampled from the curated Health Research Funder’s list found at: www.healthresearchfunders.org³⁵. The list was developed as part of an unfunded post-doctoral project by researchers in the Netherlands³⁵. It was last updated in 2016; expenditure data are reported in 2013 US dollars (USD, accounting for inflation/deflation rates by country). A detailed account of the systematic process used to identify funders and obtain expenditure data is found here: http://www.healthresearchfunders.org/faq/. At the time of retrieval for this study (August 2017), 287 health research funding bodies from 30 countries were included on the list. The list distinguishes five categories of funders: [1] philanthropic funders (n=194), [2] public funders (n=77), [3] public funders who fund health research through Official Development Assistance (public ODA)¹ (n=8), [4] multilateral funders (funding across countries) (n=7), and [5] public-private partnerships (PPP) (n=1) (Table 1). While there are some inequities in its coverage (e.g. public funders were selected from only G20 countries; paucity of funders from low income countries), the list is likely the most comprehensive source of global health research funder expenditure information in existence (personal communication, Dr. Beverley Holmes, CEO, Michael Smith Foundation for Health Research) and has been used to construct samples in at least two other studies^34,36. This study excludes commercial funders since their expenditure data are not publicly or readily available.

Sampling

To construct our sample, we sought up to 15 funders with the largest expenditures from each of the five funder categories from the list at www.healthresearchfunders.org, and aimed to include all listed lower income countries (n=4) if they were not otherwise represented in the sample. We included the latter group of funders in order to ensure representation from lower income countries, since researchers and journals from these countries have been disproportionately implicated in predatory publishing^26,37. Working with the available number of funders in each category (Table 1), we ended up with 50 funders: 15 philanthropic, 15 public, eight public ODA, seven multilateral, one PPP, and four lower-middle income country funders.

In line with previous investigations into health research funder policies^34,38, we expected that guidance for funded researchers would be publicly available and easily obtained. For each included funder, one of us (LS) visited the website using the URL provided by www.healthresearchfunders.org, or if the URL was not working, found it through a Google search using the funder name. When a funder’s website could not be located/did not work or when the funder was a duplicate, the next largest funder on the list was used. For each funder, we sought and downloaded the website section on policies for funded research in August-September 2017. If no specific policies were found, we searched the SHERPA (Securing a Hybrid Environment for Research Preservation and Access)/Juliet database (www.sherpa.ac.uk/juliet/index.php), which lists and links conditions for open access publication for some funders (though this is incomplete as it is reliant on voluntary contributions from funders and other organizations [e.g., libraries] tracking funder policies). If a funder’s website did not mention funding health research (i.e., funded other scientific research) or if the funder did not specifically award grants for research, the funding body was excluded from the sample and replaced with the next largest funder (by expenditure), where possible. Reasons for exclusion are documented in Figure 1.

Figure 1. Flow diagram for selection of included funders.

Data extraction

One assessor (LS) extracted information from the downloaded policy documents into an online form in Distiller SR, and a second assessor (KDC or MJP) verified the extracted data. Discrepancies were resolved by discussion. If additional documents were identified during extraction, we saved them and searched them for the desired data. The verification process led to clarifications in collected data or provided additional information. Since no reference standard for funder policies on publishing exists, the extracted items were derived de novo by the study team; no formal consensus process was used. The following information was assessed or extracted, as available:

If non-English websites or documents were encountered and an English-language version was not available on the website, Google Translate was used to automatically translate the websites and documents. Google Translate has recently shown 85%–97% accuracy across nine languages for translating health research studies³⁹, including the languages encountered in this study (French and German).

Data analysis

We summarized data descriptively by calculating proportions for dichotomous data; the date of funder policies/recommendations were summarized as medians and interquartile range.

Protocol deviations

In the protocol for this study, we stated that we wanted to determine whether there were differences in the number of funders with policies/statements about journal quality and predatory publishing based on the income-level of the funder country or country being funded. However, as only four funders from lower-middle income countries and none from low income countries were on the list we sampled from, there were not enough funders to enable meaningful comparisons between higher income and lower income countries.

Open access and journal selection

36 of 38 policies/recommendations (95%) specifically referred to publication in a journal as one form of dissemination for completed research (Table 4). 31 of these (86%) mentioned that research should be open access, either through journal publication (n=24, 77%) or through self-archiving the final report or accepted manuscript in a freely accessible repository (such as PMC) (n=30, 97%). One funder from India (Indian Council of Medical Research), one from France (Institut National de la Santé Et de la Recherche Médicale, INSERM), and three from the USA (US Department of Defense, Congressionally Directed Medical Research Program, and the American Cancer Society) did not mention open access in their policies about research dissemination.

13 of 36 (36%) policies recommending publication contained some guidance on how to select a journal and six (17%) listed features or requirements of publishers or journals for researchers to look for (Table 5). These six are described here. Only one funder policy (NIH) included a definition of a journal (i.e., either a publication listed in the journal section of the National Library of Medicine or one meeting stated criteria). And only one funder policy (Canadian International Development Research Council, IDRC) appeared to provide any information about ‘questionable’ journals in a guidance document entitled “Publishing in Open Access Journals”. The document lists journal features to “look for” and to “be wary of” and mentions Beall’s List³ as a resource (Table 5). One policy (Deutsche Forschungsgemeinschaft/German Research Foundation, DFG) linked to Think, Check, Submit (www.thinkchecksubmit.org) – an initiative to facilitate researchers’ assessment of the credentials of a journal – on a page supplementing their open access policy listing open access resources. Two funders distributing APC fees through the Charitable Open Access Fund (Cancer Research UK and Wellcome Trust) list the requirements of journals whose APCs are eligible for payment through the fund. The Bill and Melinda Gates Foundation provide researchers with a portal (called Chronos) through which to submit manuscripts directly to pre-selected journals whose standards are in line with their requirements.

The policies of at least three funders (German Federal Ministry of Education and Research Indian [BMBF], Indian Department of Biotechnology [DBT], Indian Department of Science & Technology [DST]) include a statement that further to making research freely accessible, researchers’ choice of journal was unrestricted.

Other journal characteristics mentioned by funders

Most funders mentioned that funded research should be peer reviewed or published in a peer reviewed journal (Table 4). Four funders made non-specific reference to selecting a “good” or “quality” journal in relation to publication of funded research; none mentioned the journal impact factor. Eight funders made statements about publication transparency or ethics. For instance, one funder discussed reproducibility in published research, three mentioned adherence to reporting guidelines, and at least six asked that metadata accompany published articles (Table 6).

Adherence to policies/recommendations

Of 38 policies/recommendations providing information about disseminating research outputs, only nine (24%) stated that they monitor adherence to either a policy (n=7) or recommendation (n=2); two philanthropic funders (Wellcome Trust and Bill & Melinda Gates Foundation) specified that they would evaluate publications of funded research reported to them to ensure they are published in journals meeting the funder’s outlined publishing requirements (Table 7). No monitoring or adherence data appears to be publicly available. Only five (13%) funders with policies or recommendations about journal publication indicated that there would be consequences for non-adherence. And only two of those (Wellcome Trust and NIH) stated that they would withhold or suspend payments if articles are not made open access.

Comparison to other research

At least four previous studies examining health research funder policies on clinical trial transparency have collected information on funder’s recommendations for disseminating research.

Two studies using similar methods evaluated trial transparency policies (i.e., those related to trial registration, access to summary results, individual data availability) for non-commercial health research funders globally (n=18)³⁴ and in the USA (n=9)³⁶. After accounting for three common funders across studies, 21 of 24 (87.5%) funders (16 of which are represented in our study) either required or supported publication or registration of trial results (neither study or their available data distinguished between publication or registration). This is in line with our findings in which 86% (38 of 44) funders had such policies/recommendations.

A third study, published in 2017 which examined research waste-reducing policies of 11 non-commercial funders (six of which are represented in our study) reported six to be explicit in requiring publication of full reports of funded research³³. In comparison, 36 of 38 policies/recommendations (95%) in our study referred to journal publication as one form of dissemination for completed research but did not indicate that it was mandatory. There may be differences in how authors of that study and interpreted language in documents or policies. The names of the six funders ‘requiring’ publication in that study were not obvious in either the publication or available data, so we are unable to investigate this further.

A study published in 2008 examined 73 UK and international non-commercial health research funders’ guidance for reporting funded clinical trials⁴². 49 funders (67%) explicitly stated that trials could or should be published. Of the three funders appearing in the 2008 sample and ours, all have maintained recommending (but not requiring) the publication of trial results. Whether funders provided any guidance on selecting a journal to publish in was not collected in the study.

No previous studies appear to have investigated whether health research funders’ provide guidance to help funded researchers select a journal for publication. Our study appears to be the most comprehensive investigation on this matter. This is surprising since our findings suggest that funders in our sample regard publication as the primary means of disseminating funded research. Further, studies show that researchers view journal publication as the primary way of disseminating research^43,44.

Strengths and limitations

This study is the first to examine the information funders provide researchers about selecting a journal in which to publish funded research. All funders in our sample that mention journal publication or provide guidance on selecting journals, do so within their open access policies. In a time where the scholarly publishing landscape has been infiltrated and confused by predatory journals, inadvertently resulting in some researchers trying to achieve open access to publish in predatory journals⁴⁵, funders can play a critical role in steering researchers in the right direction. Funders can be specific and explicit with regards to which journal features researchers should look for in order to select one that meets their open access requirements.

This study provides a benchmark by which to monitor how major health research funders are performing pre and post Plan S implementation (January 2021). Data collection occurred in August & September 2017, prior to the September 2018 announcement of Plan S. So far, 24 funders have committed to implementing Plan S, five of which were considered in this research (European Commission, Gates Foundation, MRC/UK Research and Innovation, Wellcome Trust, and the World Health Organization). Two of these, the Wellcome Trust and Gates Foundation, provided guidance (in the form of tools) to facilitate selecting a journal in line with their open access policies at the time of sampling. At least one funder (Wellcome Trust) has made changes to their open access policies in anticipation of Plan S⁴⁶.

Our study relied on publicly available information about funder expectations of funded research and was abstracted by a single person with verification by a second (i.e., not two independent people). Six funders in our sample did not provide any relevant public information. We did not seek verification on policies from funders. Data were collected at a time when publishing activities, particularly open access, was rapidly changing, in part in response to funded research being published in predatory journals^45,47. We are aware that the NIH issued a notice on their Public Access Policy in November 2017 (outside of our sampling and data collection period) with recommendations to publish funded research in journals with ‘credible practices’⁴⁸. Engaging funders in our study may have had the added benefit of increasing uptake of our findings/recommendations into practice.

The focus of this research is limited to health research funders. We have not accounted for or evaluated other potential scientific publishing gatekeepers such as academic institutions, governments, or companies carrying out scholarly research, despite the important role they can play⁴⁹.

Implications and recommendations for funders

Explicit funder policies on publication expectations. Selecting a journal in which to publish research is not a straightforward undertaking⁴⁰, particularly since the emergence of predatory journals. For funders looking to make their expectations around publishing funded research more explicit and more transparent, we propose several recommendations on how this might be achieved in Box 1, based on findings of this research and on the expertise of authors. Providing specific information about journal considerations in funders’ policies to funded researchers may facilitate more thoughtful and responsible selection of journals. Several recommendations in Box 1 pertain to the explicitness of article/journal considerations mentioned in Plan S (e.g., persistent identifiers for publications; long-term preservation/archiving; article-level metadata). All health research funders may wish to consider making aspects of their policies that pertain to publishing more explicit, whether or not they intend to implement Plan S.

The NIH is the only funder in our sample to clearly describe what it considers a journal – either those listed in the journal section of the National Library of Medicine (NLM) (https://www.ncbi.nlm.nih.gov/nlmcatalog/journals) or those meeting a comprehensive set of criteria⁴¹: (1) meets the requirements for ISSN (International Standard Serial Number) assignment; (2) content is issued over time under a common title; (3) is a collection of articles by different authors; and (4) is intended to be published indefinitely. All but the final criterion are straightforward to judge; presumably it is meant to distinguish a journal from a book or a monograph however NIH or NLM do not provide guidance on how to judge this criterion. Whether and how we can predict journals’ intentions to publish indefinitely has not been described. A more meaningful criterion for distinguishing journals from non-journals may be whether the publishing entity has archival arrangements in place (e.g., with LOCKSS, Portico, PubMed Central) to ensure perpetual access to content in the event a journal ceases to operate. Since preserving publisher content may have associated costs⁵⁰, predatory or non-credible journals (which some describe as “primarily fee-collecting operations”⁵¹) may be unlikely to seek this service.

We surprised that the three funders from India in our sample (Indian Council for Medical Research, DBT, and DST) did not mention journal credibility or predatory journals, and further, that a common policy for two Indian funders (DST and DBT), dated December 2014, recognizes “the right of researchers to publish their work in journals of their choice, because researchers are the best judges of where to publish their work”. Since at least 2016, there has been an ongoing national initiative combat predatory journals and to support researchers in their choice of journals across higher education institutes in India. The main product of this work has been a list of approved journals in which academic researchers are permitted to publish in as well as standard templates for researchers when communicating with journals⁵². The University Grants Commission (UGC), the regulator and funder of high education, has been leading the initiative. It is uncertain whether the country’s largest health research funders are on board due to their lack of guidance in this space. A coordinated approach by a range of stakeholder groups⁴⁹, which includes funders (who have innate authority to implement mandates about publishing), may facilitate improved publication decisions by researchers. Importantly, however, UGC’s list of approved journals has been plagued with numerous credibility concerns in its short existence^53,54. Explicit recommendations from India’s funders regarding credible and non-credible features of journals in which to publish may be warranted in the absence of a trusted and comprehensive list.

Facilitating and monitoring adherence to funder policies. Funders are well-positioned to provide researchers with resources and tools to help ensure that results from funded research are published in credible and discoverable journals, in line with their policies. Several organizations in our sample consistently offer more information about potential publishing routes and tools to facilitate adherence to their policies. We provide a list of tools to facilitate the development of funder policies on research outputs, adherence to such policies, and monitoring of policy adherence (Table 8).

Monitoring researchers’ adherence to their policies may help funders understand the extent to which researcher’s publishing practices are guided by their policies⁵⁵. Informing researchers that their adherence to open access policies is being monitored may facilitate better awareness of such policies and potentially increase adherence to them⁵⁶. A 2018 study examining the accessibility of research supported by 12 research funding agencies across North America and Europe with open access policies, found that 62% of almost 1.3 million articles over nine years were freely available⁹. In 2016, 90% of published research supported by the NIH and Wellcome Trust was free to access (via journal, repository, or both)⁹. Both agencies mandate the deposit of published research by publishing journals or funded authors into dedicated repositories (PMC for NIH; PMC Europe for Wellcome Trust). The remaining 10 funders in the sample did not mandate depositing in a repository alongside publication and had lower rates of freely accessible articles. For example, for the Canadian Institutes of Health Research (CIHR) only 55% of published research was freely accessible in 2016, even though the funder had a dedicated repository (PMC Canada) until 2018 (it closed due to low usage and high upkeep costs)⁵⁷. The study’s authors conclude that funders with low compliance rates used less enforcement and had less infrastructure to support compliance with their open access mandates⁹.

Areas of future research

An important area of future study is whether researchers are being funded on the basis of grant applications that include research published in predatory journals – or in journals that may not be indexed in trusted databases. Predatory journals have made their way into consideration (via CVs submitted by researchers or through institution-initiated database searches) into applications for academic career advancement^58–60. Some have called for such publications to either be discounted from consideration or for researchers who submit them for consideration to be prevented from career advancement overall^61,62. It is unknown whether researchers are including publications in predatory journals as part of their funding applications. This should be evaluated. If they are, funders may wish to consider whether this is an important consideration for awarding funding.

Underlying data

Open Science Framework: Audit of health research funder policies and recommendations on journal publication of research: Extracted Data, https://doi.org/10.17605/OSF.IO/YUDP4⁶³.

This project contains the following underlying data:

Extended data

Open Science Framework: Audit of health research funder policies and recommendations on journal publication of research: Study Forms, https://doi.org/10.17605/OSF.IO/FSUQ2⁶⁴

This project contains the following extended data:

Open Science Framework: Audit of health research funder policies and recommendations on journal publication of research: Protocol, https://doi.org/10.17605/OSF.IO/J6CSK⁶⁵

Registration of overarching OSF project: https://doi.org/10.17605/OSF.IO/Z59U6⁶⁶.

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).