Top health research funders’ guidance on selecting journals for funded research

Background: Funded health research is being published in journals that many regard as “predatory”, deceptive, and non-credible. We do not currently know whether funders provide guidance on how to select a journal in which to publish funded health research. Methods: We identified the largest 46 philanthropic, public, development assistance, public-private partnership, and multilateral funders of health research by expenditure, globally as well as four public funders from lower-middle income countries, from the list at https://healthresearchfunders.org. One of us identified guidance on disseminating funded research from each funders’ website (August/September 2017), then extracted information about selecting journals, which was verified by another assessor. Discrepancies were resolved by discussion. Results were summarized descriptively. This research used publicly available information; we did not seek verification with funding bodies. Results: The majority (44/50) of sampled funders indicated funding health research. 38 (of 44, 86%) had publicly available information about disseminating funded research, typically called “policies” (29, 76%). Of these 38, 36 (95%) mentioned journal publication for dissemination of which 13 (36.11%) offer variable guidance on selecting a journal, all of which relate to the funder’s open access mandate. Six funders (17%) outlined publisher requirements or features by which to select a journal. One funder linked to a document providing features of journals to look for (e.g. listed in the Directory of Open Access Journals) and to be wary of (e.g., no journal scope statement, uses direct and unsolicited marketing). Conclusions: Few funders provided guidance on how to select a journal in which to publish funded research. Funders have a duty to ensure that the research they fund is discoverable by others. This research is a benchmark for funder guidance on journal selection prior to the January 2021 implementation of Plan S (a global, funder-led initiative to ensure immediate, open access to funded, published research).


Introduction
Biomedical research studies supported by well-known funding organizations such as the National Institutes of Health (NIH), are published in so-called "predatory" journals 1 . Predatory journals are regarded as non-credible and are criticized for failing to provide typical or expected publishing services and their lack of transparent operations 2,3 . Such services include peer review, long term preservation of content, and indexing in scientific, bibliographic databases. Among their many shortcomings, the potential failure of predatory journals to ensure permanent discoverability of research threatens the integrity of the scientific record. Such research cannot contribute to science, thus wasting time, money, and resources 1,4 . Even if discovered, the potential impact and uptake of funded research in predatory journals may be limited due to being published in a perceived untrustworthy source. While benefits from investments in research are difficult to quantify 5 . One way funders measure returns on investments is by tracking research outputs, including scholarly journal publications 6 . Predatory journals may limit returns on funders' investments by undermining the intended promise of scholarly publishing -to enable the results of research to be known for others to build upon 7 .
Health research funders ought to be concerned that the funds they provide may be wasted or contribute to research waste as a result of funded research being published in predatory journals. They may be supporting research that is not identifiable or able to be found if published in predatory journals, potentially wasting millions of dollars of research funding. When research is easily identifiable it can reduce unintentional redundancies in research efforts and investments. Additional wastage occurs when funder investments are used to pay for article processing charges (APCs). In biomedicine, research grants and national funding agencies are the largest source of funds supporting publication of at least 50% of open access articles 8 .

Funders & open access
Most major health research funders mandate that funded research outputs be open access 9 . Open access mandates typically require researchers to ensure that research (and sometimes data) is published in an open access journal or is deposited in a publicly accessible digital repository (regardless of whether the publication was published in an open access journal), or both. Some journals may impose an embargo period only after which an article is made publicly available or can be archived in a repository (i.e., delayed-access journals). Many funders' policies allow for such delays in open access to accommodate publishers' preferences.
Open access policies are one way for funders to direct funded researchers towards publishing in credible journals abiding by established open access tenets 10 : 1. Research is/should be freely available and accessible to anyone.

2.
The copyright accompanying published work should be open, allowing for free use and re-use (i.e., allowing research to be freely built on/adapted with attribution).
To facilitate researcher adherence with funder open access policies, many biomedical journals offering open access have agreements with the PubMed Central (PMC) repository to automatically deposit their published content on authors' behalf 11 . Additionally, researchers funded by the NIH and 13 partner funding organizations in the USA can upload funder-supported publications to PMC from journals without PMC agreements 12 . Likewise, 29 funders from across Europe can submit fundersupported research to Europe PMC (which is mirrored in PMC) 13 . For some of these organizations, such as the NIH and Wellcome Trust, archiving in PMC or Europe PMC, respectively, is mandatory.
In a possible attempt to attract submissions, predatory journals appear to market themselves as 'open access' 14,15 . While research in them may indeed be free to access, discovery of their content in scientifically-curated databases is sparse and inconsistent [16][17][18] . Predatory journal articles may haphazardly appear in search engines such as Google Scholar (which indexes anything that appears formatted as a scholarly article) or in PubMed (since it includes author-uploaded articles from PMC) 19 . Additionally, we do not know whether the contents of unindexed/unarchived journals will be perpetually available if a journal ceases to operate. Such preservation is typically achieved through journal/publisher agreements with digital preservation providers (e.g. Lots of Copies Keep Stuff Safe, LOCKSS). For journals indexed in Medline, for example, this is a prerequisite of indexing 20 ; PMC functions as a preservation service (i.e., has a remit to preserve content funded by public money) 21 . It is unknown whether predatory journals, not formally indexed in Medline, PMC, or other databases with similar requirements, have digital preservation arrangements.
Most researchers have a limited understanding of what open access means beyond making research free to read [22][23][24][25] . Free use and unrestricted re-use of research is a fundamental component of open access, and licensing that permits this is a regular component of open access journals 26 . Journals running nefarious and deceptive publishing operations have likely benefited from or exploited authors' lack of awareness 27 . Indeed, few predatory journals mention licensing for articles or provide information on the use and re-use of published research 26 . Without explicit licensing for published articles, the legalities around distributing or building on research in predatory journals, for example, is uncertain. Whether researchers are deceived by predatory journals or are knowingly seeking easy and rapid publications in them (these journals tend to deliver quicker turnaround time than credible journals due to subpar or non-existent peer review 28 In January 2021, a number of international funders (including UK Research and Innovation, the Gates Foundation, Wellcome Trust, and the World Health Organization), led by Science Europe (a group representing funders across Europe), delivered a radical change to the way that funded research is published, via Plan S (coalition-s.org). Plan S, in part, requires research funders to mandate open access to funded research through publication in an open access journal or platform; requiring publications to be immediately available through an open access repository upon publication. To support this, agreed funders will pay the cost of article publishing charges (APCs) (up to a yet unannounced limit) to journals that are immediately and wholly open access (sometimes referred to as 'gold' open access).
Whether health research funding bodies, prior to Plan S, provide funded researchers with guidance or support towards selecting publishing journals in line with their policies and which facilitate proper (and permanent) access to research, and whether they monitor such policies, is unknown. Previous studies confirm that many non-commercial health research funders ' have policies requiring open access to completed research or results via publication or otherwise 33,34 . Yet none seem to have assessed whether funders provide any specific information to researchers to facilitate their choice of publishing journal. For public or charitable funders, providing such guidance or support may be one way of ensuring responsible stewardship of public or donor funds. While research publication routes exist beyond scientific journals (e.g., preprint servers, repositories) the present project examines journals as the primary vehicles of research dissemination due to funders' and academia's reliance on them as a gauge of research impact/productivity. The current work will establish a pre-Plan S baseline of health research funders' guidance on selecting journals in which to publish funded research.
The aim of this research is to describe the policies and recommendations of major health research funding bodies regarding suitable journals for funded research.

Methods
We considered the public websites of 50 health funding bodies for statements, guidance, or policies specifically mentioning the publication of funded research. Detailed methods and rationale for this study are elaborated in an a priori study protocol (https://doi.org/10.17605/OSF.IO/J6CSK) and summarized below. In line with previous investigations into health research funder policies 34,38 , we expected that guidance for funded researchers would be publicly available and easily obtained. For each included funder, one of us (LS) visited the website using the URL provided by www.healthresearchfunders.org, or if the URL was not working, found it through a Google search using the funder name. When a funder's website could not be located/did not work or when the funder was a duplicate, the next largest funder on the list was used. For each funder, we sought and downloaded the website section on policies for funded research in August-September 2017. If no specific policies were found, we searched the SHERPA (Securing a Hybrid Environment for Research Preservation and Access)/Juliet database (www.sherpa.ac.uk/juliet/index.php), which lists and links conditions for open access publication for some funders (though this is incomplete as it is reliant on voluntary contributions from funders and other organizations [e.g., libraries] tracking funder policies). If a funder's website did not mention funding health research (i.e., funded other scientific research) or if the funder did not specifically award grants for research, the funding body was excluded from the sample and replaced with the next largest funder (by expenditure), where possible. Reasons for exclusion are documented in Figure 1.

Data extraction
One assessor (LS) extracted information from the downloaded policy documents into an online form in Distiller SR, and a second assessor (KDC or MJP) verified the extracted data. Discrepancies were resolved by discussion. If additional documents were identified during extraction, we saved them and searched them for the desired data. The verification process led to clarifications in collected data or provided additional information. Since no reference standard for funder policies on publishing exists, the extracted items were derived de novo by the study team; no formal consensus process was used. The following information was assessed or extracted, as available: • Any statement(s) about the dissemination of outputs from funded research

Data analysis
We summarized data descriptively by calculating proportions for dichotomous data; the date of funder policies/recommendations were summarized as medians and interquartile range. 2 Any European Commission funding program that explicitly stated using the European Horizon 2020 guidelines for grantees were jointly represented as "European Commission". ( Figure 1)

Protocol deviations
In the protocol for this study, we stated that we wanted to determine whether there were differences in the number of funders with policies/statements about journal quality and predatory publishing based on the income-level of the funder country or country being funded. However, as only four funders from lower-middle income countries and none from low income countries were on the list we sampled from, there were not enough funders to enable meaningful comparisons between higher income and lower income countries.

Results
For the 50 funding bodies originally identified using the described sampling technique, three allocated money from a funder (European Commission 2 ) already in the sample and were replaced with the next organizations on the list. One of the replacement funders also allocated money from an included funder and was also replaced. Two funders funded non-health research and four funders did not list any research grants (and appeared to fund health development initiatives) and could not be evaluated for our purposes. Overall, six funders were excluded and lacked replacements in the categories they belonged to. 44 funders remained in the sample for which grant policies and guidelines were sought ( Figure 1). 35 funders are from high income countries, one from upper-middle income (China), three are from a lower-middle income country (India), and five are not classified by income level because they are multilateral (n=3) or fund across the European Union (EU) (n=2, Table 2).
38 of 44 funders (86%) had publicly available information for grantees about disseminating funded research outputs (Table 3).
Of the six funders that did not have publicly available information, five are from high-income countries (US, Germany, France, UK) and one funds research in the EU through publicprivate partnership. Three are philanthropic organizations and two are public-ODA funders. Information about disseminating research was contained within "policies" for 29/38 (76%) funders, "recommendations" (suggestions and guidance) for 8/38 (21%) of funders, and as a "code of conduct" for one funder (Table 3). All but one policy/recommendation referred to funded research (including results) as the unit of dissemination (37/38, 97%). Over a third of policies/recommendations also specifically mentioned the dissemination of "data" (25/38, 66%). The median implementation date or date listed on collected documents was September 2014 (IQR: Apr 2012 to Apr 2016, n=35).
Open access and journal selection 36 of 38 policies/recommendations (95%) specifically referred to publication in a journal as one form of dissemination for completed research ( 13 of 36 (36%) policies recommending publication contained some guidance on how to select a journal and six (17%) listed features or requirements of publishers or journals for researchers to look for (Table 5). These six are described here. Only one funder policy (NIH) included a definition of a journal (i.e., either a publication listed in the journal section of the National Library of Medicine or one meeting stated criteria). And only one funder policy (Canadian International Development Research Council, IDRC) appeared to provide any information about 'questionable' journals in a guidance document entitled "Publishing in Open Access Journals". The document lists journal features to "look for" and to "be wary of" and mentions Beall's List 3 as a resource (   c Policy: uses the words "policy", "must", "require"; Recommendation/ Guideline: uses the words "recommendation" "recommend", "suggest", "should", "guideline". When only year was given, January was used as default month; when a date range was given the most recent date was used.   Table 5 for verbatim text of statements about journal selection. 3 See Table 5 for verbatim text of statements about journal credibility. 4 excluding JIF. 5 2 funders indicate journal should be "high quality, peer reviewed journal"; 1 funder indicates journal should be "quality peer-reviewed journal"; 1 funder indicates what a good journal is: "Good journals have guidelines for reviewers of manuscripts committing them to strict confidentiality/ to disclose conflicts of interest and promise to respond to submitted manuscripts within a specified, short time limit, and correspondingly set their reviewers short time limits for their comments." 6 See Table 6 for transparency or ethics standards for publications. 7 Other: 1 funder encourages publication in "primary scientific journals"; 1 funder states "models and mechanisms for publication and access to research must be both efficient and cost effective";.

Other journal characteristics mentioned by funders
Most funders mentioned that funded research should be peer reviewed or published in a peer reviewed journal (Table 4). Four funders made non-specific reference to selecting a "good" or "quality" journal in relation to publication of funded research; none mentioned the journal impact factor. Eight funders made statements about publication transparency or ethics. For instance, one funder discussed reproducibility in published research, three mentioned adherence to reporting guidelines, and at least six asked that metadata accompany published articles ( Table 6).

Adherence to policies/recommendations
Of 38 policies/recommendations providing information about disseminating research outputs, only nine (24%) stated that they monitor adherence to either a policy (n=7) or recommendation (n=2); two philanthropic funders (Wellcome Trust and Bill & Melinda Gates Foundation) specified that they would evaluate publications of funded research reported to them to ensure they are published in journals meeting the funder's outlined publishing requirements (Table 7). No monitoring or adherence data appears to be publicly available. Only five (13%) funders with policies or recommendations about journal publication indicated that there would be consequences for non-adherence. And only two of those (Wellcome Trust and NIH) stated that they would withhold or suspend payments if articles are not made open access.

Discussion
Most health research funders appear to have active policies about the dissemination of funded research, typically about open access which often include statements about journal publication. Few policies contain guidance on how to select journals, list features of journals meeting funder requirements, or about the credibility of publishing outlets. Only one health research funding organization (IDRC) made specific reference to the "questionable journals" at the time of data collection (August-September 2017). Additionally, few policies describe whether funded outputs are monitored for compliance with funders' dissemination policies, and few describe any consequences for researchers' non-adherence to their policies. Information is not available on whether the NIH or Wellcome Trust, both of whom promise to withhold or suspend grant funds for breaching their open access policies, have actually ever done so 9 .
For many of the funders in our sample, information to guide research publication was found across multiple documents and not always within open access policy statements/documents where publication is mentioned. For example, the only guidance we identified that referred to predatory journals (IDRC) was contained in a PDF (entitled "Publishing in Open Access Journals") separate from the funders' main open access policy. The policy did not flag that the document contained information about predatory/questionable/non-credible journals. This unobvious placement of guidance or expectations around journal selection relies on researchers' curiosity or knowledge that important requirements of journals whose APCs are eligible for payment through the fund. The Bill and Melinda Gates Foundation provide researchers with a portal (called Chronos) through which to submit manuscripts directly to pre-selected journals whose standards are in line with their requirements.  or an equivalent license. This will permit all users of the publication to copy and redistribute the material in any medium or format and transform and build upon the material, including for any purpose (including commercial) without further permission or fees being required.
3. Foundation Will Pay Necessary Fees. The foundation would pay reasonable fees required by a publisher to effect publication on these terms.

Publications Will Be Accessible and Open Immediately.
All publications shall be available immediately upon their publication, without any embargo period. An embargo period is the period during which the publisher will require a subscription or the payment of a fee to gain access to the publication. We are, however, providing a transition period of up to two years from the effective date of the policy (or until January 1, 2017). During the transition period, the foundation will allow publications in journals that provide up to a 12-month embargo period..." "Chronos will list the most current information about which journals offer these [open access] options, i.e., the listing for Science will reflect the current agreement between the Gates Foundation and the American Association for the Advancement of Science (AAAS)."

Open Access Policy -Frequently Asked Questions
Bloodwise "If the journal is not compliant with our open access policy, grant holders can request from Bloodwise that grant underspend is used to cover the cost with a justification for publishing in a non-compliant journal"/"Researchers can determine which publishers are compliant with this policy by referring to the SHERPA/FACT database and checking against the Wellcome Trust's policy, which has the same stipulations."

None stated
Project grants -guidance for applicants

British Heart Foundation
All of our Grantholders submitting manuscripts to journals should find out in advance whether the publisher supports open access and how they can comply with paragraph 2 above. to be published appear in journals whose contributions are all accessible to users free of charge ("genuine open access journals") immediately after their publication on the Internet, and which apply the strict quality assurance procedures recognized in the relevant subject… The funds provided by the DFG may not be used to publish open essays in journals subject to subscription (according to the "Open Choice" model etc.) for Open Access."* "Guiding ideas for choosing appropriate means of publication: http://thinkchecksubmit.org" "Recommendation 12: Scientific journals shall make it clear in their guidelines for authors that they are committed to best international practice with regard to the originality of submitted papers and the criteria for authorship."/ "all good scientific journals report when a manuscript has been received and when -usually following peer review -it has been accepted"

Further information on Open Access
German Publisher is also the editor of the journal and/or editorial boards members serve on the board of multiple titles from the same publisher

Same as Cancer Research UK
Open Access Policy/Publisher Requirements * "Open Choice" is a term used by the publisher Springer to refer to hybrid journals. Table 6. Transparency tools/activities mentioned by funders.

Bill & Melinda Gates
Foundation "Publications Are Discoverable and Accessible Online: Publications will be deposited in a specified repository(s) with proper tagging of metadata."

Bloodwise (formerly Leukaemia & Lymphoma
Research) "Grant holders should make use of the ARRIVE guidelines when designing their experiments, and ensure that they report in vivo studies in accordance with the ARRIVE guidelines as far as possible."  National Institutes of Health (NIH Public Access Policy) "Compliance with this Policy remains a statutory requirement and a term and condition of the grant award and cooperative agreement, in accordance with the NIH Grants Policy Statement." (NIH Public Access Policy -FAQ) "A grantee's failure to comply with the terms and conditions of award may cause NIH to take one or more enforcement actions, depending on the severity and duration of the non-compliance. NIH will undertake any such action in accordance with applicable statutes, regulations, and policies. NIH generally will afford the grantee an opportunity to correct the deficiencies before taking enforcement action unless public health or welfare concerns require immediate action. However, even if a grantee is taking corrective action, NIH may take proactive action to protect the Federal government's interests, including placing special conditions on awards or precluding the grantee from obtaining future awards for a specified period, or may take action designed to prevent future non-compliance, such as closer monitoring. See Enforcement Actions in the NIH Grants Policy Statement (10/12).."

Pan American
Health Organisation "Research commitments shall be reflected in institutional policies and program budgeting and planning, implementation, monitoring, and evaluation; human resource management; and knowledge management."

None stated/identified
Funder Name

Adherence-monitoring strategy
Consequences for non-adherence

None stated/identified
Wellcome Trust (From Complying with our OA policy) "We actively monitor research papers authored by our funded researchers to ensure they comply with our policy. This includes the review of publications listed in ongoing grant reporting and in end of grant reports. In addition, Wellcome-funded research papers detailed in applications submitted to us are reviewed to ensure compliance." (From Complying with our OA policy) "When Wellcome-funded researchers do not comply with our open access policy, we will apply these sanctions: Applicants will be required to ensure that all their Wellcome-funded original research papers resulting from current or previous grants are compliant before we issue formal notification of any funding renewals or new grants. Where non-compliant research papers, book chapters and scholarly monographs are identified in an end of grant report, we will withhold the final 10 per cent of the total grant budget until all outputs comply. See our 10 per cent retention policy." information may be located outside of the main policy webpages or documents. If funders wish to provide guidance about journal credibility and predatory publishing, they may reach more researchers (and increase the likelihood of them reading it) by including such information within their main policies.

Comparison to other research
At least four previous studies examining health research funder policies on clinical trial transparency have collected information on funder's recommendations for disseminating research.
Two studies using similar methods evaluated trial transparency policies (i.e., those related to trial registration, access to summary results, individual data availability) for non-commercial health research funders globally (n=18) 34 and in the USA (n=9) 36 . After accounting for three common funders across studies, 21 of 24 (87.5%) funders (16 of which are represented in our study) either required or supported publication or registration of trial results (neither study or their available data distinguished between publication or registration). This is in line with our findings in which 86% (38 of 44) funders had such policies/ recommendations.
A third study, published in 2017 which examined research waste-reducing policies of 11 non-commercial funders (six of which are represented in our study) reported six to be explicit in requiring publication of full reports of funded research 33 . In comparison, 36 of 38 policies/recommendations (95%) in our study referred to journal publication as one form of dissemination for completed research but did not indicate that it was mandatory. There may be differences in how authors of that study and interpreted language in documents or policies. The names of the six funders 'requiring' publication in that study were not obvious in either the publication or available data, so we are unable to investigate this further.
A study published in 2008 examined 73 UK and international non-commercial health research funders' guidance for reporting funded clinical trials 42 . 49 funders (67%) explicitly stated that trials could or should be published. Of the three funders appearing in the 2008 sample and ours, all have maintained recommending (but not requiring) the publication of trial results. Whether funders provided any guidance on selecting a journal to publish in was not collected in the study.
No previous studies appear to have investigated whether health research funders' provide guidance to help funded researchers select a journal for publication. Our study appears to be the most comprehensive investigation on this matter. This is surprising since our findings suggest that funders in our sample regard publication as the primary means of disseminating funded research. Further, studies show that researchers view journal publication as the primary way of disseminating research 43,44 .

Strengths and limitations
This study is the first to examine the information funders provide researchers about selecting a journal in which to publish funded research. All funders in our sample that mention journal publication or provide guidance on selecting journals, do so within their open access policies. In a time where the scholarly publishing landscape has been infiltrated and confused by predatory journals, inadvertently resulting in some researchers trying to achieve open access to publish in predatory journals 45 , funders can play a critical role in steering researchers in the right direction. Funders can be specific and explicit with regards to which journal features researchers should look for in order to select one that meets their open access requirements.
This study provides a benchmark by which to monitor how major health research funders are performing pre and post Plan S implementation (January 2021 Our study relied on publicly available information about funder expectations of funded research and was abstracted by a single person with verification by a second (i.e., not two independent people). Six funders in our sample did not provide any relevant public information. We did not seek verification on policies from funders. Data were collected at a time when publishing activities, particularly open access, was rapidly changing, in part in response to funded research being published in predatory journals 45,47 . We are aware that the NIH issued a notice on their Public Access Policy in November 2017 (outside of our sampling and data collection period) with recommendations to publish funded research in journals with 'credible practices' 48 . Engaging funders in our study may have had the added benefit of increasing uptake of our findings/ recommendations into practice.
The focus of this research is limited to health research funders. We have not accounted for or evaluated other potential scientific publishing gatekeepers such as academic institutions, governments, or companies carrying out scholarly research, despite the important role they can play 49 . • Publication content is issued over time under a common title;

Implications and recommendations for funders
• Publication is a collection of articles by different authors; • Publication is intended to be published indefinitely.
3. Indicate your agency's requirements for access and discoverability of published articles Provide instructions/link to resources on how to deposit research in the suggested repository.
6. Indicate how your agency will monitor that funded research is published in appropriate journals, in line with agency recommendations/mandates° For ease of monitoring, Provide instructions for researchers about where and how to include the funding agency name and grant number in published articles (guidance here: https://www. ukri.org/wp-content/uploads/2020/10/RIN-251020-FundersAcknowledgementInScholarlyjournalArticles.pdf)° Provide instructions on if, how, and when to submit publications of funded research to the funding agency, or state how publications will be monitored otherwise° Provide specific actions/consequences that the agency will carry out when funded research is published in a journal that does not meet agency requirements The NIH is the only funder in our sample to clearly describe what it considers a journal -either those listed in the journal section of the National Library of Medicine (NLM) (https://www. ncbi.nlm.nih.gov/nlmcatalog/journals) or those meeting a comprehensive set of criteria 41 : (1) meets the requirements for ISSN (International Standard Serial Number) assignment; (2) content is issued over time under a common title; (3) is a collection of articles by different authors; and (4) is intended to be published indefinitely. All but the final criterion are straightforward to judge; presumably it is meant to distinguish a journal from a book or a monograph however NIH or NLM do not provide guidance on how to judge this criterion. Whether and how we can predict journals' intentions to publish indefinitely has not been described. A more meaningful criterion for distinguishing journals from non-journals may be whether the publishing entity has archival arrangements in place (e.g., with LOCKSS, Portico, PubMed Central) to ensure perpetual access to content in the event a journal ceases to operate. Since preserving publisher content may have associated costs 50 , predatory or non-credible journals (which some describe as "primarily fee-collecting operations" 51 ) may be unlikely to seek this service.
We surprised that the three funders from India in our sample (Indian Council for Medical Research, DBT, and DST) did not mention journal credibility or predatory journals, and further, that a common policy for two Indian funders (DST and DBT), dated December 2014, recognizes "the right of researchers to publish their work in journals of their choice, because researchers are the best judges of where to publish their work". Since at least 2016, there has been an ongoing national initiative combat predatory journals and to support researchers in their choice of journals across higher education institutes in India.
The main product of this work has been a list of approved journals in which academic researchers are permitted to publish in as well as standard templates for researchers when communicating with journals 52 . The University Grants Commission (UGC), the regulator and funder of high education, has been leading the initiative. It is uncertain whether the country's largest health research funders are on board due to their lack of guidance in this space. A coordinated approach by a range of stakeholder groups 49 , which includes funders (who have innate authority to implement mandates about publishing), may facilitate improved publication decisions by researchers. Importantly, however, UGC's list of approved journals has been plagued with numerous credibility concerns in its short existence 53,54 . Explicit recommendations from India's funders regarding credible and non-credible features of journals in which to publish may be warranted in the absence of a trusted and comprehensive list.

Facilitating and monitoring adherence to funder policies.
Funders are well-positioned to provide researchers with resources and tools to help ensure that results from funded research are published in credible and discoverable journals, in line with their policies. Several organizations in our sample consistently offer more information about potential publishing routes and tools to facilitate adherence to their policies. We provide a list of tools to facilitate the development of funder policies on research outputs, adherence to such policies, and monitoring of policy adherence (Table 8).
Monitoring researchers' adherence to their policies may help funders understand the extent to which researcher's publishing practices are guided by their policies 55 . Informing researchers that their adherence to open access policies is being monitored may facilitate better awareness of such policies and potentially increase adherence to them 56 . A 2018 study examining the accessibility of research supported by 12 research funding agencies across North America and Europe with open access policies, found that 62% of almost 1.3 million articles over nine years were freely available 9 . In 2016, 90% of published research supported by the NIH and Wellcome Trust was free to access (via journal, repository, or both) 9 . Both agencies mandate the deposit of published research by publishing journals or funded authors into dedicated repositories (PMC for NIH; PMC Europe for Wellcome Trust). The remaining 10 funders in the sample did not mandate depositing in a repository alongside publication and had lower rates of freely accessible articles. For example, for the Canadian Institutes of Health Research (CIHR) only 55% of published research was freely accessible in 2016, even though the funder had a dedicated repository (PMC Canada) until 2018 (it closed due to low usage and high upkeep costs) 57 .
The study's authors conclude that funders with low compliance rates used less enforcement and had less infrastructure to support compliance with their open access mandates 9 .

Areas of future research
An important area of future study is whether researchers are being funded on the basis of grant applications that include research published in predatory journals -or in journals that may not be indexed in trusted databases. Predatory journals have made their way into consideration (via CVs submitted by researchers or through institution-initiated database searches) into applications for academic career advancement [58][59][60] . Some have called for such publications to either be discounted from consideration or for researchers who submit them for consideration to be prevented from career advancement overall 61,62 . It is unknown whether researchers are including publications in predatory journals as part of their funding applications. This should be evaluated. If they are, funders may wish to consider whether this is an important consideration for awarding funding. Journals that fail to make research discoverable breach the basic trust that researchers and their funders have in the current publishing system. Most funded researchers publish their work under the basic assumption that their journal or publisher is following best practices to ensure future use 7 . Bodies funding health research have a responsibility to protect their investments and even more importantly, to ensure that funded research is not wasted by being published in non-credible and non-discoverable sources.

Nicholas Devito
The DataLab, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK Many thanks to the authors for their consideration of my review and their revisions. While many of my comments were addressed, I do have some additional feedback based on the responses from the authors that I believe should be further considered. That said, these are all relatively minor enough that I recommend the manuscript move forward for indexing as the analysis and related discussion is overall fundamentally sound.
-As indicated, Dr. Holmes is the CEO of a health research funding organization (not included in our sample). Her expertise of the funding landscape and whether a listing of health research funders existed or whether the chosen list was the best source of this information, was an invaluable methodological contribution to our research. We prefer not to remove this acknowledgment.
I still believe this reads awkwardly in the text and is not how personal communication citations are typically used. If it is just an acknowledgement it is ultimately redundant to your thanks of Dr. Holmes in the "Acknowledgements" section at the end of the piece. However this is a minor point.
-Extraction criteria were devised de novo based on expertise within the author team. There is no reference standard for funder policies about publishing and as such we did not use any framework for extraction. We have added a clarification of this point in the methods section rather than in the discussion section.
The addition to the methods is appropriate although I do believe there is an argument to be made that using criteria developed in-house based on expertise, while an acceptable method, could be seen as a limitation as it is lacking further validation and consideration beyond the study team.
-We did not apply an analytical framework to the extracted text and doing so would be a post-hoc analysis. The table/text is provided for reference/transparency of extracted data summarized in Tables 2-4. We do not feel that a summary is necessary.
I do not agree that a full analytic framework is required to condense multiple tables of raw verbatim text within the body of a manuscript into something more easily digestible to readers. I still strongly recommend that the authors reconsider the utility that 7 pages of Tables in the middle of the manuscript (including 4 pages just for Table 5) has for readers when Tables 2-4 summarize the extractions from this raw data as necessary for the endpoints considered and interested parties could examine the very well-curated, transparent, and clearly sign-posted raw data for themselves as necessary.
-To be clear, we did not make a post-hoc justification regarding the lack of outreach to funders. We specifically did not have the a priori intent of verifying our data with funders. We anticipated that the results of our previous work, https://www.nature.com/news/stopthis-waste-of-people-animals-and-money-1.22554, which identified a large number of funders supporting research in predatory journals, may impact/prompt changes in funder policies once published (Sept 2017). Due to the timing of the current study, (Aug -Sept 2017), we decided not to engage funders since we did not want to influence any potential changes to their policies as a result of the study process. We did not find this to be a limitation. As expected, and as noted in our discussion, we are aware of at least one funder (NIH) who, in Nov 2017, clarified their policies about publication as a result of the previous study. The specific impacts (i.e., number and type of clarifications) of engaging funders to clarify extracted data in Goldacre et al's previous work are unclear, however, we have added text to the limitations section outlining the additional potential benefit of engaging funders in the study process.
While it is true that outreach to sponsors was not specified in the protocol, it is also true that the justification for not doing so was also not provided in the protocol despite the fact that it was used in the study cited as informing your data collection/extraction methods. Therefore, while the decision not to conduct outreach was not post hoc there is no way to know if the provided justification was or not. In my original reading the provided information was not well-supported by even your own findings. Expanding on the context provided above in your revision is helpful however there is another aspect at play here. You relied on a single author for searches and only involved a second party for extractions. There is no acknowledgement of the potential that documents could have been missed by a single searcher. Outreach to the parties under investigation would allow them to confirm whether or not you have missed potentially relevant documentation on the topics of interest. It is a validation step for your methods rather than simply a way to potentially increase "uptake of our findings/ recommendations into practice."

Nicholas Devito
The DataLab, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK Many thanks for the opportunity to review this piece on health research funders' policies on journal selection for research outputs.
Overall, this piece investigates an interesting topic, has appropriate methods, appears wellexecuted, and follows open science best principles of pre-registration and open sharing of data for which it should be commended. I ultimately believe it should be recommended for indexing but a minor revision is in order. Specific points are provided below.
Can the authors please revisit the piece for a bit of editing? The informational content all appears very relevant and well-researched but the prose can carry on at times and structurally it jumps back and forth between topics and it is difficult to connect the threads. This is most applicable to the introduction (and perhaps the Discussion as well) but I think taking a critical eye towards developing a more direct, succinct, and straight-forward writing style throughout would enhance readability. A sharper consolidation and organisation of your ideas and arguments, while trying to be more concise overall, will aid readers considerably. A brief example: "Funders ought to be concerned that funded research may be published in journals that do not ensure discoverability of content ensuring it is available to contribute to future science" could surely be consolidated to something like "Funders should ensure published research is discoverable to the scientific community." One area that doesn't really come through clearly in the Intro is what funders get out of publication of their funded work and how this influences this dynamic. Why are they moving towards Open Access? What about other dissemination routes?
The personal communication citation of Dr. Beverly Holmes feels a bit out of place. Is she merely stating an opinion? If so, why can the authors not just state this opinion directly themselves since I assume they share it? I'm sure Dr. Holmes is perfectly knowledgeable and her opinion is valuable, but is citing her necessary here? I'm most familiar with citations of personal communication to convey official information not published elsewhere, from a direct source, rather than simply used to state an opinion the authors agree with. The average reader won't have any insight as to why they should trust the word of Dr. Holmes on this issue.
The link to the protocol is upfront and clear and points readers to exactly where they can easily access more detailed information about the search/data extraction methods. Very well done by the authors.
How were the extraction criteria derived? Did the authors base them on anything or did they come up with them de novo? This should be stated and potentially explored in the Discussion.
One major area I was surprised that the authors did not include in their assessments is whether the funder offers to pay the open access/APCs of work they fund. This would be an important part of the dynamic of how, where, and why certain journals may be chosen and why funders should care even more about where the research they fund ends up.
In Figure 1, it is unclear when INSERM is mentioned as to whether that is the funder being removed or the reason another funder is being removed (later context shows it's the latter, but this is unclear). Also there is a type ("allocated") in the "Excluded Funders n=6" box. I think ideally you would list which exact funders are entering or exiting the sample in this figure for clarity as you don't get a good sense of what funders are actually being assessed until the results. It's also not necessary to restate in prose everything Figure 1 already conveys (something to be cognisant of throughout). Table 5 is a bit overwhelming. Perhaps consider summarising each policy in Table 5 and make the full text extractions available as an appendix for interested readers.
I don't find the reason given for a lack of outreach to sponsors particularly compelling. The author's state they did not verify with sponsors (despite this being a feature of the work from Goldacre et al. they cite as a template for this research) because "data collection took place at a time when the publishing activities...was rapidly changing." That seems like a post hoc justification rather than an actual reason why it didn't actually occur. It also isn't supported by their own data as the median date of effect of policies reported was September 2014, 3 years prior to data collection and the higher IQR range is April 2016, over a year before the searches. Similarly, this article is cross-sectional, so I don't see that reason as relevant anyway. You could simply have discarded policies from before a cutoff date, etc. Can the authors please revisit their discussion of this limitation? Table 8 may fit better in an appendix.

If applicable, is the statistical analysis and its interpretation appropriate? Yes
Are all the source data underlying the results available to ensure full reproducibility? Yes

Are the conclusions drawn adequately supported by the results? Yes
Competing Interests: No competing interests were disclosed.
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
Author Response 01 Apr 2021 Larissa Shamseer, University of Ottawa, Ottawa, Canada Thank you for reviewing this work. Please see responses to your comments below.
We have made some changes in the introduction and discussion sections to make the wording more concise.

○
We have clarified that scholarly publishing is intended to facilitate the discoverability and uptake of funded research (into practice) and that predatory journals limit this by having unreliable discovery and questionable trustworthiness. We did not set out to investigate funders' open access (OA) policies or shifts towards OA since discoverability and other journal features compromised by predatory journals can be achieved independent of openness. We note that several sources (ROARMAP, SHERPA JULIET) list details of funders' open access policies. This research is instead concerned with hallmarks of journal validity which have been (mis)taken as signals of research quality.

○
Regarding other dissemination routes, we have clarified that this research focuses on journal publications rather than other publications routes (end of introduction section).

○
As indicated, Dr. Holmes is the CEO of a health research funding organization (not included in our sample). Her expertise of the funding landscape and whether a listing of health research funders existed or whether the chosen list was the best source of this information, was an invaluable methodological contribution to our research. We prefer not to remove this acknowledgment.
○ Extraction criteria were devised de novo based on expertise within the author team. There is no reference standard for funder policies about publishing and as such we did not use any framework for extraction. We have added a clarification of this point in the methods section rather than in the discussion section.

○
We have added text to the introduction (2nd para) outlining the waste of research funds & APCs due to predatory journals. While we did not extract specific data on funders' APC provisions, where funders provided information about journal APC support, this can be found in Table 5.

○
We have updated the suggested information in the flow diagram and left the related descriptive text in results for additional reference.

○
We did not apply an analytical framework to the extracted text and doing so would be a post-hoc analysis. The table/text is provided for reference/transparency of extracted data summarized in Tables 2-4. We do not feel that a summary is necessary.