An international, cross-sectional survey of preprint attitudes among biomedical researchers

Approach

Prior to commencing this study, a protocol was added to the Open Science Framework (OSF) prior to participant recruitment.¹¹ The MEDLINE database was used to identify participants using publicly available information. Once identified, the participants were invited to complete a survey online. Survey questions were purpose-built by a core team of authors. They focused on understanding biomedical researchers’ attitudes and opinions concerning preprinting. This study was approved by the Ottawa Health Sciences Research Ethics Board (OHSN-REB Number: 20220584-01H) on November 17, 2022.

Open science statement

The protocol, study materials, and raw and analyzed data have been made available via OSF at the time of publication submission.¹¹ A preprint of our manuscript has been posted on MedRxiv.¹²

Study design

We conducted a large-scale, anonymous, online, cross-sectional survey of authors who had published in biomedical journals contained within the NLM catalog.

Sampling framework

The sampling method followed that as described by Ebrahimzadeh et al.¹³ We downloaded the MEDLINE database that contained approximately 30,000 journals. The “Journals currently indexed in MEDLINE” filter from the NLM Catalog was used to ensure only journals were included in the study. From this list, 400 journals (Extended data: Supplementary File 1) were randomly selected using the RAND() function of Microsoft Excel for Microsoft 365 MSO (Version 2310 Build 16.0.16924.20054) (2021). Authors’ names and emails were extracted using our script from articles published from 2021/07/01 to 2022/08/01. Extended data: Supplementary File 2 outlines the semi-automated process we used, generated by Ebrahimzadeh et al.^13,14

Survey

The complete survey is provided in the Extended data: Supplementary File 3.¹⁴ The first draft of the survey was created by JYN, then pilot tested by SEP, DM, KC, and ACA; feedback from the pilot test was discussed by the team and led to a final version of the survey. The survey was then distributed, and responses were collected both using the University of Ottawa’s approved version of SurveyMonkey. The survey contained 29 questions, and SurveyMonkey estimated that it would take approximately 15 minutes to complete. The survey initially asked participants five general demographic questions including gender, research role, area of research, and country of residence. Participants were then asked nine questions about their experiences with preprints, followed by fifteen questions about their preferences and opinions regarding preprinting. The survey used adaptive formatting, whereby a participant’s prior response would influence the next question they see and the answers that might be provided. Most of the questions were multiple-choice, while a handful were open-ended. Two reminder emails were sent to the participants, each spaced one week apart after the original invitation email was sent (i.e., three emails in total). The survey was closed approximately five weeks after the initial invitation email was sent. Both the initial invitation and reminder emails clearly made mention of this to the survey invitees. Responses were collected from January to February 2023.

Data management and analysis

In accordance with Ebrahimzadeh et al.,¹³ survey data gathered from the participants were exported to Microsoft Excel. Basic descriptive statistics, such as counts and percentages, were generated based on the analysis of the quantitative data. Based on gender, employment sector, current career stage, and research area, information and replies from participants were compared and studied. Thematic content analysis was used by different members of the research team to individually code replies to specific qualitative items. To reach an agreement on the respective codes, which were categorically classified and specified into distinct tables, researchers carried out multiple rounds of discussions. The Checklist for Reporting Results of Internet E-Surveys (CHERRIES) was used to report this survey.¹⁵

Demographics

A total of 730 responses were received; the raw response rate is 3.04%, the response rate excluding the bounced emails is 3.20%, and the completion rate is 95.48%. Incomplete responses were defined as those with no questions answered on the second page of the survey onwards. Moreover, it should be noted that this response rate is underestimated because we cannot determine how many of the 22 808 authors who were emailed identified as biomedical researchers. It was also impossible to know a number of external factors that would impact the response rate, such as authors who moved institutions, were on vacation, were ill, and passed away, and thus, were unable to respond. As all questions were optional to answer, we therefore provide the total number of respondents for each question. The percentages provided were calculated using each question’s total number of responses. Over three-fifths of respondents (n=455, 62.41%) were identified as senior researchers, which we defined as researchers who started their careers after formal education over ten years ago. Of the 729 responses, participants were identified as follows: academia (n=580, 79.56%), research staff with no formal academic or industry position (n=58, 7.96%), government scientists (n=28, 3.84%), third sector (n=8, 1.10%), and pharmaceutical industry (n=5, 0.69%); none of the participants were part of the scholarly communication industry. We received 50 responses (6.86%) in which participants chose the “other” option, which mainly consisted of participants from the biotechnology industry, retired personnel, clinical researchers, clinicians, medical practitioners, and students. Detailed demographics and other aggregate participant data are presented in Table 1. The responses from participants were also stratified based on gender, employment sector, current career stage, and research area.

Experience

Familiarity with preprints

We asked how familiar participants were with the concept of a preprint based on the Framework for Open and Reproducible Research Training (FORRT) definition, “A publicly available version of any type of scientific manuscript/research output preceding formal publication”. Of the 694 responses, participants ranked how familiar they were with the concept of preprints: very familiar (n=167, 24.06%), familiar (n=269, 38.76%), and somewhat familiar (n=92, 13.26%). Only 61 (8.79%) respondents had never heard of this term.

Publishing experience

We then asked about the participants’ personal experiences with posting on preprint servers. Of the 694 responses, approximately one-third (n=237, 34.15%) had authored more than 31 publications in the past five years. Most participants had not posted a manuscript on a preprint server, with 540 (78.15%) participants of 691 responses not posting their most recent work on a preprint server, and 414 (59.74%) respondents of 693 responses having never posted a manuscript on a preprint server. However, when asked if participants would create a preprint in the future, out of 695 responses, 303 (43.60%) participants said they would not, 296 (42.59%) participants said they were unsure if they would, and 96 (13.81%) participants said they would create a preprint in the future.

Of the 335 respondents who had posted preprints before, the most common preprint server used was bioRxiv, with 148 (44.18%) respondents having published previously on this server. In addition, we asked at what point in the publication process were preprints posted, and of the 294 respondents who had previously posted preprints, 110 (37.41%) participants posted their preprint prior to submitting them to a journal, while 133 (45.24%) participants posted a preprint simultaneously while submitting to a journal.

Viewing/downloading preprints

Interestingly, although most participants did not publish their own works on preprint servers, more than two-thirds of the 695 respondents (n=481, 69.21%) had previously viewed or downloaded preprints, although only approximately a quarter (n=176, 25.32%) had cited a preprint. The most common preprint servers used to view/download preprints among 682 responses were bioRxiv (n=243, 35.63%), MedRxiv (n=148, 21.70%), and ResearchGate (n=140, 20.53%). It is important to note that unlike bioRxiv and MedRxiv that are exclusively preprint servers, ResearchGate is not perceived by all to be a preprint server, as it serves other functions, such as asking and answering questions, connecting/networking with other scientists, and finding collaborators. Therefore, as we did not ask how participants interact with ResearchGate, it is possible that participants selected it for reasons other than viewing preprints.

Peer reviewing preprints

We asked whether peer review should become a part of the preprinting process, and 684 responded, with 231 (33.77%) respondents believing that peer review should be part of the preprinting process, while 240 (35.09%) respondents said it should not. In contrast, 213 respondents (31.14%) were unsure (Table 2). We also asked if our participants had experience with peer reviewing a preprint, and from the 688 responses, the vast majority (n=638, 92.73%) stated they had not. Of those who had peer-reviewed a preprint, we wanted to know if the FAST principles were used during the process, and we received 205 responses. Of the respondents, 148 (72.20%) stated that they were not familiar with the FAST principles, 44 (21.46%) said they did not use the FAST principles, and 13 (6.34%) said they did.

We also asked if participants believed that patients or members of the public should be able to peer-review preprints. Of the 445 who expressed an opinion, 320 (71.91%) believed that patients and members of the public should not be able to peer review preprints, while 125 (28.09%) believed that they should have that opportunity.

Factors and attitudes toward preprinting

We then aimed to determine the factors that impact an author’s decision to post a preprint. We asked participants to assess which had the most significant impact on posting a preprint: employer/research institution, funding agency, lead author, co-author consensus, and journal policy. From the 651 respondents, the lead author had the most significant impact on the decision to post a preprint, followed by journal policy, co-author consensus, and funding agency, with the least impactful being the employer/research institution (Figure 1).

Figure 1. Ranking of factors that influence respondents’ decision to post a preprint.

We also asked whether employers required or prohibited the posting of a preprint prior to submission to a journal. Most employers did not, with 649 (94.06%) respondents of 690 responses stating that their employer does not require them to post a preprint, and 657 (95.08%) respondents of 691 responses stating that employers do not prohibit them from doing so. In addition, we wanted to see if a funder may impact this decision, and of the 692 responses, 639 (92.34%) participants stated that funders do not require them to post a preprint.

We also wanted to determine how familiar participants were with the preprint policies of journals that they had published with previously. First, we asked whether participants were familiar with either Sherpa Romeo or Transpose Publishing, which are resources that present the preprint policies of most peer-reviewed journals. Of the 686 responses, 623 (90.95%) were not familiar with Sherpa Romeo, and 656 (95.91%) were not familiar with Transpose Publishing. In addition, we asked what the preprint policy was for the journals of respondents’ most recent first author/co-author publications (Figure 2). Of the 684 responses, 282 (41.23%) stated that they were not familiar with the preprint policies, while 183 (26.75%) stated that preprinting was permitted. Only 34 (4.97%) respondents and 61 (8.92%) respondents stated that preprinting was prohibited or that they had no preprinting policies.

Figure 2. Preprint policies of journals respondents had most recently submitted a first author/corresponding author publication.

Respondents were asked to what extent they agreed with certain incentives or the consequences of preprinting (Figure 3). Of the 666 who answered, over half had positive attitudes toward the following incentives and consequences: 1) preprints support open science practices (n=456, 68.99%); 2) preprints should not be cited because they may contain results that lack credibility because they had not yet undergone peer review (n=433, 65.12%); 3) preprints can be used as a weapon for the dissemination of misinformation (n=417, 62.61%); 4) preprinting behaviors will increase in the biomedical field in the future (n=415, 62.32%); and 5) preprinting allows for more efficient scientific dissemination (n=400, 60.06%). In terms of whether respondents agreed or disagreed with the statement “Preprinting may encourage other researchers to steal project ideas (i.e., scooping), less than half (n=304, 45.66%) respondents agreed, 143 (21.47%) respondents remained neutral, and 219 (32.89%) respondents disagreed. On the other hand, 284 respondents (42.71%) did not agree that preprinting would ultimately lead to higher-quality research being published, while 224 (33.68%) and 157 (23.60%) respondents chose neutral and agreement options, respectively.

Figure 3. Respondents’ opinions on incentives and consequences of preprinting.

We then asked the extent to which respondents agreed with the following motivations for publishing a preprint (Figure 4). Of the 666 respondents who answered, over half had positive attitudes toward the following incentives and consequences: 1) preprinting increases authors’ visibility and allows for networking (n=489, 73.43%); 2) preprinting allows for early/more feedback on their manuscript submission (n=470, 70.58%); 3) preprinting allows authors to share studies that have been rigorously conducted but present negative/perceived low-impact results (i.e., lower likelihood of publication in a peer-reviewed journal) (n=437, 65.70%); 4) preprinting makes my research visible to all if it is published in a paywall journal (n=415, 62.89%); and 5) preprinting increases the likelihood of authors receiving more views and citations on their published article (n=365, 55.14%). In addition, slightly less than half of respondents (n=323, 48.57%) agreed that preprinting allows authors to prevent duplication of efforts, while 193 (29.02%) disagreed, and 149 (22.42%) remained neutral. In terms of if respondents agreed or disagreed with the statement “preprints aid researchers’ careers with respect to hiring, promotion, and tenure”, respondents’ opinions were more evenly split, with 205 (30.91%) respondents agreeing, 218 (32.88%) respondents remaining neutral, and 240 (36.19%) respondents disagreeing.

Figure 4. Respondents’ opinions on motivations to publish a preprint.

In addition, we asked participants for their opinions on making preprinting mandatory during the research process. Of the 665 respondents who expressed opinions, over half (n=363, 54.58%) were leaning toward disagreeing with the statement. Only approximately a quarter (n=163, 24.52%) of the respondents had positive opinions regarding this statement.

We then asked an open-ended question to allow respondents to freely express other reasons that motivated or dissuaded them from preprinting (Table 3). The 197 responses received could be primarily split into positive and negative factors. The most prevalent positive factor that motivated preprinting “preprints allow rapid dissemination”. This was followed by “preprints show research efforts,” in which preprinting allowed authors to show their research to other parties, such as other researchers, funding agencies, and universities, without the use of a publication in a journal. Lastly, the final positive factor commonly mentioned was “preprints are open-access items”.

On the other hand, the open-ended questions showed that many negative factors discouraged respondents from preprinting. The most frequent, with approximately a third of responses, was “peer review is bypassed,” as many responses stated that peer review must take place to determine the credibility and quality of research before publishing. In addition, many respondents believed that “preprinting is not needed,” since preprinting does not provide any benefits to the individual’s career and there were no issues with the current publishing system. Another negative factor was “preprints might release misinformation,” followed by “preprints are poor-quality research.” Next, many responses stated that the “journal policy prevented preprinting”, since journal policy rejected or was unclear about preprinting. “Scooping” was also mentioned frequently, along with “preprinting requires additional resources,” where respondents felt that preprinting needed extra time, effort, and funding that is deemed unnecessary for the overall publication of their work. Additionally, many respondents were “unfamiliar with preprints” which discouraged them from preprinting. Lastly, “multiple versions of the same paper” was another negative factor, where having a preprint and a journal published version of the same paper results in the release of unedited, possibly incorrect, work and causes confusion when referencing.

Opinions on the benefits and consequences of preprints

In terms of the benefits and consequences of preprints, the three statements that respondents most agreed with were: 1) preprints support open science practices, 2) preprints should not be cited because they may contain results that lack credibility because they have not yet undergone peer review, and 3) preprints can be used as a weapon for the dissemination of misinformation. These results align with the preliminary work done by Funk et al., who found that the most concerning issues with preprints from the perspective of respondents, who were mainly researchers in academia, were premature media coverage and public sharing of information prior to peer review, while respondents believed that the greatest benefit of preprints is making their work open access.

On the other hand, the top three most impactful factors among respondents with respect to deciding to publish a preprint included 1) increasing author visibility and networking opportunities, 2) allowing for early and more feedback on manuscripts, and 3) publishing of negative or perceived low-impact results. The findings by Funk et al. somewhat align with our results, as they found that only a little over half of the respondents found early feedback as “very beneficial” and were only seen as less impactful in comparison to open access and speed of dissemination. However, publishing negative or perceived low-impact results was the second least impactful factor, only being seen as “very beneficial” by approximately one-third of the respondents. This variation may be due to greater awareness of the bias of journals toward positive results, and why this is an issue for the scientific community. Nowadays, authors are pressured to only publish positive “groundbreaking” results, and journals often reject papers with negative or non-significant results to increase readers, citations, and submissions.^16–19 However, negative and non-significant results are vital to the scientific process as they allow for collective self-correcting progress and ensure that resources are not wasted on the replication of failed research.^17,18 A survey by Echevarría et al.¹⁹ showed that the majority of authors believe this information should be shared, although only a handful have tried to publish negative results. Preprints may be an appropriate alternative for publishing ‘negative’ data by allowing researchers to quickly disseminate their findings without delays of a lengthy peer review process. Also, unlike traditional journals, preprints eliminate the risk of manuscript rejection and avoid concerns related to journal metrics. Future work should focus on gaining a better understanding of researchers’ attitudes toward publishing negative results and preprint practices to ensure that negative results are appropriately disseminated within the scientific community.

Scooping is often a negative factor that impacts authors’ decision to share their work with open access, at conferences, and on preprint servers.²⁰ Scooping is when someone unethically claims a research idea or results as their own via earlier publishing and often occurs due to public sharing of research prior to publishing in a journal.^20,21 As a result, scooping can prevent researchers from publishing their work in journals and turns their research into a waste of effort.²⁰ In our survey, slightly less than half of our respondents believed that preprints encouraged scooping, and this was also mentioned as a concern against preprinting in the open-ended questions. Thus, our findings agree that perceived scooping prevents researchers from publishing in preprint servers. However, there is no evidence of scooping via preprints occurring or that preprints encourage scooping, as observed by Tennant et al.²¹ Rather, preprint servers actually protect against scooping, as preprint servers provide time-stamps and establishes priority of discovery, which does not occur when presenting research in public.²¹ Efforts should therefore be made to show researchers that scooping and its consequences are unlikely to occur, and instead, researchers should focus on the benefits of preprinting.

One hesitation toward the use of preprints revolves around the fact that preprints have not undergone peer review; therefore, many researchers believe the manuscript content has not been validated before being released to the public. This is seen in our findings, whereby almost two-thirds of our respondents believe that preprints should not be cited because the results may lack credibility since there was no peer review. Moreover, over two-thirds of our respondents had viewed or downloaded a preprint, but only a quarter had ever cited it, implying distrust in preprint content. Additionally, peer review being bypassed was the most frequent factor that negatively impacted respondents’ motivations to preprint, as seen in the open-ended question, alongside the fear that preprints will disseminate misinformation and poor-quality research. Thus, we consistently observed that peer review is regarded as highly important and necessary for publishing scholarly work by respondents. Respondents’ hesitation toward preprints due to lack of peer review and fear of misinformation indicated a lack of knowledge about the evidence regarding the quality of the published literature and the effectiveness of peer review. Research conducted by Zeraatkar et al.²² has shown that there was no evidence that preprints provided results that were inconsistent with peer-reviewed publications, specifically seen with COVID-19 treatments. Although Bai et al.²³ found differences in results between preprints and publications, the main conclusions remained consistent.

MEDLINE is also currently indexing preprints as part of the NIH Preprint Pilot project, supporting research that has been posted to eligible preprint servers for easy discoverability and preservation. In addition, it is important to note that peer review is not always effective because of inconsistency and bias, as many peer reviewers have not received formal training on how to effectively peer review, and many training opportunities and courses are not openly accessible online, inhibiting researchers from gaining this education.^22,24–27 Thus, many believe that the peer review process ensures quality research is published may negatively impact researchers’ opinions toward preprints.

Interestingly, despite many respondents distrusting preprints due to a lack of peer review, many were divided on whether peer review should be incorporated into the preprint process. This may be because preprints allow for the rapid dissemination of research as there is a delay from journal submission to publication when undergoing peer review.^1,5 If peer reviews were to be incorporated into preprinting in the same way as journals, then the work of peer reviewers would increase, and the delay would become much longer. It has thus been suggested by Soderberg et al.,⁵ that preprint servers should be modified to improve the credibility of preprints, as the current preprint feedback system allows for public feedback and discussion, which may aid in credibility or results. However, authors often fear unfair criticism from competitors, harm to their reputation, or softened criticism due to the public nature of feedback.⁸ The FAST principle may be able to alleviate these issues.⁷ Based on our findings, the majority of respondents had not peer-reviewed a preprint before, and of those that had, most were not familiar with the FAST principles. Therefore, efforts should be made to educate researchers on these principles to promote feedback on preprints. Future work may aim to better understand researchers’ opinions on peer reviewing preprints and potentially provide other novel solutions such that the benefits of peer review can be implemented in the preprinting process while simultaneously allaying the current fears of preprint feedback.

External factors that play a role on opinions and attitudes toward preprints

Journal policy is the second most impactful factor in the decision of authors to post a preprint and was frequently mentioned in the open-ended question. This may be a result of unfamiliarity towards a journal’s preprint policy, as seen in the survey. Nowadays, most, though not all, life science journals accept preprinted manuscripts for submission, and some journals either do not have a preprint policy or have contradicting statements.³ Biomedical journals should create clear and concise preprint policies that allow researchers to understand whether they can publish preprints before or during submission to a journal. In addition, our data showed that most researchers were unfamiliar with resources about preprint policies, such as Sherpa Romeo and Transpose Publishing, which gather journal policies on open access and preprinting. In the future, it would be best to resolve the issue of lack of awareness towards journal’s preprint policies by educating researchers on these types of resources to address the concerns that preprints negatively impact the chances of publishing in a peer-reviewed journal.

The respondents did not have strong opinions on whether preprints aid researchers’ careers with respect to hiring, promotion, and tenure. In addition, our findings showed that universities and research institutes had the least impact on respondents’ decision to publish a preprint, and the majority of our respondents’ employers neither prohibited nor permitted the posting of preprints. This may be because few universities policies consider preprints when making decisions regarding hiring, promotion, and tenure. It has been suggested that hiring, promotion, and tenure policies should be modified, as current methods rely heavily on quantitative metrics, which have been recognized as a flawed system.^28,29 Preprints may prevent the hiring committee’s bias toward or against a journal due to its impact factor or reputation, as suggested by Polka. The inclusion of preprints in hiring, promotion, and tenure policies may improve biomedical researchers’ attitudes toward preprinting.

Strengths and limitations

By implementing a cross-sectional survey for this project, we took a snapshot of the population of interest without following them over time. This study was able to generalize among biomedical researchers by randomly surveying a large sample of them with varying opinions regarding preprint postings. A limitation of our study design was that it was written in English, so researchers without working knowledge of English could not take the survey. In addition, as the present study was based on an English-speaking international sample, it does not consider the impact of national policies on attitudes and opinions toward preprinting. Another limitation of our sampling strategy is that the list of email addresses used potentially included inactive or invalid addresses, which may have been a result of changing professions, retiring, or passing away; furthermore, we did not consider autoreplies. Therefore, the response rate is underestimated. Our questionnaire was built on self-declared attitudes and preprint practices, which may not accurately capture independently confirmed practices. Inherent to the cross-sectional survey design, the final limitation also includes a low response rate, recall bias (in which participants do not correctly remember past events), and non-response bias (in which participants did not want to or could not complete a survey question). In addition, responses mostly reflected the opinions of senior academics, and there were less responses from early career researchers. Most responses were also from clinical and pre-clinical researchers due to our sampling of the MEDLINE database. Therefore, it is possible that researchers in fields not typically indexed in MEDLINE may have different views and experiences.