Grant writing and grant peer review as questionable research practices

1. Introduction

In industrialized societies, a large fraction of the governmental budgets for research is allocated through competitive peer review of project proposals. This popular mode of funding allocation has been criticized for not delivering the scientific goods it was intended to deliver. Evidence has shown that grant peer review is costly, that the ranking it produces lacks validity, and that it does not promote novel views (Guthrie, Ghiga, and Wooding 2018; Herbert et al. 2013; Link, Swann, and Bozeman 2008 and other references below). But whereas the epistemic defects of peer-review project funding (PRPF) have been extensively studied, its ethical shortcomings have received little scholarly attention. This is surprising, as there has been a strong increase in attention for the prevalence of misconduct and questionable research practices in other parts of science over the past decade (Xie et al. 2021). It is these ethical shortcomings of PRPF that the current paper is concerned with. More specifically, we will argue that PRPF systems prompt behaviour that violates moral values and norms that, according to prominent scientific codes of conduct, scientists, funding agencies and other stakeholders of PRPF are expected to conform to.

As we will see, PRPF systems exert a whole range of different pressures on applicants, reviewers and funders to behave unethically. Importantly, these pressures vary in strength. Sometimes, the pressures are best thought of as (mere) incentives. Those incentives make it more likely that individual researchers will act unethically, primarily because such morally questionable behaviour will increase their chances of success in grant acquisition. Yet, such incentives do not really require researchers to engage in unethical practices. On the other hand, there are also what we will call ‘forces’. Because of these forces, researchers who want to apply for research funding or who agree to serve as a grant-decision maker are required to behave in a way that most researchers deem ethically questionable.

The conclusion of this paper is that the academic community should reconsider its widespread use of PRPF systems, not just because of these systems’ (alleged) inefficiency and epistemic shortcomings, but also because they almost inevitably promote ethically questionable behaviour.

In the next section, we briefly discuss the background, prominence and epistemic shortcomings of PRPF. The third section analyses prominent scientific codes of conduct (CoC) to identify the ethical values that are supposed to guide the behaviour of researchers. The fourth section then argues that PRPF systems come with a whole range of incentives and forces that prompt violations of these values. Finally, the fifth section discusses what is needed to make the allocation of research funding more ethical.

2. PRPF: background, prominence and epistemic shortcomings

Since the 1970s, governments have distributed an increasing fraction of their resources for research on the basis of competition, following a model that the American National Science Foundation (NSF) put in place in the 1950s (England 1982). These governments urged funding agencies to organize competitions among researchers, such that project proposals are assessed by academic peers, in a way analogous to the peer review of scientific articles.

At present, science funding agencies in industrialized societies allocate much of their funding through PRPF.¹ In the US, for example, the National Institutes of Health (NIH) annually invests over $30 billion in basic and applied biomedical research. All of that money is allocated through PRPF. The same holds true for the budget that the NSF spends on basic research (about $8.8 billion per year).² The European Union, finally, allocated about €60 billion through PRPF between 2014 and 2020 with its Horizon 2020 programme (Schiermeier 2020). Typically, these PRPF funding schemes have very low success rates, making them highly competitive.³

The current widespread reliance on highly competitive PRPF has a series of consequences for researchers. Trivially, peer review of grant proposals influences what kind of research will be done, and who gets to do it. Successful applications also come with prestige, both within and outside the research community (Coate and Howson 2016). Relatedly, research careers can be made or broken by grant applications. Many research institutions now make tenure, promotions and salary raises dependent on a researcher’s success in prestigious calls (Dunn, Iglewicz, and Zisook 2020; Joiner and Wormsley 2005) and success in grant funding competitions is one of the most common criteria that institutions use for review, tenure and promotion (Rice et al. 2020). For example, several European universities give a substantial salary supplement to successful ERC-applicants for the duration of their project. In Germany, there is a bonus system for professors that regularly includes targets for grant acquisitions (Erina 2015).

Given its crucial role in today’s science, one would expect that PRPF reliably and efficiently selects the best scientific projects. Yet, the literature on research funding has raised serious doubts about this. Several studies indicate that peer review is not a very reliable or valid method for evaluating research proposals. Regarding reliability, Kaplan et al. (2008), for instance, argue that for the mandated level of precision in reviewers’ scores, the NIH needs as many as 40,000 reviewers per project instead of the 4 reviewers it now aims for. Moreover, biases such as cronyism are pervasive in grant funding (van den Besselaar 2012; Guthrie, Ghiga, and Wooding 2018).

Since low reliability is typically associated with low validity, it should not come as a surprise that reviewers often fail to predict the scientific success of the proposals they evaluate. For instance, there is no or only a weak correlation between the review score of a project and its eventual bibliometric impact (Doyle et al. 2015; Fang, Bowen, and Casadevall 2016; van den Besselaar and Sandström 2015). Plenty of anecdotal evidence suggests that review panels do a poor job at predicting success: Nobel Prize laureates have repeatedly complained about the difficulties they have experienced in getting their award-winning work past grant review committees (Bendiscioli 2019; Johnson et al. 2010; Kim 2006; Marshall 2005; Taubes 1986).

Other epistemic problems relate to the types of knowledge that are generated by PRPF funded projects. Although many funding agencies emphasize that their aim is to fund innovative research, PRPF systems might instead be conservative, and favour well-established views rather than radically new ideas (see Nicholson and Ioannidis 2012; and reviews by Guthrie, Ghiga, and Wooding 2018; Guthrie et al. 2019).

These epistemic problems might be justifiable if the costs and impacts of the system were relatively low. Yet, the costs of PRPF turn out to be very high. Link et al. (2008) show that researchers at R1-universities in the United States spend on average more than four hours a week writing project applications. Similar studies were conducted for the National Health and Medical Research Council, a major funding organization for biomedical sciences in Australia. The results of these studies indicate that the time investment in 2009 was as much as 180 years of research time to fund 620 projects, and by 2013, the costs had gone up to more than 500 years of research time – equivalent to € 41 million in salary – to fund approximately 700 projects with a total value of € 226 million (Herbert et al. 2013).

3. Ethical values and norms associated with research integrity

Individual ethical behaviour co-varies with situational factors and with personality traits (Bruton et al. 2020). Here, we will be concerned with the role of situational factors, and more specifically with how institutional and structural aspects of research environments prompt unethical behaviour. Our focus on systemic incentives fits today’s scholarship on research integrity. For example, in a recent consensus study report of the National Academies of Sciences-Engineering-Medicine (NAS 2017, 208), it is noted that “patterns of funding and organization that have emerged over the past few decades in the United States have created environments increasingly characterized by elements [ …] that are associated with cheating, such as very high stakes, a very low expectation of success, and peer cultures that accept corner cutting” (NAS 2017, 98). We concur, and analyse the characteristics of PRPF that lead to unethical behaviour below. Before doing this, we first identify the core principles of research integrity.

Scientists are bound by a number of science-specific norms and values. These norms and values are frequently made explicit in scientific CoCs. Such CoCs are almost invariably based on, or derived from, the five norms developed by Robert Merton (Radder 2019).⁴ CoCs are a good starting point for our analysis because they are the result of extended debates among a wide variety of stakeholders. They function as consensus statements, reflecting an overall agreement among these stakeholders about the norms that should guide research, and present both minimal conditions for ethical research practices and aspirations. In addition, institutions and funding agencies often adopt these CoCs as a framework for their own policies and regulations on research integrity.⁵ Hence, if we can show that PRPF threatens the core values of the CoCs, we have a strong case for urging these academic institutions and funders to revise their policies on PRPF.

We analyzed the following five CoCs: the Singapore Statement on Research Integrity (Resnik and Shamoo 2011) that was written at the second World Conference on Research Integrity, the European Code of Conduct for Research Integrity (2017), developed by the European Science Foundation and All European Academies, Doing Global Science: a guide to responsible conduct in the global research enterprise (IAP 2016) written by a committee of leading scholars on research ethics, Fostering Integrity in Research (NAS 2017), a consensus document published by the National Academy of Sciences, and Ethical Guidelines for Peer Reviewers (COPE 2013), drafted by the Committee on Publication Ethics. These CoCs were selected because of (1) their geographical focus (EU, USA, world), (2) their generality (not about one discipline or aspect, but about science in general), and (3) their authoritative status within the scientific community. We also analyzed Ethical Guidelines for Peer Reviewers (COPE 2013), because peer review is central to PRPF.

The selected CoCs differ somewhat in their definitions of integrity and misconduct. They also vary in their approach. Some CoCs are more value-based, and state which values (e.g. honesty) should guide research. Other CoCs are more norm-based, and primarily indicate which behaviours need to be sanctioned or stimulated (Godecharle, Nemery, and Dierickx 2014). The CoCs also vary with regard to the values and norms they include. This variation is largely due to the inevitable vagueness associated with the formulation of values and norms, and due to the different objectives of the CoCs. Still, the differences do not reflect deep disagreements on what constitutes misconduct or about the core values.

In order to facilitate our assessment, we have extracted from the CoCs one list of values and norms. This list ignores the subtle differences just mentioned, and captures what the CoCs have in common. Table 1 summarizes how our list maps onto the values prioritized by the various CoCs.

Our list comprises the following values and norms:

We acknowledge that there is some overlap between some of these values (as there is in the CoCs) and that alternative categorizations are possible. For example, the problem of double dipping (i.e. researchers using the same project in different funding schemes) is as much a violation of norms like solidarity or generosity as it is a violation of honesty. Thus, an alternative classification that includes solidarity and generosity would be viable as well. Still, our categorization fits our purposes, as it captures the main ways in which PRPF invites ethically questionable practices, and helps us (in Section 4) in structuring them.

Importantly, on a final note, in the remainder we will construe the five values of research integrity primarily as ethical values. We acknowledge, though, that each of them is directly or indirectly related to epistemic values. Consequently, many of the problems we discuss are epistemic as well as moral problems. It is no coincidence that this is so: research integrity is concerned with scientific research, and society values such research primarily because of the epistemic goods it delivers. Given this entanglement of ethical and epistemic considerations, our focus on the ethical aspects of PRPFs is inevitably somewhat artificial. However, such a focus is useful here, as it more clearly brings out an important point that has not received due attention: the ethical dimension of the problems that PRPF systems give rise to. In any case, to the extent one regards these values as merely epistemic, our paper complements the epistemic worries about PRPF raised and reviewed in Section 2.

4. PRPF prompts violations of research integrity

This section argues that PRPF forces or incentivizes researchers to violate each of the five aforementioned values. Many ethically questionable practices can be categorized as a violation of more than one value. When that is the case, we just place and discuss it under one value. Our focus will be on ethical problems that arise for individual researchers who apply for, evaluate and receive research grants. System-level moral issues, such as the inefficiency of PRPF systems, which are primarily associated with policy-makers and funding organizations, fall outside our scope.

Before we turn to these practices, it is good to note that most of them have not, or only rarely, been studied. We refer to empirical work whenever it exists, but this is unfortunately not the case for all practices. Even though this means that there is not always published evidence that backs up our claims that these practices are prevalent, we believe that most people familiar with academia and its funding processes will recognise the practices we discuss and know how common they are. We too have at some point engaged in some of these practices, and we expect that most readers of this paper also either engage in them or know of colleagues who do.

5. Discussion and conclusions

The violations we have listed are not meant to be exhaustive, and they are unlikely to capture all the senses in which PRPF systems prompt ethically questionable research practices. In addition, some may find that most of the questionable research practices we discussed are minor issues, and it is true that none of the problems presents a clear knock-down argument against PRPF. Still, we have seen that PRPF systems force or incentivize researchers to violate, in one way or the other, each of the five norms and values commonly associated with research integrity and included by all major CoCs. And while many of these may seem like only minor violations of the COCs, some, such as double-dipping, self-plagiarism and ghost authorship, are unquestionably problematic and serious. Listing these minor and major violations side by side shows that ignoring these problems comes at a substantial ethical cost, as the issues are numerous, pervasive and unlikely to disappear on their own. In this concluding section, we briefly consider three options for reform.

The first option is to mitigate the perverse incentives associated with PRPF by eliminating or modifying those features of PRPF that prompt questionable behaviour. For instance, funding agencies could remove the demand to formulate strict timelines that indicate expected successes and measurable targets (e.g., number of papers, targeted journals, milestones). Alternatively, they could formally ask reviewers to estimate their confidence in their own reviews, and applicants to disclose any other funding schemes they have applied to with the same project. While making such changes would undoubtedly solve some of the issues we have discussed, this option will presumably remain sub-optimal. To start, many of the features of PRPF were introduced for good reasons. For example, measurable targets make it easier for reviewers to evaluate the output of the project. Second, the likely impact of some of the changes would be limited. If a funding body no longer required applicants to formulate milestones, applicants would arguably continue mentioning them, because they intuit that milestones strengthen their application. Similarly, reviewers’ estimates of uncertainty would also be subject to uncertainty, thus propagating the same problem on a different level. Third, some of the incentives we have discussed seem to be intrinsic to PRPF systems and cannot be substantively modified without abandoning PRPF altogether. For instance, as long as grant decision-makers are human, the norm of impartiality will be hard to conform to.

A second option is to draft regulations for the specific context of PRPF— a ‘CoC for grant writing and reviewing’—and to implement mechanisms for the enforcement of those regulations. Some funding agencies have already set steps in this direction. The Flemish and Dutch research councils, for example, require applicants in many of its funding schemes to indicate whether they have submitted or plan to submit their proposal with other funding agencies. It remains to be seen whether such measures will effectively reduce the prevalence of the practices that they target, such as double dipping. But, in any case, it is doubtful that regulatory work will be enough to address all the worries that we have raised. Indeed, as we have seen, PRPF systems are still subject to cronyism, in spite of codes of conduct that explicitly disapprove of cronyism (van den Besselaar 2012).

A final, more radical option is to put into effect alternative allocation systems. Various such systems have been proposed, primarily with the aim of addressing the epistemic shortcomings of PRPF (Guthrie 2019). These alternatives include peer-to-peer distribution (Bollen et al. 2017), allocation on the basis of past performance (Bolli 2014; Roy 1985), a (modified) lottery among short project proposals (Fang and Casadevall 2016), bicameral grant review (Forsdyke 1991), using AI for peer review (Checco et al. 2021), and baseline funding (Vaesen and Katzav 2017). Of these alternatives, bicameral grant review, allocation on the basis of past performance and peer-to-peer distribution are most similar with PRPF and, in that they still involve (serious) competition among applicants; accordingly, they most likely share PRPF’s shortcomings. Relying on AI for peer review to increase fairness and decrease costs is a promising alternative, but does not seem possible yet. More research on this topic is needed, in particular to ensure that avoiding human biases does not come at the cost of introducing algorithmic biases (Checco et al. 2021). The two other alternatives, i.e. lottery-based systems and baseline funding, seem more promising with respect to research integrity. While they might suffer from different (unforeseen) moral problems, they seem less sensitive to many of the issues discussed in this paper. This is because they differ from PRPF in three crucial respects.

First, baseline funding and lottery-based systems substantively minimize reliance on judgements that we have seen to be problematic. These judgements include unjustifiable predictions in applications and review reports (violation of accountability), omitting crucial methodological details in project proposals (violation of honesty), and biased grant evaluations (violation of impartiality).

Second, baseline and lottery-based systems are relatively difficult to game. This is because they disregard many of the allegedly salient, but easily manipulated, differences among applicants that PRPF grant decisions are informed by. For instance, baseline and lottery-based systems are largely immune to grantsmanship and do not reward the many questionable practices that applicants might use to pimp their publication track record (e.g., salami-slicing, cutting corners, plagiarism, not publishing negative results).

Finally, the credit and prestige that applicants derive from a baseline or lottery-based grant would, relative to the credit and prestige derived from a PRPF grant, be minor or even nil. Indeed, there is little merit in acquiring a grant based on chance (lottery-based) or when every researcher gets one (baseline funding). An additional benefit of decoupling (alleged) merit and funding is that it would temper the Matthew effect and the overconfidence associated with repeated success in PRPF competitions. A non-merit based system would promote intellectual humility, a value that is both epistemically and ethically desirable (Alfano, Tanesini, and Lynch 2020).

A way to summarize these three differences between PRPF and the two alternatives is that only the former distributes funding on the basis of competition between researchers. In various ways, the competition for funding incentivizes researchers to cut corners and violate generally accepted norms of research integrity. This is interesting, as competition also seems to incentivize researchers to violate CoCs in other parts of science such as the research process and journal publications (Anderson et al. 2007a; Fang and Casadevall 2015; Fanelli 2010; Tijdink, Verbeke, and Smulders 2014). Our paper thus adds to these existing arguments for making science and its funding less competitive. Moreover, given PRPF’s epistemic shortcomings, and the likely epistemic advantages of baseline and lottery-based funding, there are also non-ethical reasons to take these alternative systems seriously.

Data availability

No data are associated with this article.