The jury is out: a new approach to awarding science prizes

Part 1: Nomination

To nominate a potential awardee, nominators fill in clearly structured nomination documents with the help of an interactive online platform. The interface provides all necessary definitions and information and helps the nominator organise the nomination correctly into three individual sections: “achievements by the nominee”; “prominence endowed upon the nominee”; and “impact originating from the nominee’s work”. Achievements are defined in line with DORA as the

Prominence, in contrast, is defined as

The prominence section is included because many nominators and panel members still today explicitly or implicitly rely on, independently google or outright demand such measures, especially when evaluating highly prestigious prizes. Rather than deceiving ourselves about these circumstances, we instead try to impose honesty and transparency in their use by strictly containing prominence information to one dedicated section. The definition of impact is the same as outlined in the assessment framework and guidance on submissions of the Research Excellence Framework 2014 (www.ref.ac.uk/2014), which defines impact as

Nominators are asked to keep their statements in the three sections concise and comprehensible to both an educated audience and non-expert members of the public, avoiding jargon. They are informed transparently about how the evaluation panel will use their statements. All information is provided directly within the online nomination mask such that no additional submission guidelines or further documentation are necessary. The same information texts are provided to the members of the evaluation panel. Separating the nomination text into three sections this way allowing the members of the evaluation panel to easily assess the actual achievements of the individual researcher independently of their prestigious associations, and the impact of their work without it being conflated with the work’s quality. Furthermore, it allows evaluators to easily refer to, evaluate and comment on these different aspects of a nomination independently and to compare them individually across nominations.

Any claims made by nominators within the achievements and impact sections have to be supported by references. No more than ten references, distributed freely across these two sections, can be used. Citations are entered into the text as bracketed numbers (e.g. [3]). The actual reference itself, however, only consists of information about the reference type (e.g. “journal article”, “book”, “radio interview”, “code” etc) and the respective abstract, synopsis or description, deliberately omitting any information about author, title, journal, publisher or publication date etc. Furthermore, upon submission of a nomination, the achievements and impact sections are anonymised such that they contain no names, gender or other identifying information but referred only to “the nominee” or the gender-neutral pronoun “they”. Any mention of, for example, institutions, collaborators or associations are also replaced with neutral references such as “at a Swiss university”, “with an established collaborator” or “as editor of an established journal” etc. Fully anonymised nominations can then be generated by separating the prominence section from the now anonymised achievements and impact sections. These fully anonymised nominations, consisting only of the achievements and impact sections with anonymised citations and without any mention of institutions or journal names etc, are then sent to evaluation panel members for pre-evaluation (Figure 1A).

Figure 1. Pre-evaluation.

(A) During pre-evaluation, each nomination (in this case, for example, “Nomination C”) consists of the fully anonymised achievement (Ach) and impact (Imp) sections only (without the prominence sections). References contain only a description of the reference type (e.g. “Journal Article”) and the respective abstract or synopsis. (B) “Nomination C” is then reviewed by a subset of evaluation panel members (here panellist 3 through 7), who each compare the nomination to a different set of other nominations respectively (in this case with some overlap). Each panellist ranks their set according to their personal overall preference (i.e. “who should win the prize?”, overall ranking (OR)) and additionally distribute gold (G), silver (S) and bronze (B) medals for the achievements and impact section respectively.

Part 2: Pre-evaluation

Nominations are sent out to the individual panel members to review ahead of the panel meeting. The panel is comprised of at least seven and up to 11 members. In addition to international academic experts, it also includes two non-academic representatives of society and is age- and gender-balanced.

The nominations are distributed in a systematic, pseudo-random manner. They are assigned to panellists randomly albeit within the confines that every nomination is reviewed by six members of the panel only and that those six reviewers always include one of the two non-academic panel members and the panel’s topic-expert (e.g. all psychology nominations are read by the psychologist in the panel etc, Figure 1B). Furthermore, we ensure that each panel member’s individual collection of nominations to review is different to all others panel members’ sub-sets such that each nomination is compared to different nominations in every case and every nomination is compared to every other nomination at least once (Figure 1B). Importantly, this network distribution of nominations means that each panel member has to only pre-evaluate a sub-set of nominations (the number of which will depend on the total number of submitted nominations) instead of all of them, thus reducing the burden on the evaluators. In turn, the evaluation panel members have to commit to reading through their respective sub-set of nominations in full, including all the abstracts provided in the reference list, while also formally and explicitly committing to not seeking out additional information beyond what was provided (i.e. no googling of nominees etc).

The panel members are asked to pre-evaluate their respective sub-set of nominations in two steps. In the first step, they only receive the fully anonymised achievements and impact sections of the nominations in their respective sub-set (i.e. without the prominence section, Figure 1A). These anonymised nominations have to be monotonically ranked according to the evaluation criteria and the individual panel member’s personal assessment. Additionally, panel members are asked to assign an equal number of gold, silver and bronze medals across the individual achievements and impact sections respectively. If, for example, a panel member has to pre-evaluate 15 nominations in their sub-set, they would have to separately assign five bronze, five silver and five gold medals for achievements and for impact, respectively. The medals provide an intuitive way of generating tercile rankings of achievements and impact, in line with the overall ranking (i.e. first is better than second etc., Figure 1B). Finally, we ask panel members to guess who the nominated person is, if they believed they have an idea, to keep tabs on the extent to which anonymisation was successful.

All assessments throughout the whole evaluation procedure are always based on rankings instead of ratings. The advantage of using rankings (specifically weaker than/ stronger than) instead of ratings (generally weak/strong) is that they are internally normalised, thus mitigating the risk that a very generous or overly strict panel member might skew the evaluation. For example, by strictly specifying the number of gold, silver and bronze medals that can be allocated by a panel member, the chance of receiving more or fewer medals of a particular kind is independent of the individual panel members to whom the nomination is allocated. Instead, it is only a function of those individual panel members’ evaluation of the respective nomination compared to the other nominations within their sub-set.

To further facilitate comparison across nominations, average and median values are calculated after pre-evaluation for each nomination based on the overall ranking and the medals it receives from the panel members who pre-evaluated it. The average overall ranking of a nomination is calculated as the mean value across its individual sub-set rankings. The average achievement and impact medals are calculated as the mean values across its individually assigned medals [defined as Gold = 1, Silver = 2, Bronze = 3 (Figure 2A), calculations of the median scores are done analogously]. These statistics are simple though calculating the mean across ranking data is, to some extent imperfect. However, presenting data in this very straight forward form allows all panel members to always easily and intuitively understand the statistics and their underlying data, which is crucial if they are to accountably argue based upon them (Figure 2).

Figure 2. Ranking data.

(A) To provide an overview, the pre-evaluation data for the example of “Nomination C” from Figure 1 is listed in the table. Replacing the medals with numbers (gold = 1, sliver = 2, bronze = 3), allows for the calculation of ranks (here only the average ranks are shown), which in turn allows for a positioning of each nomination relative to the others for the overall ranking (OR), the achievement ranking (ACH) and the impact ranking (IMP) respectively (plots). (B) In actual fact, the information provided in these rank plots needs to be substantially richer to be genuinely informative. It consists of the number of panellists who evaluated a nomination (n), the mean and median ranking value (red and blue dots respectively), the raw individual rankings from the panellists (gray dots) as well as the mean and median ranking (right y-axis) for each nomination (shown is the normalised overall ranking across 24 nominations based upon real-world evaluation plots but populated with random data for data-protection reasons, similar plots are also generated for the achievement and impact rankings respectively).

In the second step of pre-evaluation, after returning their rankings, the panel members are provided with a detailed report and the full set of non-anonymised nominations. The analysis report provides three important results: (1) a detailed overview of all individual rankings and their mean and median positions across all nominations (see above); (2) a threshold calculated based upon these results, where only the nominations above this threshold are going to be considered further in the evaluation panel meeting; and (3) additional analyses of potential biases in the data.

The threshold is drawn based upon the following rule:

The panel members are then asked to indicate in writing whether they agree with the analysis and thresholding of nominees as described in the analysis report; or feel that one of the excluded nominees should still be included in the further evaluation. In the latter case, the panel member has to then also state in writing why the respective nominee should be singled out despite being below the analysis report’s threshold. Any such argument has to also be presented personally at the beginning of the panel meeting so that the panel can vote to support or reject the proposition.

The additional analysis in the report consists of correlation tests between the panel members’ rankings and additional parameters to illuminate any potential relationship between them, which might indicate a certain degree of bias. We compare the rankings of panel members from the same or similar research areas to those of panel members from other research areas than the respective nomination and we also correlate the panel member’s rankings against the gender and age of the nominee. A correlation between rank and age would not necessarily be unexpected as more senior researchers have had more opportunity to excel. Nonetheless, these additional analyses can provide some insight into any potential systematic trends in the data, which might have to be addressed during the panel meeting, if they are of potential concern.

Along with the analysis report, the panel members are also provided with the full set of all original, non-anonymised nominations, now also including the prominence sections. The complete set of original nominations is offered to the panel members for their information only and they are not required to do any more work in preparation for the evaluation meeting at this point, except read the small set of nominations remaining above the threshold, in case they have not done so already as part of their own pre-evaluation sub-set. The panel members are, however, encouraged to read through all nominations and to revisit their own original sub-set, this time with added information such as name, gender, institution, and all the prestigious distinctions and recognitions outlined in the prominence section to the extent they feel necessary such that they can agree with the proposed thresholding and the resulting inclusion and exclusion of nominees. They can also use this information to argue against the thresholding if need be (see above).

Part 3: Evaluation meeting

The evaluation panel meeting is held online and led by two chairs. The academic chair is responsible for the content of the evaluation, and a separate, independent procedural chair is responsible for the evaluation process and compliance. The procedural chair guides the panel through the different steps of the evaluation. Whenever an individual nominee is discussed, all data on the respective nomination is displayed on screen: the panel members who pre-evaluated the nomination and the ranks and medals they assigned, the resulting position across all overall rankings and all achievement and impact medal rankings (Figure 2), the nomination synopsis, recusals, etc. The data allows for informed discussions (e.g. “nomination A ranked higher in achievements than nomination B”) and targeted questions (e.g. “why did panellist 7 give nomination C only a bronze medal for impact?”). Voting is conducted digitally and anonymously. All results are displayed as soon as the last vote is cast.

If a panel member at this stage strongly supports a nomination that they had originally ranked low, this becomes apparent quickly as all ranking information is displayed. Thus, if panellists change their mind because they now know who they are talking about, this can and should be discussed and challenged by the panel. The procedural chair is also tasked to look out for such discrepancies and highlight them, thus increasing transparency in the argumentation for or against a nomination.

All nominations, which are evaluated in the panel meeting (i.e. those that remained after thresholding), are discussed individually in detail to decide if they should be removed from the competition or forwarded to the final evaluation round. Each discussion starts off with a short plea against the respective nomination by the panel member who had pre-evaluated it and ranked it lowest compared to the other five pre-evaluators. Their statement is then followed by an argument for the nomination by the panel member who had ranked it highest during pre-evaluation. The two opening statements can then be commented upon by the remaining four pre-evaluators and then discussed further in the full panel. Finally, an anonymous vote is taken, whether the nomination should be put forward or whether it should be eliminated, before moving on the next nomination. To keep the process efficient, only a limited number of nominations can be brought forward to the final discussion (e.g. three to five nominations).

While the initial discussions strictly focus on individual nominations only (i.e. should this nomination be put forward for this prize), the final session opens the floor to comparative discussions across all nominations (i.e. should this nomination be awarded over that nomination). It nonetheless is equally carefully structured to ensure fairness. All nominations and panel members are given equal air-time and the procedural chair intervenes when secondary or anecdotal information about a nominee is introduced. Such information is neither transparent nor fair and can work only for or against nominees familiar to members of the panel. The two non-academic panel members have the same rights and responsibilities as all the other panel members throughout the evaluation, in this step they do have one additional task, however, which is to confirm officially to the panel that in their view the remaining nominations are indeed impactful beyond academia, giving them a certain veto right in that regard. Once the comparative discussion converges on a few finalists, votes are cast to decide on the top two candidates and ultimately the winner. The first vote askes the panel members to individually rank all remaining nominations. The rankings reveal the final top two candidates from which a second vote between the two finalists declares the winner. The second vote is necessary to ensure that the majority of the panel ultimately cast their vote for the final winner.