The jury is out: a new approach to awarding science prizes [version 1; peer review: awaiting peer review]

Research evaluation is often understood as something similar to a competition, where an evaluation panel’s task is to award the most excellent researchers. This interpretation is challenging, in as far as excellence it is at best a multi-dimensional concept and at worst an illdefined term because it assumes that there exists some ground truth as to who the very best researchers are and all that an evaluation panel needs to do is uncover this ground truth. Therefore, instead of focusing on competition, the Swiss National Science Foundation focused on active decision-making and sought inspiration in the deliberation proceedings of a jury trial for the design of a new evaluation procedure of an academic award. The new evaluation procedure is based upon fully anonymised documents consisting of three independent parts (achievements, impact and prominence). Before the actual evaluation meeting, the panel, which includes nonacademic experts, pre-evaluates all nominations through a pseudorandomly structured network, such that every nomination is reviewed by six members of the panel only. Evaluation decisions are based upon anonymous votes, structured discussions in the panel, ranking as opposed to rating of nominees and data-rich figures providing an overview of the positioning of the nominee along various dimensions and the ranking provided by the individual panel members. The proceedings are overseen by an academic chair, focusing on content, and a procedural chair, focusing on the process and compliance. Combined, these elements form a highly-structure deliberation procedure, consisting of individual steps, through which nominations proceed and which each either feed into the next step or into the final verdict. The proposed evaluation process has been successfully applied in the real world in the evaluation of the Swiss Science Prize Marcel Benoist, Switzerland’s most prestigious academic award.


Introduction
Prestigious academic prizes are usually awarded based on evaluation procedures where a group of experts ultimately decides who to award. However, during evaluation, experts may rely on inappropriate metrics, 1,2 and discussion dynamics in the group may stray, 3,4 which in turn can result in unfair processes 5,6 and unsatisfactory outcomes. [7][8][9][10] These circumstances are well known, and various initiatives, such as the San Francisco Declaration on Research Assessment (DORA), 11 the Leiden Manifesto, 12 the Metric Tide, 13 the Hong Kong Principles 14 and the h-group 15 have highlighted them and provided guidelines on how to improve upon them. Inspired by these initiatives, the Swiss National Science Foundation (SNSF) has devised a novel evaluation procedure for the evaluation of academic awards. Essential to its innovative features is our interpretation of research evaluation less as a competition and more as an active decision-making process.
The task of an evaluation panel is often understood as that of finding the strongest applicant. However, while evaluation panels can usually discern the weakest and strongest contestants easily, they frequently struggle to delineate unequivocally, who from the midfield they might also want to champion or discuss and which one to ultimately single out as the winner. 16 Comparison of scientific track records is multi-dimensional and the method of scoring individual dimensions as well as their respective weighting is controversial and often biased. Whereas in other competitions one can call upon millimetres and milliseconds as measures of merit, in research evaluation it is the formulation of a convincing argument within the evaluation panel, which ultimately calls the winner. Therefore, rather than thinking of research evaluation as something akin to a sports competition, it may be more useful to look to a different analogy, that of a jury trial. 17 A jury does not have privileged access to an objective ground truth as a referee standing at the finish line might. Instead, the justification of a jury as a legal decision-making body is that it is comprised of peers and that, based on the trial proceedings, at this moment no other group of individuals are more informed and/or better qualified to make a decision upon the matter laid before themeven if a differently comprised jury might arrive at a different conclusion. The same is true for a panel of experts deciding which researcher to award. They cannot objectively establish who is the best scientist. 18 Still, they can deliberate fairly and systematically upon why they might support one over another and can provide rational argumentation for this decision. It is important to note that a jury trial does not deny the presence of bias and predilection. Instead, it aims at controlling these forces by regulating deliberation and proceedings and making the final verdict democratic. 19 Jury trials have their own set of challenges. 20 However, in the presence of biases and other human limitations they intentionally work toward a fair and rational outcome by systematically structuring, segmenting and transparently formalising their proceedings. 17,19,20 This approach is not yet widespread among academic award committees, as a brief inquiry among the world's most established and prestigious academic prize institutions revealed (Box 1).

A proposal
Inspired by the analogy of a jury trial and the lack of such clear and transparent processes among academic prize committees, we propose a new award evaluation procedure. The procedure was also implemented in a first instance during the evaluation of the Swiss Science Prize Marcel Benoist (MBP), Switzerland's most prestigious academic award (see Application to the Swiss Science Prize Marcel Benoist, 21 ). The new evaluation procedure is based upon the following three core principles: Box 1: Evaluation guidelines for academic awards.
Upon inquiry, we found that for many prestigious academic awards there are either no detailed evaluation guidelines available or they are not publicly accessible. The following anonymised quotes are drawn from responses we received from some of the world's most prestigious academic award committees in response to the question: "[…] May I ask you to possibly send me (a link to) a description of the proceedings, which take place during the actual jury meeting (i.e. is there first a pre-selection or consultation? How are all the nominations reduced down to the final winners? Are there rules or customs for this process?). […]": "The reason you do not find detailed information regarding the exact proceedings of the […] Committee's work is that there are none." "The committee is entirely free to structure their work the way they find appropriate" "The Foundation keeps matters related to your query confidential" "[…] according to the statutes of the […] Foundation, the selection is treated confidentially" "While the members of our prize Selection Committees are public, the proceedings are not" "There are no written guidelines for how the committee decides upon a winner" • Research evaluation is an active decision-making process by the evaluation panel. It is not the description of some objective ground truth by onlookers. The evaluation proceedings need to be structured to handle the complexity of the evaluation task appropriately.
• Evaluation and the documents under scrutiny should comprise of clearly delineated individual parts such that the verdict can be synthesised from the sum of many individual smaller steps. Assessment should not consist of unstructured, open-ended discussions that try to simultaneously consider all aspects of monolithic evaluation documents.
• Each step of the evaluation procedure must be transparent and well-defined, easy to understand with a clearly formulated aim and comprehensible outcome, which in turn should form the basis of the next step of the evaluation and/or feed directly into the final verdict.
In addition to these three principles, we also designed the process to rely on preparatory work such that the actual evaluation meeting itself can be streamlined and managed online. Flying in international experts for evaluation meetings is arguable neither necessary nor sustainable. The evaluation process is divided into three parts: nomination, preevaluation and finally the evaluation panel meeting itself. Each part is subdivided into individual steps.

Part 1: Nomination
To nominate a potential awardee, nominators fill in clearly structured nomination documents with the help of an interactive online platform. The interface provides all necessary definitions and information and helps the nominator organise the nomination correctly into three individual sections: "achievements by the nominee"; "prominence endowed upon the nominee"; and "impact originating from the nominee's work". Achievements are defined in line with DORA as the Prominence, in contrast, is defined as "[…] prestigious distinctions and recognitions endowed upon the nominee by others as opposed to accomplishments attained by the nominee themselves [they] may include awards, titles, distinctions, nominations, and prestigious association such as employment in famous universities, collaborations with famous people or publications in famous journals etc (e.g., while the content of a seminal publication should be described in Achievements, the fact that it was published in a prestigious journal can be mentioned here, if you wish to do so)." The prominence section is included because many nominators and panel members still today explicitly or implicitly rely on, independently google or outright demand such measures, especially when evaluating highly prestigious prizes.
Rather than deceiving ourselves about these circumstances, we instead try to impose honesty and transparency in their use by strictly containing prominence information to one dedicated section. The definition of impact is the same as outlined in the assessment framework and guidance on submissions of the Research Excellence Framework 2014 (www.ref.ac. uk/2014), which defines impact as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia." Nominators are asked to keep their statements in the three sections concise and comprehensible to both an educated audience and non-expert members of the public, avoiding jargon. They are informed transparently about how the evaluation panel will use their statements. All information is provided directly within the online nomination mask such that no additional submission guidelines or further documentation are necessary. The same information texts are provided to the members of the evaluation panel. Separating the nomination text into three sections this way allowing the members of the evaluation panel to easily assess the actual achievements of the individual researcher independently of their prestigious associations, and the impact of their work without it being conflated with the work's quality. Furthermore, it allows evaluators to easily refer to, evaluate and comment on these different aspects of a nomination independently and to compare them individually across nominations.
Any claims made by nominators within the achievements and impact sections have to be supported by references. No more than ten references, distributed freely across these two sections, can be used. Citations are entered into the text as bracketed numbers (e.g. [3]). The actual reference itself, however, only consists of information about the reference type (e.g. "journal article", "book", "radio interview", "code" etc) and the respective abstract, synopsis or description, deliberately omitting any information about author, title, journal, publisher or publication date etc. Furthermore, upon submission of a nomination, the achievements and impact sections are anonymised such that they contain no names, gender or other identifying information but referred only to "the nominee" or the gender-neutral pronoun "they". Any mention of, for example, institutions, collaborators or associations are also replaced with neutral references such as "at a Swiss university", "with an established collaborator" or "as editor of an established journal" etc. Fully anonymised nominations can then be generated by separating the prominence section from the now anonymised achievements and impact sections. These fully anonymised nominations, consisting only of the achievements and impact sections with anonymised citations and without any mention of institutions or journal names etc, are then sent to evaluation panel members for pre-evaluation ( Figure 1A).

Part 2: Pre-evaluation
Nominations are sent out to the individual panel members to review ahead of the panel meeting. The panel is comprised of at least seven and up to 11 members. In addition to international academic experts, it also includes two non-academic representatives of society and is age-and gender-balanced.
The nominations are distributed in a systematic, pseudo-random manner. They are assigned to panellists randomly albeit within the confines that every nomination is reviewed by six members of the panel only and that those six reviewers always include one of the two non-academic panel members and the panel's topic-expert (e.g. all psychology nominations are read by the psychologist in the panel etc, Figure 1B). Furthermore, we ensure that each panel member's individual collection of nominations to review is different to all others panel members' sub-sets such that each nomination is compared to different nominations in every case and every nomination is compared to every other nomination at least once ( Figure 1B). Importantly, this network distribution of nominations means that each panel member has to only preevaluate a sub-set of nominations (the number of which will depend on the total number of submitted nominations) instead of all of them, thus reducing the burden on the evaluators. In turn, the evaluation panel members have to commit to reading through their respective sub-set of nominations in full, including all the abstracts provided in the reference list, while also formally and explicitly committing to not seeking out additional information beyond what was provided (i.e. no googling of nominees etc).
The panel members are asked to pre-evaluate their respective sub-set of nominations in two steps. In the first step, they only receive the fully anonymised achievements and impact sections of the nominations in their respective sub-set consists of the fully anonymised achievement (Ach) and impact (Imp) sections only (without the prominence sections). References contain only a description of the reference type (e.g. "Journal Article") and the respective abstract or synopsis. (B) "Nomination C" is then reviewed by a subset of evaluation panel members (here panellist 3 through 7), who each compare the nomination to a different set of other nominations respectively (in this case with some overlap). Each panellist ranks their set according to their personal overall preference (i.e. "who should win the prize?", overall ranking (OR)) and additionally distribute gold (G), silver (S) and bronze (B) medals for the achievements and impact section respectively.
(i.e. without the prominence section, Figure 1A). These anonymised nominations have to be monotonically ranked according to the evaluation criteria and the individual panel member's personal assessment. Additionally, panel members are asked to assign an equal number of gold, silver and bronze medals across the individual achievements and impact sections respectively. If, for example, a panel member has to pre-evaluate 15 nominations in their sub-set, they would have to separately assign five bronze, five silver and five gold medals for achievements and for impact, respectively. The medals provide an intuitive way of generating tercile rankings of achievements and impact, in line with the overall ranking (i.e. first is better than second etc., Figure 1B). Finally, we ask panel members to guess who the nominated person is, if they believed they have an idea, to keep tabs on the extent to which anonymisation was successful.
All assessments throughout the whole evaluation procedure are always based on rankings instead of ratings. The advantage of using rankings (specifically weaker than/ stronger than) instead of ratings (generally weak/strong) is that they are internally normalised, thus mitigating the risk that a very generous or overly strict panel member might skew the evaluation. For example, by strictly specifying the number of gold, silver and bronze medals that can be allocated by a panel member, the chance of receiving more or fewer medals of a particular kind is independent of the individual panel members to whom the nomination is allocated. Instead, it is only a function of those individual panel members' evaluation of the respective nomination compared to the other nominations within their sub-set.
To further facilitate comparison across nominations, average and median values are calculated after pre-evaluation for each nomination based on the overall ranking and the medals it receives from the panel members who pre-evaluated it. The average overall ranking of a nomination is calculated as the mean value across its individual sub-set rankings. The average achievement and impact medals are calculated as the mean values across its individually assigned medals [defined as Gold = 1, Silver = 2, Bronze = 3 (Figure 2A), calculations of the median scores are done analogously]. These statistics are simple though calculating the mean across ranking data is, to some extent imperfect. However, presenting data in this very straight forward form allows all panel members to always easily and intuitively understand the statistics and their underlying data, which is crucial if they are to accountably argue based upon them ( Figure 2).
In the second step of pre-evaluation, after returning their rankings, the panel members are provided with a detailed report and the full set of non-anonymised nominations. The analysis report provides three important results: (1)  The threshold is drawn based upon the following rule: "Include only nominees with a mean AND median rank of seven or better in the overall ranking, OR a mean AND median rank of three or better in the achievement OR impact ranking." The panel members are then asked to indicate in writing whether they agree with the analysis and thresholding of nominees as described in the analysis report; or feel that one of the excluded nominees should still be included in the further evaluation. In the latter case, the panel member has to then also state in writing why the respective nominee should be singled out despite being below the analysis report's threshold. Any such argument has to also be presented personally at the beginning of the panel meeting so that the panel can vote to support or reject the proposition.
The additional analysis in the report consists of correlation tests between the panel members' rankings and additional parameters to illuminate any potential relationship between them, which might indicate a certain degree of bias. We compare the rankings of panel members from the same or similar research areas to those of panel members from other research areas than the respective nomination and we also correlate the panel member's rankings against the gender and age of the nominee. A correlation between rank and age would not necessarily be unexpected as more senior researchers have had more opportunity to excel. Nonetheless, these additional analyses can provide some insight into any potential systematic trends in the data, which might have to be addressed during the panel meeting, if they are of potential concern.
Along with the analysis report, the panel members are also provided with the full set of all original, non-anonymised nominations, now also including the prominence sections. The complete set of original nominations is offered to the panel members for their information only and they are not required to do any more work in preparation for the evaluation meeting at this point, except read the small set of nominations remaining above the threshold, in case they have not done so already as part of their own pre-evaluation sub-set. The panel members are, however, encouraged to read through all nominations and to revisit their own original sub-set, this time with added information such as name, gender, institution, and all the prestigious distinctions and recognitions outlined in the prominence section to the extent they feel necessary such that they can agree with the proposed thresholding and the resulting inclusion and exclusion of nominees. They can also use this information to argue against the thresholding if need be (see above).

Part 3: Evaluation meeting
The evaluation panel meeting is held online and led by two chairs. The academic chair is responsible for the content of the evaluation, and a separate, independent procedural chair is responsible for the evaluation process and compliance. The procedural chair guides the panel through the different steps of the evaluation. Whenever an individual nominee is discussed, all data on the respective nomination is displayed on screen: the panel members who pre-evaluated the nomination and the ranks and medals they assigned, the resulting position across all overall rankings and all achievement and impact medal rankings (Figure 2), the nomination synopsis, recusals, etc. The data allows for informed discussions (e.g. "nomination A ranked higher in achievements than nomination B") and targeted questions (e.g. "why did panellist 7 give nomination C only a bronze medal for impact?"). Voting is conducted digitally and anonymously. All results are displayed as soon as the last vote is cast.
If a panel member at this stage strongly supports a nomination that they had originally ranked low, this becomes apparent quickly as all ranking information is displayed. Thus, if panellists change their mind because they now know who they are talking about, this can and should be discussed and challenged by the panel. The procedural chair is also tasked to look out for such discrepancies and highlight them, thus increasing transparency in the argumentation for or against a nomination. Replacing the medals with numbers (gold = 1, sliver = 2, bronze = 3), allows for the calculation of ranks (here only the average ranks are shown), which in turn allows for a positioning of each nomination relative to the others for the overall ranking (OR), the achievement ranking (ACH) and the impact ranking (IMP) respectively (plots). (B) In actual fact, the information provided in these rank plots needs to be substantially richer to be genuinely informative. It consists of the number of panellists who evaluated a nomination (n), the mean and median ranking value (red and blue dots respectively), the raw individual rankings from the panellists (gray dots) as well as the mean and median ranking (right y-axis) for each nomination (shown is the normalised overall ranking across 24 nominations based upon real-world evaluation plots but populated with random data for data-protection reasons, similar plots are also generated for the achievement and impact rankings respectively).
All nominations, which are evaluated in the panel meeting (i.e. those that remained after thresholding), are discussed individually in detail to decide if they should be removed from the competition or forwarded to the final evaluation round. Each discussion starts off with a short plea against the respective nomination by the panel member who had pre-evaluated it and ranked it lowest compared to the other five pre-evaluators. Their statement is then followed by an argument for the nomination by the panel member who had ranked it highest during pre-evaluation. The two opening statements can then be commented upon by the remaining four pre-evaluators and then discussed further in the full panel. Finally, an anonymous vote is taken, whether the nomination should be put forward or whether it should be eliminated, before moving on the next nomination. To keep the process efficient, only a limited number of nominations can be brought forward to the final discussion (e.g. three to five nominations).
While the initial discussions strictly focus on individual nominations only (i.e. should this nomination be put forward for this prize), the final session opens the floor to comparative discussions across all nominations (i.e. should this nomination be awarded over that nomination). It nonetheless is equally carefully structured to ensure fairness. All nominations and panel members are given equal air-time and the procedural chair intervenes when secondary or anecdotal information about a nominee is introduced. Such information is neither transparent nor fair and can work only for or against nominees familiar to members of the panel. The two non-academic panel members have the same rights and responsibilities as all the other panel members throughout the evaluation, in this step they do have one additional task, however, which is to confirm officially to the panel that in their view the remaining nominations are indeed impactful beyond academia, giving them a certain veto right in that regard. Once the comparative discussion converges on a few finalists, votes are cast to decide on the top two candidates and ultimately the winner. The first vote askes the panel members to individually rank all remaining nominations. The rankings reveal the final top two candidates from which a second vote between the two finalists declares the winner. The second vote is necessary to ensure that the majority of the panel ultimately cast their vote for the final winner.

Application to the Swiss Science Prize Marcel Benoist
The MBP is the most prestigious academic prize in Switzerland. Founded in 1920 it awards scholars who "[…] made the most useful discovery […] that is of particular relevance to human life" and is therefore in its essence an impact prize. The award is rotated tri-annually through natural, biological and medical sciences, and the humanities and social sciences.
Since 2018 the Swiss National Science Foundation is responsible for the evaluation of the MBP, in partnership with the Marcel Benoist Foundation.
The process presented here has been successfully applied to the evaluation of the MBP four times so far. 21 The individual nomination sections consisted of: achievement, 800 words; impact, 800 words; and prominence, 500 words. Anonymisation took roughly four hours per nomination. However, not sending nominations out for external peer review also saved time, making the overall preparation efforts relatively efficient. Pre-evaluation and thresholding worked well and in one instance predicted the winner. Discussions during the evaluation meeting relied significantly on the information provided in the on-screen pre-evaluation data resulting in good discussion dynamics and transparent decision-making processes.
Compliance of individual panel members (e.g. no googling of nominations) and overall adherence to DORA guidelines (e.g. weighting the actual achievements of a nomination higher than the journal in which it was published) were some of the main challenges, which we encountered during the implementation of this process; they are, however, not unique or particular to this evaluation procedure. The process presented here heavily relies on preparation ahead of the actual evaluation meeting; due diligence in vetting nominations and good preparation especially by the procedural chair are crucial to ensure smooth and efficient evaluation. Overall, the experience with this evaluation procedure has been very positive and the SNSF will continue using it for the evaluation of the MBP and possibly other prizes in the future and some of its aspects are now also being adapted to other funding procedures.

Discussion
Many ideas, both tried and new, were combined in the design of this evaluation procedure 22,23 : interpreting evaluation as more similar to a jury trial than a competition, segmenting and anonymising nominations, pre-evaluation, selectively assigning nominations to a network of panellists, using rankings instead of ratings, including non-academic experts in the evaluation panel, conducting voting anonymously and through dedicated infrastructure, displaying all nominee-specific and comparative evaluation data to the panel during discussions, dividing up the chairing duties between an academic and a procedural chair. Together, these innovations form a coherent, transparent and well-defined structure of individual steps, through which nominations proceed and within which panel members are supported in their deliberations.
Despite best efforts, for prestigious awards some nominees will almost always be familiar to at least some panellists while others may remain unrecognised. At the same time, maintaining the anonymity of nominations during panel discussions is difficult and often not feasible. Acknowledging these challenges, we propose a pragmatic middle ground where nominations are kept anonymous during pre-evaluation but anonymity is then lifted in a controlled manner ahead of the meeting allowing for non-anonymous discussions. 23,24 This approach provides the added benefit that it can also highlight biases originating solely from personal or prominence-based information as they stand in visible contrast to the pre-evaluation ranking based on the anonymised texts. Fully automated anonymisation for academic texts is, to our knowledge, not available or not reliable enough. However, a surge in privacy requirements, not least due to the General Data Protection Regulation (EU) 2016/679 have led to a proliferation of anonymisation tools, 25 which, for example in combination with Research Organization Registry data, could, in future, be adapted for such purposes.
The evaluation process described here relies on nomination texts, which is unfortunate in those cases where a superficial nomination may disadvantage an otherwise very strong nominee. It is important to make nominators aware of this potential confound and the importance of a high-quality nomination on their part. Tampering with the process by, for example, adding additional information where needed, is not appropriate as it undermines the very essence of a nomination award. Such intervention leads down a slippery slope where one might as well just ask for the name of the person to be nominated and then leave the collection and selection of arguments to dedicated experts.
Nomination awards can also result in multiple nominations being submitted for some nominees but not for others. This can create an unfair advantage as multiple nomination texts provide opportunity to describe different accomplishments and to cite more works than just one text (although there will of course be some overlap between nominations). At the same time, multiple nominations may generate an implicit bias simply by suggesting that "more people think this person should win than that person". Both situations need to be avoided, as the winner should be determined by the evaluation panel and not a majority vote from nominators.
To control for such circumstances, one can divide up multiple nominations for the same nominee among the preevaluators such that, for example, three panel members pre-evaluate only one and the other three the other nomination.
During the second step of pre-evaluation, both nominations can then be provided along with all other non-anonymised information. As with the prominence section, panel members can now still change their mind based upon the additional information but any such changes are again highlighted due to their contrast to the original ranking data, allowing for the panel to call them out and discuss them transparently.
DORA encourages that evaluators read the work they are asked to assess as opposed to relying on short-cuts such as journal-derived metrics. 11 However, if evaluation organisers expect their evaluators to adhere to these guidelines, then they must also ensure that evaluators will actually be able to fulfil this mandate. You cannot commit to DORA and at the same time send out publication lists that are too long for evaluators to realistically read with due diligence. To address this problem, we only allowed for ten references to be cited across all three text sections (achievements, impact, prominence). However, for every ten nominations this would nevertheless add up to 100 papers to read, which is still a lot. Reducing the number of references even further may be difficult as academic awards, such as the MBP, are often given for a large body of work as opposed to a single project. Instead, we provided an abstract for each of the ten references and in turn asked the panel members to commit to reading all abstracts for every nomination in their sub-set. Providing abstracts instead of citations or full text publications also allowed us to maintain anonymity during pre-evaluation.
Researchers have a right to fair evaluation as they can make or break their careers. 26 Furthermore, small systematic biases can sum up to damage progress. 27 Despite best efforts, evaluators and evaluation processes are frequentlyand to some extent inevitablyimplicitly biased, and nominees may therefore not always be given their fair chance. The underrepresentation of women in academia generally and amongst academic awardees specifically is one example of the cumulative impact that biases at every stage of a scientific career can have. [28][29][30] We believe many of the techniques discussed here could be transferred to other prize evaluations and assessments at other career stages, such as the hiring of faculty and the awarding of fellowships or grants. We hope that they may inspire change. High-quality and fair research evaluation requires not only a change in culture and a commitment to do better but most importantly also the actual implementation of fair and transparent processes.
Research evaluation functions as a gatekeeper of science. By assessing grant proposals in funding organisations, we determine who will be a scientist and what research will be conducted. By evaluating submissions to academic journals, we determine which research will be communicated to which audience. By evaluating nominees for academic prizes, we decide what research we want to celebrate and what academic role models we create for future generations. It is paramount to ensure that we can argue coherently and transparently how these important verdicts are reached in each of these situations.

Data availability
No data are associated with this article The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias • You can publish traditional articles, null/negative results, case reports, data notes and more • The peer review process is transparent and collaborative • Your article is indexed in PubMed after passing peer review • Dedicated customer support at every stage • For pre-submission enquiries, contact research@f1000.com