Metrics
Views
4054
Downloads
686
Get PDF
Get XML
Cite
Export
Track
Research Note
Revised  An Open Science Peer Review Oath [version 2; referees: 4 approved, 1 approved with reservations]
Jelena Aleksic1Adrian Alexa2Teresa K Attwood3[...] Neil Chue Hong9Martin Dahlö4Robert Davey5Holger Dinkel6Konrad U Förstner7Ivo Grigorov8Jean-Karim Hériché6Leo Lahti10Dan MacLean11Michael L Markie12Jenny Molloy13Maria Victoria Schneider5Camille Scott14Richard Smith-Unna15Bruno Miguel Vieira16as part of the AllBio: Open Science & Reproducibility Best Practice Workshop
Jelena Aleksic1Adrian Alexa2[...] Teresa K Attwood3Neil Chue Hong9Martin Dahlö4Robert Davey5Holger Dinkel6Konrad U Förstner7Ivo Grigorov8Jean-Karim Hériché6Leo Lahti10Dan MacLean11Michael L Markie12Jenny Molloy13Maria Victoria Schneider5Camille Scott14Richard Smith-Unna15Bruno Miguel Vieira16as part of the AllBio: Open Science & Reproducibility Best Practice Workshop
PUBLISHED 09 Jan 2015
Revised
Author affiliations Author Affiliations
Grant information Grant Information
OPEN PEER REVIEW
REFEREE STATUS
This article is included in the The Future of Scholarly Publishing channel.

Abstract

One of the foundations of the scientific method is to be able to reproduce experiments and corroborate the results of research that has been done before. However, with the increasing complexities of new technologies and techniques, coupled with the specialisation of experiments, reproducing research findings has become a growing challenge. Clearly, scientific methods must be conveyed succinctly, and with clarity and rigour, in order for research to be reproducible. Here, we propose steps to help increase the transparency of the scientific method and the reproducibility of research results: specifically, we introduce a peer-review oath and accompanying manifesto. These have been designed to offer guidelines to enable reviewers (with the minimum friction or bias) to follow and apply open science principles, and support the ideas of transparency, reproducibility and ultimately greater societal impact. Introducing the oath and manifesto at the stage of peer review will help to check that the research being published includes everything that other researchers would need to successfully repeat the work. Peer review is the lynchpin of the publishing system: encouraging the community to consciously (and conscientiously) uphold these principles should help to improve published papers, increase confidence in the reproducibility of the work and, ultimately, provide strategic benefits to authors and their institutions.

One of the foundations of the scientific method is to be able to reproduce experiments and corroborate the results of research that has been done before. However, with the increasing complexities of new technologies and techniques, coupled with the specialisation of experiments, reproducing research findings has become a growing challenge. Clearly, scientific methods must be conveyed succinctly, and with clarity and rigour, in order for research to be reproducible. Here, we propose steps to help increase the transparency of the scientific method and the reproducibility of research results: specifically, we introduce a peer-review oath and accompanying manifesto. These have been designed to offer guidelines to enable reviewers (with the minimum friction or bias) to follow and apply open science principles, and support the ideas of transparency, reproducibility and ultimately greater societal impact. Introducing the oath and manifesto at the stage of peer review will help to check that the research being published includes everything that other researchers would need to successfully repeat the work. Peer review is the lynchpin of the publishing system: encouraging the community to consciously (and conscientiously) uphold these principles should help to improve published papers, increase confidence in the reproducibility of the work and, ultimately, provide strategic benefits to authors and their institutions.

Revised  Amendments from Version 1

We have taken into consideration all of the reviews and comments, and have made the manuscript shorter and clearer. We have further discussed what we hope to achieve from researchers using the oath, and we have balanced the article to explain how we believe the oath can encourage the practice of open science. The oath itself now comprises four concise principles, and we elucidate the open-science principle with accompanying guidelines, which should make it easier to apply in practice.

See referee responses

Open science and the future of peer-review

Science builds on itself. New knowledge is gained in the context of the enlightenment of earlier discoveries and, for the foundations to remain solid each discovery must be accurate and reliable. But science in the real world is messy, and advances haltingly and piecewise. Often, prior information is incomplete, so conclusions drawn need revising in the light of new, later evidence. This means that self-reference and self-checking are cornerstones of the scientific method; and, having reported our experiments, it is vital that readers are able to repeat them and understand how we reached our conclusions. Various studies have shown that, for a study to be successfully reproduced, information presented in the literature is often inadequate and the underlying data not readily available1 - a significant drawback. Several commentators have concluded that weaknesses in the way that research investigations are currently conducted, and how their results are disseminated via article publication have become detrimental to the scientific process27. As the technological sophistication of science increases, and the equipment used becomes more specialised, the data generated are harder to represent in traditional media, and reporting how experiments were performed so that independent researchers can repeat them becomes progressively harder.

Open science is a movement that seeks to ensure that the results and the data of scientific research are, and continue to be, available to all. One way in which reproducibility issues can be tackled is through the use of open-science and open-data practices8,9. As attendees of the AllBio: Open Science & Reproducibility Best Practice Workshop, we discussed how the problem of keeping science transparent and reproducible in an increasingly technology-driven, and specialised, domain could be addressed.

One route, at the heart of scientific endeavour, is through the peer-review process. Peer review is an important gatekeeper and key component of scientific discourse. Before research findings can be formally accepted, they must be evaluated and commented upon by other domain experts, who provide advice about the quality or validity of the work to journal editors and/or readers (whether via pre-publication peer review, open peer review or post-publication peer-review systems). Importantly, peer review happens at a personal, rather than institutional, level and is carried out by individuals; it is therefore an ideal mechanism for engaging most researchers, given that all scientists peer review or are peer reviewed.

Peer review oaths

The peer-review process is not infallible1012; its weaknesses are many and varied, including the time available to perform thorough reviews, reviewers’ expertise, journals’ perception of relevance/interest/impact, and so on. Arguably, one of the most significant problems – certainly the one that generates most friction – is that reviewers can safely dispense self-serving and biased critiques, fully protected by the mask of anonymity. In some instances, anonymity may be the only pragmatic option, but in the interest of objectivity and constructive discourse it shouldn’t be the cornerstone of the peer-review process. Scientists have become sufficiently frustrated by these issues to devise ad hoc solutions to help safeguard the quality of reviews, and allow reviewers to affirm that they will review in an ethical and professional way, and encourage clearer review processes. This has led to the articulation of various reviewer’s oaths (e.g.,1315); it is these that inspired our discussions and prompted this article.

An outcome of our debate was the formulation of an addition to these oaths that helps codify the role of reviewers, encouraging and promoting best practices to ensure that the science they review is as open and reproducible as possible. The amendment includes guidelines not only on how to review professionally, but also on how to support transparent, reproducible and responsible research, while optimising its societal impact and maximizing its visibility.

The open-science peer-review oath (which can be found on figshare for reuse16) we therefore propose is as follows (Box 1).

Box 1. Open peer review oath.

Principle 1: I will sign my name to my review

Principle 2: I will review with integrity

Principle 3: I will treat the review as a discourse with you; in particular, I will provide constructive criticism

Principle 4: I will be an ambassador for the practice of open science

Towards ‘open science friendly’ reviews

The declaration in your review

Peer-review oaths tend to be short declarations that reviewers make at the start of their written comments, typically dictating the terms by which they will conduct their reviews. The novelty in our amendment is the addition of a new declaration, which we recommend including in review preambles:

I will be an ambassador for open science

For it to be successful requires reviewers to follow the spirit of the statement throughout their reviews, which may involve a little additional work. The statement itself is derived from the rationale outlined below. It is for reviewers to decide how to implement this on a case-by-case basis – it is case specific, but nevertheless provides a framework for nurturing useful and hopefully fruitful discussions.

Guidelines for open science reviewers

Here is how we envisage the review process from an open-science reviewer's point of view:

  • I will work with you to help improve your research, as I believe that peer review should be an open, supportive and collaborative process. I will therefore sign my review and state my identity.

  • I will, before I review, ensure that I have everything (raw data, detailed methods/protocols, etc.) necessary to make a complete, independent assessment.

  • I will encourage the application, and, where necessary, provide guidance for any open-science best practices relevant to my field that would help support transparency, reproducibility, re-use and integrity of your research.

  • I will check that any data and software code are consistent with the text, that any digital object identifiers and accession numbers are correct and correctly cited, and that any models presented are archived, referenced and accessible.

  • I will check that the data and any software code and support documentation that underpin the published concept are made available in a manner that provides long-term unrestricted access.

  • I will advise, where necessary, on how to achieve better transparency and availability (in terms of materials and methodology, data and code access, versioning, algorithms, software parameters and standards), with the understanding that adherence to best open-science practices may require further effort from you.

  • I will support others in writing open reviews, where it is appropriate for me to do so.

  • I will decline to review if I am not an appropriate reviewer (whether because of my expertise or because of my relationship with the author(s)). In doing so, I will provide journal editors with an honest appraisal of these issues, and will openly explain how I reached my decision so that alternative reviewers may be found.

Conclusion

We believe that combining the open-science principle with some of the other key peer-review principles will help the scientific community to repeat published experiments, and to reach the same, or similar, conclusions.

Author affiliations Author Affiliations
Article Versions (2)
Copyright
Download
 
Export To
metrics
VIEWS
4054
 
downloads
686
Citations
0
 
0
CITE
how to cite this article
Aleksic J, Alexa A, Attwood TK et al. An Open Science Peer Review Oath [version 2; referees: 4 approved, 1 approved with reservations] F1000Research 2015, 3:271 (doi: 10.12688/f1000research.5686.2)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
track
receive updates on this article
Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Referee Status: ?
Key to Referee Statuses VIEW
ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservationsKey revisions are required to address specific details and make the paper fully scientifically sound
Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions
Read more about the unique F1000Research publication and peer review model here.

Version 2
VERSION 2
PUBLISHED 09 Jan 2015
Revised
Views
46
Cite
Referee Report 02 Feb 2015
Chris Chambers, School of Psychology, Cardiff University, Cardiff, UK 
Approved
VIEWS 46
The authors have responded ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Chambers C. Referee Report For: An Open Science Peer Review Oath [version 2; referees: 4 approved, 1 approved with reservations]. F1000Research 2015, 3:271 (doi: 10.5256/f1000research.6351.r7278)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
Views
51
Cite
Referee Report 22 Jan 2015
Lawrence P Kane, Department of Immunology, University of Pittsburgh, Pittsburgh, PA, USA 
Approved
VIEWS 51
The authors have done a nice job responding to my critique of the original version, and the resulting revised version is much easier to read. Reading the other reviews, and the comments, I can see that this article is sparking ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Kane L. Referee Report For: An Open Science Peer Review Oath [version 2; referees: 4 approved, 1 approved with reservations]. F1000Research 2015, 3:271 (doi: 10.5256/f1000research.6351.r7277)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
Views
45
Cite
Referee Report 20 Jan 2015
Etienne Joly, Equipe de Neuro-Immunogénétique Moléculaire (ENIGM), Bâtiment CNRS, IPBS CNRS Université Paul Sabatier, Toulouse, France 
Approved
VIEWS 45
The manuscript has been improved compared to the first version. Despite the authors laudable intentions, I am still unconvinced that this will prove terribly useful to the scientific community, but I do not see any reason why this paper should ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Joly E. Referee Report For: An Open Science Peer Review Oath [version 2; referees: 4 approved, 1 approved with reservations]. F1000Research 2015, 3:271 (doi: 10.5256/f1000research.6351.r7275)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
Views
63
Cite
Referee Report 12 Jan 2015
Vitaly Citovsky, Department of Biochemistry and Cell Biology, University of New York at Stony Brook, Stony Brook, NY, USA 
Approved with Reservations
VIEWS 63
The revised paper is much improved in terms of its (now appropriate and not tedious) length and clarity of statements. I still see a problem with the proposed absolute requirement (listed as the first principle) of disclosing the reviewer's identity. ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Citovsky V. Referee Report For: An Open Science Peer Review Oath [version 2; referees: 4 approved, 1 approved with reservations]. F1000Research 2015, 3:271 (doi: 10.5256/f1000research.6351.r7276)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
Version 1
VERSION 1
PUBLISHED 12 Nov 2014
Views
56
Cite
Referee Report 10 Dec 2014
Lawrence P Kane, Department of Immunology, University of Pittsburgh, Pittsburgh, PA, USA 
Approved with Reservations
VIEWS 56
The principles outlined here are important, and this piece is certainly timely. I have two suggestions to improve the manuscript.  First, as currently constituted, the manuscript is a bit repetitive, with a large text box recapitulating what is also laid ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Kane L. Referee Report For: An Open Science Peer Review Oath [version 2; referees: 4 approved, 1 approved with reservations]. F1000Research 2015, 3:271 (doi: 10.5256/f1000research.6078.r6800)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
  • Author Response 12 Jan 2015
    Dan MacLean, The Sainsbury Laboratory, UK
    12 Jan 2015
    Author Response
    Thanks for your helpful and candid report Larry. We agree the manuscript was a little repetitive and so we have made it much more succinct, by concentrating on the open ... Continue reading
COMMENTS ON THIS REPORT
  • Author Response 12 Jan 2015
    Dan MacLean, The Sainsbury Laboratory, UK
    12 Jan 2015
    Author Response
    Thanks for your helpful and candid report Larry. We agree the manuscript was a little repetitive and so we have made it much more succinct, by concentrating on the open ... Continue reading
Views
48
Cite
Referee Report 09 Dec 2014
Suzanne Scarlata, Department of Physiology and Biophysics, Stony Brook University, Stony Brook, NY, USA 
Approved
VIEWS 48
I think this article makes many good points, and I also agree with the other critiques. However, I do see danger in full transparency. The problem is that significance of study can be  subjective and used in a biased way ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Scarlata S. Referee Report For: An Open Science Peer Review Oath [version 2; referees: 4 approved, 1 approved with reservations]. F1000Research 2015, 3:271 (doi: 10.5256/f1000research.6078.r6987)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
  • Author Response 12 Jan 2015
    Dan MacLean, The Sainsbury Laboratory, UK
    12 Jan 2015
    Author Response
    Thanks Suzanne for your helpful and candid report. We have now satisfied the other reviews to make the oath clearer and simpler to use. We take your point that perceived ... Continue reading
COMMENTS ON THIS REPORT
  • Author Response 12 Jan 2015
    Dan MacLean, The Sainsbury Laboratory, UK
    12 Jan 2015
    Author Response
    Thanks Suzanne for your helpful and candid report. We have now satisfied the other reviews to make the oath clearer and simpler to use. We take your point that perceived ... Continue reading
Views
67
Cite
Referee Report 27 Nov 2014
Etienne Joly, Equipe de Neuro-Immunogénétique Moléculaire (ENIGM), Bâtiment CNRS, IPBS CNRS Université Paul Sabatier, Toulouse, France 
Approved with Reservations
VIEWS 67
This manuscript was written by participants of a workshop entitled "AllBio: Open Science & Reproducibility Best Practice Workshop" which took place at the TGAC in Norwich UK in September 2014. The alleged purpose of the oath and manifesto proposed in ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Joly E. Referee Report For: An Open Science Peer Review Oath [version 2; referees: 4 approved, 1 approved with reservations]. F1000Research 2015, 3:271 (doi: 10.5256/f1000research.6078.r6802)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
  • Author Response 12 Jan 2015
    Dan MacLean, The Sainsbury Laboratory, UK
    12 Jan 2015
    Author Response
    Thanks Etienne for your very helpful and candid report.  We agree that the article needed to be made clearer in order to convey its intentions to the reader and we ... Continue reading
COMMENTS ON THIS REPORT
  • Author Response 12 Jan 2015
    Dan MacLean, The Sainsbury Laboratory, UK
    12 Jan 2015
    Author Response
    Thanks Etienne for your very helpful and candid report.  We agree that the article needed to be made clearer in order to convey its intentions to the reader and we ... Continue reading
Views
95
Cite
Referee Report 26 Nov 2014
Chris Chambers, School of Psychology, Cardiff University, Cardiff, UK 
Approved with Reservations
VIEWS 95
Overall, I believe this is a laudable proposal for a code of practice in academic peer review. In one sense it would be nice if such a code was unnecessary; after all, most of the practices outlined here should form ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Chambers C. Referee Report For: An Open Science Peer Review Oath [version 2; referees: 4 approved, 1 approved with reservations]. F1000Research 2015, 3:271 (doi: 10.5256/f1000research.6078.r6797)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
  • Author Response 09 Jan 2015
    Michael Markie, F1000Research, UK
    09 Jan 2015
    Author Response
    Thanks Chris for your very helpful and candid report. Like all of the reviewers have suggested we have simplified the oath and made it much clearer and simpler to use. ... Continue reading
COMMENTS ON THIS REPORT
  • Author Response 09 Jan 2015
    Michael Markie, F1000Research, UK
    09 Jan 2015
    Author Response
    Thanks Chris for your very helpful and candid report. Like all of the reviewers have suggested we have simplified the oath and made it much clearer and simpler to use. ... Continue reading
Views
61
Cite
Referee Report 25 Nov 2014
Vitaly Citovsky, Department of Biochemistry and Cell Biology, University of New York at Stony Brook, Stony Brook, NY, USA 
Approved with Reservations
VIEWS 61
This article addresses a very important issue of peer review. Although, many of us tend to regard it in a way of "if it ain't broke, don't fix it", we also are often frustrated with the process and wish it ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Citovsky V. Referee Report For: An Open Science Peer Review Oath [version 2; referees: 4 approved, 1 approved with reservations]. F1000Research 2015, 3:271 (doi: 10.5256/f1000research.6078.r6804)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
  • Author Response 12 Jan 2015
    Dan MacLean, The Sainsbury Laboratory, UK
    12 Jan 2015
    Author Response
    Thanks Vitaly for your very helpful and candid report. We have now explained what we hope to achieve from the oath, how we think it could address the issue of ... Continue reading
COMMENTS ON THIS REPORT
  • Author Response 12 Jan 2015
    Dan MacLean, The Sainsbury Laboratory, UK
    12 Jan 2015
    Author Response
    Thanks Vitaly for your very helpful and candid report. We have now explained what we hope to achieve from the oath, how we think it could address the issue of ... Continue reading

Comments on this article Comments (5)

Version 2
VERSION 2 PUBLISHED 09 Jan 2015
Revised
  • Reader Comment 13 Jan 2015
    Daniel S. Katz, University of Chicago, USA
    13 Jan 2015
    Reader Comment
    I think this version is a good improvement - it removes most of the extra work by the reviewer that seemed to be implied in the previous version, and the ... Continue reading
  • Author Response 12 Jan 2015
    Dan MacLean, The Sainsbury Laboratory, UK
    12 Jan 2015
    Author Response
    Thanks for your comment Daniel, we appreciate your thoughts. We have now simplified the principles of the oath and devised guidelines that concentrates solely on the open science principle. We ... Continue reading
  • Author Response 12 Jan 2015
    Dan MacLean, The Sainsbury Laboratory, UK
    12 Jan 2015
    Author Response
    Thanks Jonathan for your very helpful and candid thoughts. We agree with you that the oath was indeed too long and not as impactful as it could be. For this ... Continue reading
Version 1
VERSION 1 PUBLISHED 12 Nov 2014
Discussion is closed on this version, please comment on the latest version above.
  • Reader Comment 22 Dec 2014
    Daniel S. Katz, University of Chicago, USA
    22 Dec 2014
    Reader Comment
    ​The suggested oath, broken down by principles and items, is quite interesting, with some of the principles having clear value and little overhead, while others that have potential value also ... Continue reading
  • Reader Comment (Member of the F1000 Faculty and F1000Research Advisory Board Member) 13 Nov 2014
    Jonathan Eisen, University of California Davis Medical Center, USA
    13 Nov 2014
    Reader Comment Member of the F1000 Faculty F1000Research Advisory Board Member
    I have written a mini review of the paper on the UC Davis "Innovating Communication in Scholarship" blog: http://icis.ucdavis.edu/?p=505.
    Competing Interests: I am an informal/formal advisor to F1000 on some of their open science activities though I had no role in this paper.
  • Discussion is closed on this version, please comment on the latest version above.
CHANNEL
This article is included in the following channel:
DOI: 10.12688/f1000research.6005.2
Observation Article
Open peer review at four STEM journals: an observational overview [version 2; referees: 2 approved, 2 approved with reservations]
DOI: 10.12688/f1000research.7460.1
Editorial
On the evolving open peer review culture for chemical information science [version 1; referees: not peer reviewed]
DOI: 10.12688/f1000research.7968.1
Opinion Article
Towards an open science publishing platform [version 1; referees: 2 approved]
Articles that may interest you
Browse by related topics
Alongside their report, referees assign a status to the article:
Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations - key revisions are required to address specific details and make the paper fully scientifically sound
Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions
Sign In
If you've forgotten your password, please enter your email address below and we'll send you instructions on how to reset your password.

The email address should be the one you originally registered with F1000.

Email address not valid, please try again

You registered with F1000 via Google, so we cannot reset your password.

To sign in, please click here.

If you still need help with your Google account password, please click here.

You registered with F1000 via Facebook, so we cannot reset your password.

To sign in, please click here.

If you still need help with your Facebook account password, please click here.

Code not correct, please try again
Email info@f1000.com for further assistance.
Server error, please try again.