The imperative to find the courage to redesign the biomedical research enterprise

Methodological rigor (MR) score

The first metric, the MR score, is the most critical and the easiest to implement. It would be based on established attributes of reproducible research, such as blinding, power calculation, randomization, use of controls, validation of results in an independent cohort, availability of raw data and analysis codes.^4,21,22 Though the quickest way forward is to assign an MR score to a paper based on a self-reported checklist by the author(s), this may generate bias and cannot be applied to already-published papers. Thus, at the BRN’s inception, reviewers would need to validate the self-reported MR score against the Methodology section of the paper. The latter can be accomplished through an open review system, associated e.g., with free/public pre-print service such as medRxiv. MedRxiv is run by a consortium of two academic institutions and the BMJ, and funded by a private foundation, the Chan Zuckerberg Initiative. MedRxiv has proven to be an invaluable aggregator of science associated with the COVID-19 pandemic. Reviewers’ curated MR scores would serve as the dataset for the development, optimization, and validation of an automated, word-recognition based algorithm, which assigns MR score by scanning the Methods section of scientific papers.

Scoring of all publications can produce an MR score for individual scientists, averaged across their publications. This would incentivize all co-authors to support methodological rigor for every experiment, effectively self-regulating high-quality experimental design. The MR score would also help non-experts to judge the technical rigor of the presented work.

Scientists may have two objections: one, the retrospective grading may not accurately reflect the quality of the original experimental design, because in the past, journals restricted word count for the Methods section. This is remediated by standardizing an individual paper’s MR score to the yearly averages, a solution used in standardized testing. Two, the authors may disagree with an assigned score. True mistakes, if validated in an open review, would improve the automated algorithm, possibly through public competition(s) similar to the Netflix prize.

The evolution of MR scores on a population level would objectively measure the success of the BRN strategy to combat irreproducibility. Improvements in a scientist’s standardized MR scores could represent professional growth. Additional training initiatives intended to improve methodological deficiencies and currently imposed on all scientists, could be more effectively targeted to the scientists with consistently low MR scores.

Reproducibility score (R) score

Unless we demand pre-registration of all experimental designs akin to pre-registration of clinical trials in www.clinicaltrials.gov, the MR score will depend on what scientists choose to disclose. Some fear that pre-registration of all experiments will increase administrative burden²³ and stifle innovation. Others believe that a lack of preregistration substitutes scientific predictions by less credible “postdictions”,²⁴ exacerbating irreproducibility. A solution that integrates both views is to objectively assess reproducibility of results, or in broad understanding, the “scientific truth”²⁵ of each study, post-publication.

Using language recognition, the BRN would assemble all related publications to a sub-network (Figure 2B), where the publication describing the crucial discovery that spurred a subsequent line of investigation would represent the main hub located at the center of the subnetwork. For example, the discovery of cytokine IL17 would sit at the hub of a subnetwork, and subsequent papers describing its role in autoimmune diseases would be one of the most successful “branches” of this IL17 subnetwork. Each new, related publication would be placed in the periphery of such a subnetwork, while minimizing the network “distance” between closely related research papers.

To illustrate how the BRN associates related articles and uses their MR score to assign a Reproducibility, or R score, consider three related articles that investigate the role of IL17F in predicting therapeutic response to the drug interferon beta (IFNb) for treating multiple sclerosis (MS). Study #1²⁶ was published in the high impact journal, Nature Medicine. By using only 26 MS patients and not employing methods to prevent bias, this study expands observations in mouse models to humans, concluding that high blood levels of IL17F predict poor response to IFNb. Two validation studies (Studies #2²⁷ and #3²⁸) that collectively (and blindly) analyzed samples from 357 randomly selected MS patients demonstrated unequivocally that serum levels of IL17F do not predict IFNb response in MS. In contrast to PubMed, which does not automatically associate related studies, the BRN algorithm places the validation studies in proximity of the Study #1 (i.e., on the same branch). Additionally, the BRN automatically integrates knowledge (Figure 2B) from related studies: because Studies #2 and #3 have high MR scores and validate each other, both receive high R scores. In contrast, Study #1 receives a low R score because it has a low MR score and was, in fact, not reproducible. To compute the R score, the algorithm compares the results of the two congruent studies “weighted” by their MR scores against the MR score of the conflicting study, to assign the probability that the specific result is true: the probability that serum IL17F levels predict therapeutic response to IFNb based on this integrated knowledge is very low – let’s say 0.1%; therefore, the Study#1 receives an R score of 0.001, whereas Studies#2 and #3 receive R scores of 0.999.

The goal is to employ technology to curate knowledge, while rewarding scientists for greater rigor. Because experimental rigor positively correlates with reproducibility,²¹ receiving consistently low R scores for consistently high MR scores will question accuracy of self-reported experimental designs. Such discrepancies would become exceedingly rare, as the mere understanding that objective curation will identify contradictions would naturally limit any temptation to cheat.

The most obvious objection against implementing the R score is its potential inaccuracy. Like every machine-learning algorithm, generated R scores will not be 100% accurate at its first iteration, but access to large, curated datasets (provided by author’s contestation and human reviewers’ adjudication of R scores), will allow data-driven optimizations. Ultimately, this will generate algorithm(s) that consistently outperform humans.²⁹

Societal impact (SI) score and its scientific, commercial, and public health contributions

Innovation and relevance of scientific discoveries to health are essential attributes of the NIH mission. Indeed, science can be well-designed, reproducible, but still irrelevant.¹¹ To assure that biomedical science is innovative and generates societal value, we must learn to quantify its worth objectively.

The impact of a discovery on science itself can be measured within the BRN as a growth of a new branch of scientific explorations. For example, basic scientific discoveries, such as the reprogramming of somatic cells to induced pluripotent stem cells (iPS;³⁰) or CRISPR-CAS9 genetic editing both established new branches of science.³¹ Thus, their scientific impact is enormous. To assess the public health and commercial impacts of scientific discoveries, the BRN must link to databases of patents, newly approved drugs, medical products, and new technologies. This will trace the origin of societal and public health advances back to individual scientists in order to develop the integrated SI scores. The goal of the SI score is to objectively measure the impact of individual discoveries and cumulative impact of individual scientists or even scientific institutions, which can be used to reward institutional leaders. Current attempts to assign value to a publication or a researcher, such as NIH’s Relative Citation Ratio are insufficient, because they rely on the number of accrued citations. In our concrete example, the irreproducible publication in the prestigious journal is easily found in internet searches and continues to accrue citations years after its findings were unequivocally refuted: Study #1²⁶ accrued (as of May 2021) 590 citations, while the two congruent validation studies, published in less prestigious journals, accrued only 114 citations in total. This example is hardly an exception: a major effort to reproduce important scientific discoveries found non-reproduced articles published in higher impact journals and were accruing more citations than reproducible articles.³

Undoubtedly, the SI score is the most difficult metric to derive. The challenge is to assign the accurate proportion of any given publication’s SI score to individual discoveries, as no discovery is made in isolation (exemplified in CRISPR technology, see the Broad Institute’s CRISPR Timeline). In other words, as a pie grows (i.e., measured by the growth of subsequent publications and their societal impact), the number of scientists credited for the share of a pie also expands proportionally to the calculated impact of their discoveries. The SI score will likely receive the greatest push-back, as scientists have limited influence over commercialization of their discoveries and comparing societal impact of discoveries in different scientific fields may seem unsurmountable. But should we shrink from the enormity of this task and accept that the opinion of a few “experts” who nominate and select scientists for prestigious awards, is the best we can ever do?

Open peer review

Second, a single-blinded review system is morally indefensible. It empowers reviewers to determine the fate of the publication without accepting public responsibility (or reward) for the review. This system institutionalizes bias by limiting scientific contributions from women and minorities, and has shown itself to be incapable of weeding out irreproducible science.³

Open peer review is often disparaged on the assumption that few scientists would choose to participate, fearing retaliation for a critical review. If this is true, the scientific culture is broken. Engaging in discourse on validity and interpretation of scientific discoveries is a precondition to scientific progress. Providing constructive criticism of presented work and identifying ways for its improvement, while respecting the dignity of the scientists who produced it, underlies scientific maturity. This is very different from what can occur now, the writing of anonymous reviews containing publicly indefensible arguments and refusing to take personal responsibility for their accuracy.

The validity of science is the responsibility of all scientists. The irreproducibility crisis reflects on all of us, whether we adhere to sound or poor experimental design. We are the ones who form and embrace scientific culture and the only ones who can reform it.

Reviewer impact (RI) score

If we want to achieve sustainable open peer review,³⁴ we must reward scientists for their willingness to provide critical, creative thinking. Let’s return to the example described in the legend of Figure 2 and consider three hypothetical scenarios: in scenario #1, Study #1 is submitted to the BRN and a reviewer alerts the authors to the methodological flaw through open review. The authors acknowledge their mistake and amend or retract the problematic experiment in their paper. Because there were no negative societal consequences, the authors’ MR score would go down slightly for submitting a flawed experiment, and the reviewer would accrue positive RI score for identifying the paper’s shortcoming and triggering a correction.

In scenario #2, the authors submit a flawed experiment but choose to ignore the raised criticism. With the critical review in the public domain, an economical, well-designed (i.e., of high MR score) independent validation study is performed and fails to reproduce original finding. Now, the authors accrue not only a low MR score but also a low R score and negative SI score, equivalent to the value of wasted resources.

Finally, in scenario #3, which in many ways reflects the status quo, all reviewers fear the consequences of posting a negative review of the experiment coming from an influential scientist. The reported findings appear to be so important that a pharma company decides to perform a new clinical trial based on the published data. After considerable expenditures, the trial unequivocally demonstrates the irreproducibility of the original publication. The authors would accumulate not only low MR and R scores, but also highly negative SI score due to accumulated costs of wasted resources. Everybody loses. Through transparent, responsible review we can correct mistakes while they are still small, help each other grow as scientists and maximize the value of our discoveries.

Scientists should freely publish their experiments, and any reader should be able to transparently (and succinctly) critique them, accepting full responsibility for the review. Reviewers with habitually frivolous criticism not supported by data or scientific understanding would themselves accrue low cumulative RI scores, prompting them to change this behavior. By contrast, critiques from scientists with high cumulative RI scores would impel the authors to carefully consider the reviewers’ arguments and perform additional experiments to ensure the validity of findings.

The BRN would automatically limit irreproducible science if career rewards depended on high MR, R, and SI metrics. If performing bad science were detrimental to one’s career, why would anybody conduct much less try to publish flawed experiments? And, if scientists read papers anyway (which they must do) and rating them were a simple task that promoted their careers, why would they not review?

Finally, an open review process would level the playing field of scientific education by allowing all trainees and even journalists to learn critical thinking from the greatest minds. Everybody wins.

Dynamic publications

We also wholeheartedly embrace the concept of dynamic publications.³⁵ Publications should be transparently modified (with the modification history recorded in the BRN) based on new experiments or evolved knowledge. Such modifications may enhance clarity, adjust conclusions, and add, improve, or retract experiments in response to open review. Furthermore, scientists who were not original authors should be able to attach results of subsequent experiments to an existing publication, rather than publishing independent papers. For example, an investigator publishes an intriguing finding that lacks an independent validation cohort. Another scientist has such a cohort and validates the published findings. Rather than submitting this as a new publication (and thus needlessly duplicating information already contained in the introduction and discussion of the original publication), this independent lab could simply submit the results of the validation experiment and its related methods to the BRN, and link it to the previous paper. If the results subsequently translate into societal value, all contributing scientists will share the benefit proportional to their contribution. This way, many scientists can efficiently strengthen or refute existing publications, solving the current problem of (not) publishing negative data³⁶ and replication studies, which are essential for assessing the generalizability of discoveries.^37,38

Sharing raw data and dynamic re-analyses and re-use of published cohorts

Related to the publication process is the issue of depositing full datasets into the public domain, not only to verify published results but also for the datasets to be reused to test new hypotheses. The societal benefit of sharing raw data is widely recognized^39,40 but practically left at the discretion of the scientists. For example, the NIH encourages data sharing, and many journals request that the authors include a statement on data availability, such as: “Anonymized data not published within this article will be made available by request from any qualified investigator.” Such requests are not always granted and the declarations in the publication cannot be enforced.⁴¹

The reason for the lack of data sharing, especially from well-designed clinical studies, is that these datasets remain valuable; they allow scientists to ask new questions and publish new papers—the same value society loses if the data are not shared. Because different researchers ask different questions, some essential analyses may never be performed. Acknowledging truth on both sides, the solution is to incentivize data-sharing by assigning SI score to the investigators who generated valuable datasets whenever their reuse led to new societal benefits. If one team published raw data after spending 10 years generating an extremely well-characterized longitudinal cohort, and a second team re-analyzes these data and gains valuable insights, then both teams should share the resulting SI score for such invention, proportional to their efforts and intellectual contributions. Everybody wins, especially patients.

Re-envisioning the role of publication industry experts in the research enterprise

Would the BRN destroy the current publication industry? Not necessarily. First, the BRN may be based on public-private partnership and employ current publication industry experts. Second, the BRN would provide new job opportunities, such as science writers analyzing aggregate knowledge to identify new trends and knowledge gaps, or to promote commercialization of discoveries by guiding investment funds. These would allow the current scientific publication workforce to directly advance science and health.