Poor statistical reporting, inadequate data presentation and spin persist despite Journal awareness and updated Information for Authors

Eligibility criteria

Original research papers published in the Journal of Neurophysiology in the years 2019 and 2020 were eligible for inclusion. Papers were excluded if they were editorials, reviews, errata, comments or rebuttals. Full-text PDF files of eligible papers were downloaded through the library of the University of New South Wales in January 2022.

Audit items

Items from the original audit

We assessed whether one or more of the following measures were used to summarise the variability of data or the size of an effect: standard error of the mean (SEM), standard deviation (SD), 95% confidence interval (95% CI) or interquartile range (IQR). We also assessed whether any summary statistic was not defined; this typically happens when the ± symbol or errors bars on plots are used but are not defined. The Guidelines for reporting statistics in journals published by the American Physiological Society,¹ a document that is referenced in the Journal of Neurophysiology Information for Authors, states that the SEM should not be used to summarise the variability of data or the size of an effect.

Additionally, we assessed whether papers reported exact p-values consistently, inconsistently, or not at all. The use of exact p-values is recommended by the Guidelines for reporting statistics in journals published by the American Physiological Society.¹

Finally, we assessed whether authors interpreted p-values close to, but above the selected threshold for statistical significance, typically p=0.05, as statistically significant or statistical trends (i.e. spin). This practice is misleading: more often than not, if additional data are collected, these p-values increase.³ To be able to compare our results to those of a more recent audit,³⁴ we also determined the prevalence of spin across all papers, regardless of whether or not such a p-value was present.

Items required or strongly encouraged in updated Information for Authors

Although several new reporting practices were introduced in the 2016 and 2018 updates to the Information for Authors for American Physiological Society journals (underlying data: Extract from Information for Authors), we selected four items that were broadly relevant, easily assessed, and, in two cases, related to items from previous audits.

We audited two required reporting practices introduced in August 2016. First, we assessed whether the number of samples or animals that contributed to plotted results were specified in figures or figure legends. Second, if statistical results were reported in a figure or figure legend, we assessed whether the statistical test used was specified in the figure legend.

We also audited two strongly encouraged reporting practices introduced in August 2018. First, we assessed whether data underlying the results were made available upon request or in a public repository. Second, we assessed whether graphical data “showed the range of data points in relation to the mean value and significance (i.e. dot-whisker plots) rather than formats that may mask variability (i.e. bar graphs)” and was “formatted to best convey the variability in the results” (underlying data: Extract from Information for Authors). It is unfortunate that the Information for Authors specifically reference’dot-whisker plots’ given that a mean (dot) and a two-sided error bar (whisker) is not different from a bar graph. That is, a dot-whisker plot can mask variability just as much as a bar graph. Nevertheless, because this reporting practice was introduced to encourage authors to convey the variability of plotted data and results, we scored this item as’yes’ when figures met this requirement: if the individual data points used to generate the summary statistics were plotted (regardless of the type of summary statistic used), or if box-and-whisker (median, IQR, range) or violin plots were encouraged.

Items from previous audits

To broaden the scope of the present audit and to allow for direct comparison with our previous results, we selected four additional items from two previous audits.^32,34

We assessed whether a sample size calculation was performed prior to the start of the study. In confirmatory studies, it is important to determine a priori the sample size required to achieve sufficient statistical power or generate precise estimates of the investigated effects.^4,35,36

We also assessed whether the study protocol was registered prior to the start of the study. Study registration is one of the simplest ways to improve the rigour of a study; it combats the selective reporting of outcomes, the cherry-picking of results, and p-hacking.^37–41

Next, we assessed whether plots included individual data points. As indicated in the Journal of Neurophysiology Information for Authors, “the same bar graph can represent many different datasets” (underlying data: Extract from Information for Authors). Similarly, in their series of educational editorials published in the Journal of Physiology and the British Journal of Pharmacology, Drummond & Vowler³⁰ encourage researchers to “use methods of data presentation that allow inspection, not concealment, of the nature of the distribution of the data.” While closely related to the audit item on variability of plotted data and results, the present item was included to allow for a direct comparison with previous audit results.

Finally, we assessed if a probability threshold was defined when frequentist statistics were used. This recommendation comes from the Guidelines for reporting statistics in journals published by the American Physiological Society¹: “Define and justify a critical significance level appropriate to the goals of your study”.

Scoring manual and pilot testing

A scoring manual was created to increase the consistency across investigators (extended data: Scoring manual). Next, five eligible papers were selected randomly and audited by all investigators. A consensus meeting was held and agreement across investigators was assessed. Wording of audit items and the scoring manual was refined to improve the consistency and accuracy of audit results.

Data extraction and analysis

Each investigator was randomly allocated $\sim$80 papers to audit, which they assessed independently.

If investigators were unsure how to score an audit item for a given paper, it was discussed amongst the team and agreement was reached by consensus.

Data were summarised with counts and percentages of papers that fulfilled the scoring criteria for each audit item. As it was not directly relevant to the audit, risk of bias was not assessed. Data processing and analysis were performed in Python (v3.9). Raw data and summary results are provided (underlying data: Audit Results). Although this study is not a systematic review, PRISMA guidelines were adhered where possible⁴² (extended Data: PRISMA Checklists).

Statistical analyses

For audit items from the original audit of papers published in 2015 in the Journal of Neurophysiology,³³ we compared the proportions of responses for results from the current audit to those of the previous audit using the proportions_ztest function from the Python statsmodels package,⁴³ with the level of significance set at $\alpha$=0.05.

Interpreting audit results

Compared to the original audit, the majority of poor reporting practices were unchanged or worsened. For example, $\sim$60% of audited papers reported SEM while $\sim$40% of papers failed to specify a threshold value for their statistical tests. Also, p-values between 0.05 to 0.1 were misinterpreted as statistical trends more than 60% of the time, a result that confirms researcher bias and highlights that spin continues to be a major problem in science.²⁰ Worryingly, the number of papers that included undefined summary statistics increased from 12.5% of papers in 2015 to 22.6% of papers in 2019-2020. Such high rates of spin and poor scientific reporting are worrisome. Overall, results of the present audit show that, at least in this journal, highlighting poor reporting practices does nothing to remedy the problem.

In 2016 and again in 2018, the American Physiological Society introduced several required or strongly encouraged reporting practices. Our results indicate that easy-to-implement required reporting practices were consistently adhered to in only a third of audited papers. Similarly, strongly encouraged reporting practices were consistently adhered to in only 12.0 to 25.0% of audited papers. While the introduction of new reporting requirements to improve the quality, transparency and reproducibility of published papers is well intentioned, our results indicate that they were not universally adopted and enforced at the Journal of Neurophysiology.

Why does poor research reporting persist?

The above results raise an important question: Who is ultimately responsible for enforcing the reporting requirements outlined in a journal’s Information for Authors? The obvious answer is the journal itself, its staff and its editors. However, as highlighted by our own work and that of others,^12–23 staff and editors across several journals do not take on this responsibility. While it is possible that editors do not agree with reporting practices required by their journal and thus do not choose to enforce them, a more plausible explanation is that editors, like the rest of us, are overworked.⁴⁴ They do not have the time to ensure papers they handle comply with every aspect of their Information for Authors. In line with this possibility, personal communication with an Associate Editor at the Journal of Neurophysiology confirms that editors are not required to read in their entirety the papers they handle. A similar comment was shared by an Associate Editor of the European Journal of Applied Physiology, highlighting that this is not an isolated practice. In addition, some editors lack knowledge about reporting guidelines while others fear that authors will not submit to a journal renowned for its strict reporting guidelines.⁴⁵ In the same vein, journal staff likely do not have the time (or expertise) to review every paper with a fine tooth comb to ensure they comply with reporting requirements. Some journals, including the Journal of Neurophysiology, use SciScore (https://sciscore.com/) to assess and score submitted manuscripts in an attempt to enhance the rigor and reproducibility of published papers. Unfortunately, these tools are far from perfect and their output requires expert review. Moreover, the items assessed by SciScore at the Journal of Neurophysiology do not address any of the items in the present audit. Thus, despite being the obvious choice to enforce their own reporting requirements, many journals are not currently enforcing compliance with their Information for Authors.

Who else might be responsible for this enforcement? Reviewers? Reviewers are expected to focus on the science, not compliance with reporting requirements. Having said this, several journals require reviewers to complete a series of tick-box questions to indicate whether or not the paper they are reviewing complies with key reporting practices. At the Journal of Neurophysiology, reviewers are required to answer the following questions:’Should any bar graphs be presented as box-whisker plots or dot-whisker plots, instead of bar graphs, to enhance clarity?’,’Do bar graphs show individual data values?’, and, with regards to figure legends,’Do they include details for each individual experiment (sample number (n), sex, statistics test)?’. Based on the results of the present audit, this approach does not seem effective.

What about authors? Authors are expected to read a journal’s Information for Authors when preparing their manuscript. However, Information for Authors can be lengthy, convoluted and difficult to navigate, with each journal having their own bespoke version. Authors may not follow the Information for Authors to save time. Since the Journal of Neurophysiology has a rejection rate of $\sim$55%,⁴⁶ authors may not feel it is worth their time to comply with all of the requirements and recommendations in the Information for Authors. This is especially true when authors can peruse the current issue of the Journal of Neurophysiology and see that reporting requirements are not enforced. Authors already have to deal with high rejection rates, pedantic submission processes and tedious formatting requirements (at submission or acceptance) that amount to free labour for journals and publishers.⁴⁷ To pass the responsibility of compliance with Information for Authors to the authors would be taking further advantage of the skewed power dynamic that exists between journals, publishers and authors. Having said this, a simple checklist related to compliance with reporting requirements could be required at submission or acceptance. However, someone would still need to verify and enforce results from these checklists. As mentioned above, a similar checklist is already in place at the Journal of Neurophysiology for reviewers, and it appears to be largely ineffective. Thus, it is unclear whether introducing a checklist for authors to complete would fare any better.

Possible solutions to improve research reporting

A possible solution would be for editors to be assigned a small team of junior editors. These junior editors would assess papers that go out for review for compliance with reporting practices outlined in the Information for Authors. Such an approach, if adhered to and properly enforced, could ensure required and recommended reporting practices become the norm rather than the exception. In fact, some American Physiological Society Journals have introduced an Early Career Editorial Fellowship Program.⁴⁸ This is a great initiative, and the scope of the program could be broadened to tackle the tasks mentioned above. Unfortunately, few Information for Authors address good reporting practices⁴⁹; an issue that would need to be addressed for this solution to work.

Another possible solution would be for journals to employ additional staff with the experience and expertise to review submitted manuscripts. It may be that software tools can be used in the first instance, with journal staff carefully reviewing papers only once they are likely to be accepted. The obvious problem with this solution is that some journals are run on small budgets, while other journals are run by large profit-driven corporations that are unlikely to support the creation of such positions.

In healthcare, published guidelines do not automatically translate into clinical practice.⁵⁰ This type of behavioural change must be supported by multiple strategies that target all aspects of a system.⁵¹ A similar approach may be required to change adherence to reporting guidelines in academic journals.

Limitations

We acknowledge there are limitations to conducting an audit of papers published in a single journal. How generalisable are our results? While the present study can be viewed as a case study, its results are, for the most part, in line with two previous large-scale audits: one that involved over 750 papers published in the Journal of Physiology and the British Journal of Pharmacology,³² the other that involved nearly 500 papers published by a medical research institute across dozens of journals and disciplines.³⁴ Our results are also in line with other recent reports of poor scientific reporting,^17–23,52 including an audit where 61% of published COVID-19 studies used incorrect statistical methods.⁵³ Although the present audit focused on a single journal, we did not set out to target the Journal of Neurophysiology per se. Our previous 2015 audit³³ and the introduction of new reporting requirements presented a unique opportunity to answer the research questions tackled here. Whether the present results are applicable to other journals remains to be determined.

To reduce the bias in data extraction to a minimum, it would preferable if future audits were performed in duplicate. Two investigators would audit each paper and any disagreements would be resolved in discussion with a third investigator.