An assessment of the autism neuroimaging literature for the prospects of re-executability

Introduction

There is concern about the status of reproducibility in science in general and neuroimaging neuroscience in particular (Button et al., 2013; Gorgolewski & Poldrack, 2016). A particularly germane concern was expressed by Kapur and colleagues in lamenting: “a profusion of statistically significant, but minimally differentiating, biological findings; ‘approximate replications’ of these findings in a way that neither confirms nor refutes them” (Kapur et al., 2012). The replication of a specific finding (or reproducibility of a specific analysis), as reflected in a publication, has many details and nuances to it (Kennedy et al., 2019). Often, we are searching for the ‘generalizability’ of a finding: does the finding hold true when using ‘similar’ data and a ‘similar’ analysis. The similarity of data (or analysis) is a fuzzy concept. One could have a population with the same number of subjects with the same diagnosis, having the same mean age and same gender distribution as a target population; however, if the diagnosis in question is a ‘spectrum’-diagnosis (for example, autism, schizophrenia, depression, etc.), despite the ‘sameness’ of my sample in the aforementioned categories, the detailed nature of the characteristics of my sample in the features of the diagnosis itself can still be quite variable. At the level of a biological finding, we typically do not predicate the finding on an exact acquisition protocol, or a specific analysis protocol; rather, it is implicit in our finding that it should hold for other valid acquisitions and analyses of the reported types. There is increasing evidence that this implicit assumption of similarity, when it relates to the specific details of acquisition or analysis, does not necessarily hold (Glatard et al., 2015).

Some have argued that the starting point for the structured exploration of the generalizability of a specific finding (and thus a cornerstone to the quest for reproducibility) lies in the original finding itself being re-executable (Ghosh et al., 2017; Kennedy, 2019). Starting from the re-execution of a finding will allow for the systematic exploration of the generalizability of that finding, over changes in data and analysis. To date, when new studies find different findings from prior studies, it is too easy to simply argue that differences in the subject population or analysis workflow differences account for the discrepancy. In this paper we concentrate on assessing the technical prospects of re-executability of a publication. As introduced above, there are many other factors that will contribute to the actual generalization of the findings including subject population details, data acquisition details, the nature of the processing and statistics (even if they can be re-executed), the underlying biological effect size, if present, etc. (see Figure 1). Take for example, the subject population. Too often researchers communicate a finding based on a convenience sample without any statement indicating that the results might not generalize to a sample that more accurately reflects human diversity (e.g. DeJesus et al., 2019; Henrich et al., 2010; Hruschka et al., 2018; Rad et al., 2018). Comprehensive and standardized description of all these additional factors are critical as well, but are beyond the scope of this evaluation. Our groups and others are looking into reporting standards for these areas as well.

Figure 1. Essential elements of a publication.

Elements of a publication that comprise a starting point for a structured exploration of the generalizability of a specific finding. The outlined areas define the technical prospects of re-executability of a finding that are evaluated in this survey.

The potential impact of reproducibility issues become most obvious when trying to make sense of the accumulated literature on specific topic areas (Rane et al., 2015). For this reason, we have chosen a particular area, ‘autism’ as a way to focus the literature for this survey, so that the conclusions we reach can have potential specific implications for that topic area. We feel that the autism focus, however, will generate findings that will have similar implications to other psychiatric and developmental application areas.

In this paper, we: 1) develop a specification for what constitutes an assessment of the technical re-executability for a given publication in each of the domains of: data, software, execution environment, statistics and results; 2) codify this assessment in survey form; and 3) apply the survey to a subset of the autism neuroimaging literature published recently (~2018). From the results of this survey, we can begin to generalize the state of the re-executability of the recent autism neuroimaging literature, in order to identify trends and opportunities for the enhancement of the re-executability status in support of greater overall generalizability (and hence reproducibility) of the literature. The survey template could also be applied as part of the publication review process, in order to prospectively attempt to enhance these aspects of reproducibility.

Methods

Survey development

Following the concept of a ‘re-executable publication’ (Kennedy, 2019), in order to assess the prospects of re-execution of a given paper, we assess 1) the availability of the starting data, 2) the perceived completeness of the analysis description (both data processing and statistical assessment), and 3) the availability of the detailed complete results (in order to verify accuracy of re-execution). Regarding the ‘availability of the starting data’, we assess if the publication indicates how someone¹ (other than the authors themselves) could appropriately access the data. The ‘precision of the analysis description’ ultimately asks if a reader who is reasonably skilled in the necessary domains, could precisely carry out the prescribed analysis steps. Specifically, are the software versions, operating system and complete parameters somehow made available to the reader? The ‘detailed complete results’ assesses if the publication indicates how to obtain the complete results, in order to both verify that the re-execution generates the same result and to overcome the limitations of only a selected summary being presented, which impedes a more complete meta-analysis of the literature.

In each of the three assessment areas, the survey distinguished between the theoretical potential for reproduction (such as complete descriptions of data used, software and commands executed, and statistical tests applied) and the practical potential for reproduction (whether the data is in fact accessible, whether the software is still available and will run). While the survey did not require the raters to actually reproduce the various steps, they were asked to use their professional judgement and past experience to determine the potential reproducibility. In these ‘judgement’ questions we allow responses of ‘Yes’, ‘Approximately’, ‘I’m not sure’, and ‘No’ to allow some degree of confidence in these judgements. For ‘results availability’, we coded ‘Yes’ if all of the results were indicated as being available, ‘Partially’ if some of the results were indicated as being available, and ‘No’ if none of the results were indicated as available or no indication of the results availability was provided. Note that our assessments are not if the analysis or data accessibility is ‘optimum’, or even ‘correct’, but rather if the assessor could redo the approach as described.

Figure 2 provides an overview of the survey design.

Figure 2. High-level survey design.

OS, operating system.

The survey was constructed in Google Forms. The details of the logic and wording of the survey forms was piloted (10 articles, three raters) within our own group, and then released for public comment to the BrainHack Slack² channel in August, 2018. The final complete (serialized) text of the survey is provided in S1 (see Extended data; Hodge et al., 2020c).

Results

Survey results

A high-level summary of the survey results are represented in Figure 3. The complete set of question-by-question results are provided in S3 (see Underlying data; (Hodge et al., 2020c).

Figure 3. Survey results summary.

The 50 publications are summarized on the main factors of data availability, software specification, statistical specification and results availability.

Publication availability: 38 of the 50 (76%) publications appear to have ‘free full text’ available, according to the PubMed search. Of these, 33 are indexed in PubMed Central. Overall, 43 were freely available through either PubMed Central, Google Scholar or publisher or other websites.

Data availability: 17 of the 50 (34%) publications make reference to the availability of the data used in the publication. However, the publications that indicate availability are mostly reusing data from the large repositories, whereas the publications that do not indicate data availability are principally locally conducted studies. Thus, this indicates that a large fraction of the data being used in publications are not available to the community. 3 of these 17 indicate ‘available upon request’. For the data that is available, the following resources are indicated: ABIDE 1 (Di Martino et al., 2014), ABIDE 2 (Di Martino et al., 2017), FCP/INDI (Mennes et al., 2013), COINS (Scott et al., 2011), LORIS (Das et al., 2012), NITRC (Kennedy et al., 2016), Preprocessed Connectomes Project (Puccio et al., 2016), UKBiobank (Miller et al., 2016), Brain Genomics Superstruct Project (Holmes et al., 2015), ADHD-200 (HD-200 Consortium, 2012), and Human Connectome Project (Glasser et al., 2016).

Image analysis: Virtually all of the publications surveyed indicate the imaging analysis software used (45 of 50, 90%). Most publications indicate the use of multiple tools. However, specific tool versions are indicated only about half of the time. Thirty-five different publicly released tools (plus a number of in-house packages) are used in this collection of 50 papers. Not surprisingly, the following tools are used in over 10 publications each: SPM (Ashburner et al., 1998), FSL (Jenkinson et al., 2012), and FreeSurfer (Makris et al., 2003). The specific operating system used is rarely reported (1 of 50, 2%). Overall, our raters felt that in 80% of the publications a skilled image analyst could (or might be able to) repeat the analysis.

Statistical analysis: In approximately two thirds of the publications (66%), the statistical software is indicated, again with variable indication of version and no reporting of the operating system upon which the software was running. In summary, our raters felt that in 29 of the 50 papers (58%), a skilled statistical analyst could (or might be able to) repeat the analysis.

Results availability: Availability of the detailed results is fairly rare. All or partial results are available in seven of the 50 publications (14%).

Other observations: Two publications which were clinical trials indicated preregistration (with the EU Clinical Trials Register and ClinnicalTrials.gov). None of the non-clinical trials publications reviewed indicated pre-registration (Nosek et al., 2019).

Discussion

The recent past literature of autism neuroimaging presents a somewhat consistent picture with respect to the prospects of re-executability with regard to the characteristics we examined in this report. Concerns of this sort have been raised in numerous contexts. The Organization for Human Brain Mapping’s Committee on Best Practices in Data Analysis and Sharing (COBIDAS)⁴, for example, digs very deeply into the recommendations for reporting and sharing in the literature. The work here is complementary as it takes a high-level gestalt view of re-executability.

Data availability is low, as we would expect to see given the current state of affairs. Figure 3 indicates that there may be a trend towards better data availability (more “Yes” values in the data access column as PubMed ID increases, a good proxy for relative date of publication).

While 80% of the publications were deemed to have repeatable image analysis, the low rate of specifying software version and vanishing rate of specifying operating system is troublesome, since these details can make a difference in results (Ghosh et al., 2017; Glatard et al., 2015). Even if there are currently only limited software options in some analysis domains, which may implicitly implicate the operating system used, such limitations are not guaranteed to persist through time and should not be assumed for the reader.

A smaller fraction of papers indicates statistical software other than image analysis software, perhaps in the belief that the statistical techniques are more important than the software used to implement the technique.

In both cases there is a distinct difference between the theoretical and practical ability to reproduce both the image analysis and statistical analysis. Rater confidence in the ability to re-execute image analysis and statistical analysis are similar, regardless of the fraction of cases where the software is specified.

The complete results availability criterion was rarely met. Lack of results availability causes a number of problems. Primarily, it is harder to confirm replication (or the degree to which replication was or was not achieved) without the complete set of reported observations, not just the summary tables or figures. Resorting to visual interpretations of ‘similarity’ of published figures remains fraught with issues that can hamper true understanding of new results compared to prior results. Lack of detailed results sharing also compromises subsequent meta-analytic studies that would strive to integrate observations across multiple publications. Finally, lack of complete results exacerbates the publication bias (Jennings & Van Horn, 2012) through focus on the (relatively few) statistically significant observations while not reporting the large set of observations that are not significant. Examples of complete results availability include when the individual statistical maps for a fMRI analysis are available in a resource such as NeuroVault⁵, the individual segmentation results of a processing workflow are available at NITRC⁶ or Zenodo⁷, etc.

None of the reviewed publications indicated pre-registration (Nosek et al., 2019). This is not surprising as pre-registration is a fairly new phenomenon, and its uptake in the literature can be expected to take a while. However, as a ‘baseline’ observation, it is still important to note, so that changes in the prevalence of the pre-registration practice can be monitored.

Conclusions

In conclusion, we feel that the survey results presented here reflect a state of neuroimaging publication practices that leaves ample room for improvement. While reuse of existing data is good, the majority of new data being collected for use in publications is not made publicly available. While the listing of software used is good, important details for reproducibility, such as version, detailed parameters, operating system, etc. are not fully disclosed. Similarly, statistical assessment details are variably reported, making re-execution problematic and approximate. Finally, as very little of the complete results of a publication are disclosed, assessment of the similarity of future replication attempts is severely hampered. Given the overall state of uncertainty about how reproducible (and representative) specific neuroimaging findings are, it seems prudent to begin to tighten up the variables that we as authors do have in order to better support the effective accumulation of knowledge about conditions we study. Promoting best practices in ethical data sharing, complete analytic approach disclosure, and complete results reporting seem to be critical in integrating the complex set of observations we collectively have published about the brain and how it develops and ages. The implications of these observations are that authors should redouble their efforts to be comprehensive in their reporting, even after the publication, to make as accessible as possible the detailed methods and results that they are reporting on. Specifically, authors, reviewers and editors should insist on the complete declaration of: data source and availability status, all software and versions used for data analysis and statistical assessment, the operating system (and version) for data and statistical analysis, and the disposition of the analytic results. Such a ‘checklist’ would be a valuable asset for the community and will be the subject of future efforts. This future checklist should be developed in conjunction with journal specific guidelines, and other checklists (established in conjunction with the COBIDAS report (Nichols et al., 2017), statistical reporting (Dexter & Shafer (2017), Nature Neuroscience Reporting Checklist, etc.). In such a way, publishers, editors and reviewers can impart more influence in the manuscripts that they encounter, in an effort to increase the transparency and completeness of the published record that they are playing their role in creating. Together, we hope that we can move the field forward and generate a literature that is more amenable to supporting the understanding of how our collective observations fit together in supporting the understanding of the brain.

Data availability