An assessment of the autism neuroimaging literature for the prospects of re-executability

Introduction

There is concern about the status of reproducibility in science in general and neuroimaging neuroscience in particular (Button et al., 2013; Gorgolewski & Poldrack, 2016). A particularly germane concern was expressed by Kapur and colleagues in lamenting: “a profusion of statistically significant, but minimally differentiating, biological findings; ‘approximate replications’ of these findings in a way that neither confirms nor refutes them” (Kapur et al., 2012). The replication of a specific finding (or reproducibility of a specific analysis), as reflected in a publication, has many details and nuances to it (Kennedy et al., 2019). Often, we are searching for the ‘generalizability’ of a finding: does the finding hold true when using ‘similar’ data and a ‘similar’ analysis. The similarity of data (or analysis) is a fuzzy concept. One could have a population with the same number of subjects with the same diagnosis, having the same mean age and same gender distribution as a target population; however, if the diagnosis in question is a ‘spectrum’-diagnosis (for example, autism, schizophrenia, depression, etc.), despite the ‘sameness’ of my sample in the aforementioned categories, the detailed nature of the characteristics of my sample in the features of the diagnosis itself can still be quite variable. At the level of a biological finding, we typically do not predicate the finding on an exact acquisition protocol, or a specific analysis protocol; rather, it is implicit in our finding that it should hold for other valid acquisitions and analyses of the reported types. There is increasing evidence that this implicit assumption of similarity, when it relates to the specific details of acquisition or analysis, does not necessarily hold (Glatard et al., 2015).

Some have argued that the starting point for the structured exploration of the generalizability of a specific finding (and thus a cornerstone to the quest for reproducibility) lies in the original finding itself being re-executable (Ghosh et al., 2017; Kennedy, 2019). Starting from the re-execution of a finding will allow for the systematic exploration of the generalizability of that finding, over changes in data and analysis. To date, when new studies find different findings from prior studies, it is too easy to simply argue that differences in the subject population or analysis workflow differences account for the discrepancy.

The potential impact of reproducibility issues become most obvious when trying to make sense of the accumulated literature on specific topic areas (Rane et al., 2015). For this reason, we have chosen a particular area, ‘autism’ as a way to focus the literature for this survey, so that the conclusions we reach can have potential specific implications for that topic area. We feel that the autism focus, however, will generate findings that will have similar implications to other psychiatric and developmental application areas.

In this paper, we: 1) develop a specification for what constitutes an assessment of the re-executability for a given publication in each of the domains of: data, software, execution environment, statistics and results; 2) codify this assessment in survey form; and 3) apply the survey to a subset of the autism neuroimaging literature published recently (~2018). From the results of this survey, we can begin to generalize the state of the re-executability of the recent autism neuroimaging literature, in order to identify trends and opportunities for the enhancement of the re-executability status in support of greater overall generalizability (and hence reproducibility) of the literature. The survey template could also be applied as part of the publication review process, in order to prospectively attempt to enhance these aspects of reproducibility.

Methods

Survey development

Following the concept of a ‘re-executable publication’ (Kennedy, 2019), in order to assess the prospects of re-execution of a given paper, we assess 1) the availability of the starting data, 2) the precision of the analysis description (both data processing and statistical assessment), and 3) the availability of the detailed complete results (in order to verify accuracy of re-execution). Regarding the ‘availability of the starting data’, we assess if the publication indicates how someone¹ (other than the authors themselves) could appropriately access the data. The ‘precision of the analysis description’ ultimately asks if a reader who is reasonably skilled in the necessary domains, could precisely carry out the prescribed analysis steps. Specifically, are the software versions, operating system and complete parameters somehow made available to the reader? The ‘detailed complete results’ assesses if the publication indicates how to obtain the complete results, in order to both verify that the re-execution generates the same result and to overcome the limitations of only a selected summary being presented, which impedes a more complete meta-analysis of the literature.

In each of the three assessment areas, the survey distinguished between the theoretical potential for reproduction (such as complete descriptions of data used, software and commands executed, and statistical tests applied) and the practical potential for reproduction (whether the data is in fact accessible, whether the software is still available and will run). While the survey did not require the raters to actually reproduce the various steps, they were asked to use their professional judgement and past experience to determine the potential reproducibility. In these ‘judgement’ questions we allow responses of ‘Yes’, ‘Approximately’, ‘I’m not sure’, and ‘No’ to allow some degree of confidence in these judgements. For ‘results availability’, we coded ‘Yes’ if all of the results were indicated as being available, ‘Partially’ if some of the results were indicated as being available, and ‘No’ if none of the results were indicated as available or no indication of the results availability was provided.

Figure 1 provides an overview of the survey design.

Figure 1. High-level survey design.

OS, operating system.

The survey was constructed in Google Forms. The details of the logic and wording of the survey forms was piloted within our own group, and then released for public comment to the BrainHack Slack² channel in August, 2018. The final complete (serialized) text of the survey is provided in S1 (see Extended data; Hodge et al., 2020c).

Results

Survey application

A high-level summary of the survey results is represented in Figure 2. The complete set of question-by-question results are provided in S3 (see Underlying data; (Hodge et al., 2020c).

Figure 2. Survey results summary.

The 50 publications are summarized on the main factors of data availability, software specification, statistical specification and results availability.

Discussion

The recent past literature of autism neuroimaging presents a somewhat consistent picture with respect to the prospects of re-executability with regard to the characteristics we examined in this report.

Publication availability: 38 of the 50 (76%) publications appear to have ‘free full text’ available, according to the PubMed search. Of these, 33 are indexed in PubMed Central. Overall, 43 were freely available through either PubMed Central, Google Scholar or publisher or other websites.

Data availability: 16 of the 50 (32%) publications make reference to the availability of the data used in the publication. However, the publications that indicate availability are reusing data from the large repositories, whereas the publications that do not indicate data availability are principally locally conducted studies. Thus, this indicates that a large fraction of the data being used in publications are not available to the community. For the data that is available, the following resources are indicated: ABIDE 1 (Di Martino et al., 2014), ABIDE 2 (Di Martino et al., 2017), FCP/INDI (Mennes et al., 2013), COINS (Scott et al., 2011), LORIS (Das et al., 2012), NITRC (Kennedy et al., 2016), Preprocessed Connectomes Project (Puccio et al., 2016), UKBiobank (Miller et al., 2016), Brain Genomics Superstruct Project (Holmes et al., 2015), ADHD-200 (HD-200 Consortium et al., 2012), and Human Connectome Project (Glasser et al., 2016).

Image analysis: Virtually all of the publications surveyed indicate the imaging analysis software used (44 of 50, 88%). Most publications indicate the use of multiple tools. However, specific tool versions are indicated only about half of the time. While this may seem a minor point, software version can make a difference in results (Glatard et al., 2015; Ghosh et al., 2017). In this collection of 50 papers, 35 different publicly released tools (plus a number of in-house packages) are used. Not surprisingly, the following tools are used in over 10 publications each: SPM (Ashburner et al., 1998), FSL (Jenkinson et al., 2012), and FreeSurfer (Makris et al., 2003). The specific operating system used is rarely reported (1 of 50, 2%). Overall, our raters felt that in 80% of the publications a skilled image analyst could (or might be able to) repeat the analysis.

Statistical analysis: In approximately two thirds of the publications (66%), the statistical software is indicated, again with variable indication of version and no reporting of the operating system upon which the software was running. In summary, our raters felt that in 29 of the 50 papers (58%), a skilled statistical analyst could (or might be able to) repeat the analysis.

There is a distinct difference between the theoretical and practical ability to reproduce both the image analysis and statistical analysis. While the software packages used for the image analyses are specified in nearly all the cases, only some give the rater confidence that a skilled analyst could actually re-execute the analysis. Similarly, for the cases where the statistical tools are specified, only a handful are descriptive enough to give confidence in re-executability.

Results availability: Availability of the detailed results is fairly rare. All or partial results are available in seven of the 50 publications (14%). Lack of results availability causes a number of problems. Primarily, it is harder to confirm replication (or the degree to which replication was or was not achieved) without the complete set of reported observations, not just the summary tables or figures. Resorting to visual interpretations of ‘similarity’ of published figures remains fraught with issues that can hamper true understanding of new results compared to prior results. Lack of detailed results sharing also compromises subsequent meta-analytic studies that would strive to integrate observations across multiple publications. Finally, lack of complete results exacerbates the publication bias (Jennings & Van Horn, 2012) through focus on the (relatively few) statistically significant observations while not reporting the large set of observations that are not significant.

Other observations: None of the reviewed publications indicated pre-registration (Nosek et al., 2019). This is not surprising as pre-registration is a fairly new phenomenon, and its uptake in the literature can be expected to take a while. However, as a ‘baseline’ observation, it is still important to note, so that changes in the prevalence of the pre-registration practice can be monitored.

Conclusions

In conclusion, we feel that the survey results presented here reflect a state of neuroimaging publication practices that leaves ample room for improvement. While reuse of existing data is good, the majority of new data being collected for use in publications is not made publicly available. While the listing of software used is good, important details for reproducibility, such as version, detailed parameters, operating system, etc. are not fully disclosed. Similarly, statistical assessment details are variably reported, making re-execution problematic and approximate. Finally, as very little of the complete results of a publication are disclosed, assessment of the similarity of future replication attempts is severely hampered. Given the overall state of uncertainty about how reproducible (and representative) specific neuroimaging findings are, it seems prudent to begin to tighten up the variables that we as authors do have in order to better support the effective accumulation of knowledge about conditions we study. Promoting best practices in ethical data sharing, complete analytic approach disclosure, and complete results reporting seem to be critical in integrating the complex set of observations we collectively have published about the brain and how it develops and ages. The implications of these observations are that authors should redouble their efforts to be comprehensive in their reporting, even after the publication, to make as accessible as possible the detailed methods and results that they are reporting on. Specifically, authors, reviewers and editors should insist on the complete declaration of: data source and availability status, all software and versions used for data analysis and statistical assessment, the operating system (and version) for data and statistical analysis, and the disposition of the analytic results. Such a ‘checklist’ would be a valuable asset for the community and will be the subject of future efforts. This future checklist should be developed in conjunction with journal specific guidelines, and other checklists (established in conjunction with the COBIDAS report (Nichols et al., 2017), statistical reporting (Dexter & Shafer (2017), Nature Neuroscience Reporting Checklist, etc.). In such a way, publishers, editors and reviewers can impart more influence in the manuscripts that they encounter, in an effort to increase the transparency and completeness of the published record that they are playing their role in creating. Together, we hope that we can move the field forward and generate a literature that is more amenable to supporting the understanding of how our collective observations fit together in supporting the understanding of the brain.

Data availability