Key practices in reporting guidelines and standards for research on spinal cord stimulation: a scoping review

Causal graph

Causal inference is a framework to identify and estimate causal effects from observational data, particularly when randomized trials are not feasible.^12,13 Subject-matter expertise is used to specify a plausible causal directed acyclic graph.¹⁴ The effects of potential strategies (i.e. exposures) on outcomes are then identified from the causal graph by procedures to minimize bias, before causal effects are estimated.^12,15

An initial causal graph was specified to show how potential strategies affect rigor, transparency and reproducibility (Figure 1). Potential strategies, other factors and outcomes are represented as nodes in the graph, and arrows between nodes indicate the direction of causation.

Figure 1. Initial causal graph of potential strategies to improve research rigor, transparency and reproducibility.

This scoping review was designed to synthesize information from reporting criteria on rigor, transparency and reproducibility, and update nodes and paths in the graph. The scoping review is reported using the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) reporting guideline extension for scoping reviews.¹⁶ The protocol and registration are available on the Open Science Framework (https://osf.io/a9jw2/). The protocol was developed using the Joanna Briggs Institute (JBI) methodology for scoping reviews.¹⁷

Eligibility criteria

Articles were included from inception to 12 September 2024 if they were (1) standards, guidelines, recommendations or statements (study types; ‘unit of study’ in JBI methodology) for research on humans, (2) on rigor, transparency and reproducibility, in the fields of stimulation or neuromodulation (JBI ‘context’) and (3) written in English. Systematic or narrative reviews were included if they met the above criteria. Articles on neuromodulation were included if they involved stimulation. Clinical practice guidelines, primary research, commentaries, opinions and editorials were excluded.

Search strategy

Four databases (Scopus, Embase, Medline, Web of Science) were searched from inception to 12 September 2024. Searches included the following keywords: (1) standard OR guideline OR recommend OR statement, (2) research OR study OR reporting, (3) open science OR reproducible OR transparent OR rigor OR integrity OR quality, (4) stimulation OR neuromodulation.

The complete search strings for each database are available (S1 File- refer to extended data) and were developed in consultation with an academic librarian.

Article selection

Records retrieved from the databases were uploaded onto the review software Covidence¹⁸ and duplicates removed. Titles and abstracts retrieved from the searches were screened by the lead reviewer. Full-text PDFs of included articles were retrieved and assessed for inclusion by two independent reviewers. Differences were resolved by discussion. All articles that fulfilled eligibility criteria were selected for inclusion in the scoping review.

Data extraction

Two reviewers (KN and JD; KN and MH) independently extracted data using a standardized Excel sheet. The following data were extracted from each article: stimulation type, clinical area, and text on responsible practices from the article, which was classified into one of five main types ( Table 1). A type labeled “Other” was included to capture potential types of responsible practices that did not fall into one of the main types. Examples of types of responsible practices are shown ( Table 1). Types of responsible practices, except for Other, are indicated as nodes in the initial causal graph (Figure 1).

Data synthesis

Descriptive data were summarized in tables. Text on responsible practices from the article was classified into one of five main types. Text on responsible practices that could not be classified under any listed type were classified under Other. Data were compiled in tables showing data entered by each reviewer side by side.

Consensus discussion was used to assess the causal graph and determine if existing nodes and paths should be retained, revised or deleted, or if new nodes and paths should be added. This process is broadly consistent with guidance on developing causal graphs for research using observational data.^19,20

Pairs of reviewers differed in classifying types of responsible practice in one of these three ways: (1) at least one reviewer classified the responsible practice as Other, (2) reviewers classified the responsible practice differently, (3) the responsible practice was classified by one reviewer but not by the other. Differences between reviewers, or considerations flagged by reviewers for further discussion, were discussed.

Differences between reviewers were resolved through consensus discussion and used to refine the causal graph. When there were no differences between reviewers (i.e., the responsible practice type and supporting text were the same for reviewers 1 and 2), no changes were made to the causal graph. When there were differences between reviewers (i.e. the responsible practice type or the supporting text differed between reviewer 1 and 2), these were discussed and a decision was made to retain, revise or delete existing nodes and paths. When responsible practices from the article were classified by either reviewer under Other or flagged for further consideration, these were discussed, and a decision was made to either revise an existing node or add a new node. No statistical tests were performed.

Article characteristics

In total, 9,970 records were retrieved. After de-duplication, 5,770 titles and abstracts were screened by the lead reviewer. Of these, 19 were identified for potential inclusion and the full text articles were assessed by two independent reviewers. Nine of these satisfied inclusion criteria (Figure 2). Two articles identified through weekly article alerts up till 29 November 2024 using the same search strategies also met the inclusion criteria, resulting in 11 included articles (Figure 2).

Figure 2. PRISMA flow diagram.

All articles were published within the last 13 years. Each article was one of the following types: formal reporting guideline, standard or checklist developed through a Delphi process (n = 6 articles), general recommendations developed without a systematic consensus process (n = 4) or a systematic review (n = 1; Table 2). The articles guided the reporting of research in spinal cord stimulation (n = 6), transcranial electric stimulation (n = 3) and magnetic stimulation (n = 2), in clinical areas of pain (n = 4), chronic health conditions (n = 2), psychiatric disorders (n = 2), spinal cord injury (n = 1), and neurophysiology in healthy individuals (n = 2).

Resolution of differences

Types of responsible practices and examples are shown ( Table 1). Agreement between reviewers meant relevant nodes and paths of the causal graph remained unchanged. Reviewers agreed on nodes for Reporting of experimental conditions, Protocol registration, and Open science practices.

Differences between reviewers were resolved through consensus discussion and used to refine the causal graph (S1 Table- refer to data statement). Reviewers disagreed on whether the node Reporting study design to minimize bias was worded in sufficient detail to capture reporting practices for rigor, transparency and reproducibility. For example, the article text “Eligibility criteria for participants” was classified by reviewer 1 under Reporting study design to minimize bias, but the same text was classified by reviewer 2 under Other with the comment that eligibility criteria and participant characteristics (e.g. age, sex, comorbidities) must be reported regardless of impact on bias as they provide context needed to interpret research findings. Likewise, the article text “Level of expertise and training of repetitive peripheral magnetic nerve stimulation operator” was classified by reviewer 1 under Reporting experimental conditions with the comment that a separate node to capture training was unnecessary as surgeons performing spinal cord stimulation generally have similar training, but it was classified by reviewer 2 under Other with the comment that a separate node on training or education could be considered. To resolve these differences, the node Reporting study design to minimize bias was revised to Reporting study, participant and investigator characteristics to capture these characteristics.

Reviewers also disagreed on whether the node Reporting stimulation parameters was worded in sufficient detail to capture reporting practices for rigor, transparency and reproducibility. For example, the article text “Electrode manufacturer, product number and size” was classified by reviewer 1 under Reporting stimulation parameters, but it was classified by reviewer 2 under Other with the comment that ‘parameters’ typically refer only to stimulation settings such as frequency, amplitude and pulse width. To resolve these differences, the node Reporting stimulation parameters was revised to Reporting stimulation configuration and parameters to capture additional characteristics such as stimulation hardware details, electrode placement, device material and device manufacturer.

The reporting of adverse events was specified in 5 out of the 11 included articles, but reviewers observed that this was not captured in the causal graph, either explicitly or implicitly. For example, the article text “Ongoing monitoring for treatment-emergent adverse effects” was classified by both reviewer 1 and reviewer 2 under Other, with both commenting that it should instead be classified under Reporting study design to minimize bias or that a separate node on adverse events could be considered. Likewise, the article text “Side effects (e.g. discomfort, pain)” was classified by reviewer 1 under Other with the comment that a separate node on adverse events could be considered, but it was classified by reviewer 2 under Reporting experimental conditions. To resolve these differences, a new node, Reporting adverse events, was added. The refined causal graph is shown (Figure 3, expanded definitions of nodes are provided in S2 Table- refer to data statement).

Figure 3. Refined causal directed acyclic graph of strategies to improve research rigor, transparency and reproducibility, showing the revised nodes.

Study limitations

The restriction of eligibility to articles in English, and title and abstract screening by a single investigator, may have excluded relevant guidelines published in other languages and influenced the consistency of study selection. Also, this scoping review could only assess guidelines and standards for the reporting of research on spinal cord stimulation, but provides limited information on enablers and barriers of these practices. Other aspects of the causal graph (e.g. the influence of conflicts of interest, or the pressure to publish) are beyond the scope of this study.