Taxonomy of interventions at academic institutions to improve research quality

The search

Because the potential range of potential interventions and terms used was broad and unknown, we used a 2-stage process for the search. Stage 1 used a set of seed articles and reviews identified by the authors from a preliminary search which identified several articles including a review of journal interventions. We then used a forward and backward citations search of this set of articles to widen the pool of potential articles. Stage 2 then conducted a word frequency analysis on these eligible articles to identify key terms to build a search strategy for the full database searches.

This Stage 1 search identified a key review article on interventions to improve adherence to reporting guidelines for journals, which included a suggested taxonomy.⁹ We then drafted an initial taxonomy and used the other interventions identified from the searches to test and modify this proposed intervention taxonomy.

Participatory design stage

In this next stage, we sought input from others regarding the taxonomy and for further examples of interventions. To ensure the taxonomy was reflective of research practice in institutions, we invited possible end-users to assist in co-development using aspects of participatory design research.¹¹ During the 2021 Association for Interdisciplinary Meta-Research and Open Science Conference (https://www.ivvy.com.au/event/aimos2021), we held a workshop with approximately 40 participants to further refine the draft taxonomy. Detail on the audience involvement in the development of the taxonomy was outlined in the conference schedule, and attendees were informed that their contributions would add to the development of the taxonomy. Workshop participants included ~30 researchers at different career levels, ranging from PhD students to professors.

Briefly, the steps of the workshop process were:

Following the workshop, we used the participant input to develop the revised taxonomy, collect further potential examples, and revise the taxonomy again. In the following year, the taxonomy was presented at the 2022 Association for Interdisciplinary Meta-Research and Open Science Conference (https://aimos.community/aimos2022), where authors asked for any final feedback. No conference attendees responded to this. Ethics approval was not required as the process involved negligible risk, and the de-identified workshop participants were all sent a draft copy and invited to be authors on the manuscript. No workshop participants responded to the invitation.

The taxonomy

Interventions were first classified according to the research stage of their implementation: before study conduct, during study conduct, and after study conduct. Research stages were then further subclassified into education, grant writing, protocol writing, research conduct & analysis, manuscript writing, manuscript submission, or post-publication. Table 1 highlights which type of behaviour change interventions, as classified by Michie’s behaviour change wheel, are represented at each research stage.

‘Whole of institution’ was included as an additional category, separate to the research stages, as some interventions relate to two or more stages of research or support overall research practices in that institution. Similarly, ‘Institutional Culture and Individual Ethos’ was added to the taxonomy to highlight the influence of the culture of the institution including their overall research aims and mission, and those that work in the institution and their individual ethos, values, and attitudes towards research practices.

Table 1 gives examples of the interventions identified. The full set of interventions are displayed at the Open Science Framework website (“Taxonomy of interventions at academic institutions to improve research quality - Full Taxonomy Table and Example Interventions” Overall, we identified 93 different possible interventions).

The types of interventions varied widely, from whole of institution policies – such as modifying hiring and promotion criteria to emphasise rigorous research design, reproducibility, and transparency, to highly specific departmental-level interventions such as developing mentor-mentee relationships. Most interventions are applicable to researchers at all levels of experience. Several interventions are specific to particular areas of research e.g., registration of clinical trials in healthcare, but others, such as mentoring, or journal clubs are relevant to multiple disciplines. We did not subclassify by disciplines.

In reviewing the taxonomy, several themes emerged. Many of the interventions require a substantial and long-term investment in people– e.g., hiring of specific experts; training of research assistants and others on data collection methods and techniques; co-design with patients and public/end-users. Though ad hoc seminars have value, most of the interventions require individuals or teams or to be embedded in institutions, even if the intervention is to provide ‘just in time’ advice. For example, a publications officer to check adherence of papers to reporting guidelines might have to be well established for them to be able to provide on the spot advice at a time of need.

Education of researchers and research support staff can happen by a variety of formal and informal methods. There was some suggestion that some of the training had to be compulsory e.g., included in the curriculum for undergraduate and postgraduate research training. However, there was also specific recognition of the role of informal networks including peer-to-peer learning and mentor-mentee relationships. We note that mentor-mentee learnings can be in both directions, as more junior staff are sometimes the instigators of novel research practices learned during their research skill development. As training in undergraduate and postgraduate programs are constantly changing, more experienced researchers can be exposed to this by mentoring a student or Early Career Researcher.

There were surprisingly few technical interventions suggested. Most of these also included an element of human intervention e.g., use of pull-requests and code commentary by collaborators and/or external peers on shared codebases. Notably one of the technical interventions was to cease subsidising (through purchase of site licenses) the use of certain software programs (e.g., statistical and spreadsheet) that are not conducive to reproducibility, and to instead promote and encourage the use of open science source software and practices.

Surprisingly, the ‘incentives’ row has more empty cells than the other rows. This indicates either a lack of awareness of participants involved in reviewing the taxonomy, and/or a lack of incentives being implemented and available at research institutions to encourage researchers to participate in quality research practices.

Following the classification, we searched for papers that had described and/or assessed these interventions. During the search processes, eleven articles evaluating interventions were found. All the interventions that had been assessed were in the manuscript and grant writing or education phase of research (Table 2).

Limitations

The interventions in the taxonomy we present are not a comprehensive list of all possible interventions. We did not assess adherence to any of the interventions or examine their effectiveness, except where there were previously published papers. However, a recent review protocol has been published by a group of authors who aim to identify such interventions that investigate research reproducibility and replicability.¹⁸ As the participants and authors were largely or exclusively from the sciences, the list of interventions may not include interventions within Humanities and Social Sciences.

A strength of this research is the use of both published peer-reviewed literature and end-user engagement to develop the taxonomy.

Future directions

Given most interventions outlined in the taxonomy have not been evaluated for their impact on research quality and reproducibility, there is a clear need for more institutional interventions be evaluated. Priority areas for evaluation should be those currently in common use at institutions, to assess their value. Implementation of new or different interventions could be those that are no- or low-cost, such as open access tools and software to enhance research practices, e.g., Overleaf, and JASP, and adaptation of policies and the research environment to promote open science practices.

Institution culture and individual researcher ethos have a strong influence over the reproducibility, quality, and transferability of research practices. The UK Reproducibility Network encourages institutions to examine their research culture and how it may or may not be supportive of producing robust and credible research.¹⁹ The implementation and evaluation of interventions outlined in our taxonomy should be considered along with the institution’s current culture and potential shifts that could be made to encourage and promote open science practices.

Finally, it is vital to explore the paucity of ‘incentive’ interventions that were identified. Incentivisation is important in workload models of research institutions, much like universities have incentives for their education and teaching of degrees and coursework, they need incentives for research quality. The kind of incentives depend heavily on the institutional structures and availability of resources to create or fund incentives. It is recommended that future research could be guided by this taxonomy further to identify how incentivisation of quality research practices could be better implemented.