Characteristics, utilisation and influence of viewpoint articles from the Structured Operational Research and Training Initiative (SORT IT) – 2009-2020

Background

In 2009, the International Union Against Tuberculosis and Lung Disease (The Union) and Médecins Sans Frontières (MSF) began an operational research (OR) training course. In 2012, the Special Programme for Research and Training in Tropical Diseases (TDR), hosted at the World Health Organisation (WHO), adopted this model and formed a partnership-based programme known as the Structured Operational Research and Training Initiative (SORT IT). SORT IT seeks to strengthen health systems, improve programme performance, and enhance public health through OR. To this effect, SORT IT enrols participants from low and middle-income countries (LMIC) and teaches them the practical skills of conducting and publishing OR in peer reviewed scientific journals with the goal of influencing policy and/or practice¹. Details of the structure and organisation of the SORT IT programme have been described elsewhere¹.

In addition to capacity building, each SORT IT course brings together a rich diversity of people (faculty and participants) from different backgrounds, expertise, experience and countries. This richness often gives rise to vibrant discussions, new ideas and innovative reflections on health-related issues of global interest. Various aspects of OR capacity building to improve public health are also discussed and critically reflected upon. This dynamic - which characterises the modus-operandi of SORT IT – has resulted in the conception and publication of over 40 “viewpoint” articles (classified by journals as viewpoints, editorials, and perspective articles). These viewpoint papers are secondary outputs of SORT IT courses and present critical analysis of a health or policy issue of interest² and/or challenge the norms of practice (“business as usual”). Many of these papers include a ‘call for action’ to improve public health or advocate for effective approaches to OR capacity building.

Since the inception of SORT IT, we have closely monitored and evaluated the characteristics and outputs of the OR courses, together with the influence that original research papers have had on policy and practice^3–9. However, we have yet to carry out any formal assessments of viewpoint articles – in particular examining who are the voices behind the papers, to what extent the papers have been used by others and whether/how they might have influenced policy and or practice.

This study aimed to assess the characteristics, utilisation and influence of published viewpoint articles derived from all SORT IT courses conducted between August 2009 and March 2020. Specific objectives were to report on i) the characteristics of publications, including LMIC contributions to authorship, collaborative partnerships (all High Income Countries - HIC vs HIC-LMIC); ii) access type and use of published evidence (views, downloads and citations); and iii) how ‘call for action’ papers influenced the implementation of OR capacity building within the SORT IT partnership, and policy and/or practice at national and international levels.

Methods

Results

Discussion

This is the first time that viewpoint papers derived from SORT IT courses have been formally assessed. Just over a third were ‘calls for action’; the rest were descriptive in nature. LMIC authorship and HIC-LMIC collaboration on these papers was high, but LMIC-first authorship and female representation was low. Three fifths of the ‘call for action’ papers contributed to the evolution of OR capacity building within the SORT-IT partnership.

The study strengths were that i) all discussion articles produced by the SORT IT partnership over the last 10 years were included and ii) we used universally acceptable parameters to assess access to, and utilisation of the viewpoint articles. The main limitations of this study were that i) online data on views and downloads of articles were only available for 17% of the papers, ii) citation counts might have been underreported without access to subscription-based databases like Web of Science and Scopus. That said, Google Scholar has been shown to be one of the most comprehensive sources for citations across different subject areas¹¹, so we believe that the effect of this limitation is minimal; and iii) the influence on policy and practice was only assessed through self-assessment by the authors of the paper; there was no external objective validation of the results and as such the methodology is open to bias. This is an important limitation of this aspect of the perspective paper.

This study highlights several interesting findings. First, although more than two thirds of viewpoint papers had LMIC authorship, the proportion of lead and last authors from LMICs was relatively low. Since these papers deal with issues specific to LMICs, having first and last authors from LMICs with knowledge and experience of these issues should be encouraged. Although course participants (predominantly from LMICs) may have an abundance of knowledge and experience to share in discussions and debates, most are not trained in how to craft these ideas and reflections into a paper. Furthermore, these viewpoint papers are not a primary output expected of the SORT IT courses; they are a ‘secondary output’. Thus, the writing process is often led by experienced faculty members (who often enrich or generate ideas from discussions with course participants and formulate these into thoughts), many of whom are from HICs. Fortunately, these faculty members have extensive experience of working in LMICs and are aware of their contexts. However, there is clearly room to do better, to find ways to encourage more LMIC-lead authors – participants and facilitators. Going forward, efforts should be made to increase the number of LMIC lead-authors on viewpoint papers that are generated from SORT IT courses. Many of the individuals who have been trained in SORT IT courses go on to perform OR in the future^4,5,8, and those who have co-authored a viewpoint paper, could be encouraged to lead on a viewpoint paper back in their home country as they become more experienced in scientific writing.

Second, female representation on the papers was low. This may be because there were fewer female mentors and mentees as compared to males on the SORT-IT courses^3,6, particularly experienced faculty members. This is corroborated by findings from a previous evaluation of SORT IT courses showing that, of those participants enrolled on SORT IT courses between 2009 and 2014 who went on to become facilitators on subsequent courses, only 37% were female⁶. As the authors of this evaluation pointed out, socio-cultural factors may also explain this observation - anecdotal evidence suggests that female facilitators experience difficulties in leaving home and travelling to distant course locations. One way to boost female involvement in paper writing is through the SORT IT alumni network; the latter provides an opportunity to actively encourage female SORT IT alumni to become OR fellows and to become drivers of OR.

Third, most of the journals in which viewpoints were published were open access which we recognise as being particularly important if we want to optimise readership in LMICs and ultimately influence policy and practice. Zachariah et al.⁶, in their evaluation of 20 SORT IT courses conducted between 2009 and 2014, showed that article views/downloads for immediate open access articles were double those of closed access journals. We were unable to assess this correlation, as views and downloads of the viewpoint papers were only available for just under a fifth of the papers. Journals could clearly do a better job of presenting altmetric data of views, downloads, and citations of papers available on their websites. Such metrics are important and should be shared publicly as standard practice on journal websites. In our study, citation counts were used as proxy for utilisation. Establishing the relative value of a single citation count, however, is difficult in the absence of some sort of reference measure. A software tool that many online publishing platforms currently utilise (and which was available for two thirds of our papers) is Dimensions - a linked research data platform¹². Dimensions provides various metrics, one of which allows the citation performance of an article relative to similarly aged articles in its subject area to be compared. We cannot vouch for the validity of such a tool (this would need to be further explored), but metrics like this appear to be complimentary to citation counts and may provide a more nuanced overview of the utilisation of papers.

Fourth, about three quarters of our viewpoints had an influence within or outside SORT IT and this compares favourably with the 55–66% change in policy and practice reported from previous SORT IT evaluations^7,9. However, if we focus on changes in policy and practice outside the SORT IT partnership, this occurred in only a third of our ‘call for action’ papers - in other words, they had less influence in the wider realm than other (original research) SORT IT publications. This is probably because the papers in question are opinion pieces. One opinion paper may not be enough (policy and practice is influenced by all sorts of different inputs) and may not wield the same weight as an original research study when it comes to changing national or global thinking.

Finally, reflecting on the types of viewpoints that are being generated during SORT IT courses (one third being calls for action; two thirds being descriptive) should make us consider the added value of these types of papers. Our ‘call for action’ papers are meant to catalyse change and given the rich diversity and experience of people attending SORT IT courses this initiative should be championed. In addition to monitoring the OR outputs of our SORT IT courses, assessing viewpoint papers appears to be useful and encourages us to systematically monitor whether people we train in SORT IT go on to write viewpoint and perspective papers as they reflect a higher level of scientific thinking and critical reflection. Viewpoints contribute to challenging “more of the same” in health care and to “do things differently”.

In conclusion, viewpoint articles generated during SORT IT courses appear to complement original OR studies and are valuable contributors to the dissemination of OR practices in LMICs.

Data availability