Understanding the contribution of UK public health research to clinical guidelines: a bibliometric analysis

Analysis of references cited on NICE guidelines

We identified 62 currently active NICE public health guidelines from the NICE website (data extracted on 26/09/2016 ). The guidelines included are listed in Table 1. These guidelines were grouped into ten field categories, as indicated in Table 1, by two members of the team (GC, SG) based primarily on their titles, though a brief review of content was also conducted where necessary (e.g. if the title was ambiguous).

The references were extracted from the guidelines manually into a database, producing a total of 31,409 references across all guidelines. It should be noted that many of these guidelines do not include all or, in some cases, any of the references for the evidence that underpinned their development directly. Often, separate evidence summaries are provided (also on the NICE website) for the guidelines – sometimes several for one guideline – which provide references for the underpinning evidence. We extracted the references from these evidence summaries as well as directly from the guidelines themselves.

The references were matched in the Web of Science (WoS) database. Once duplicates were removed and documents which were not journal articles excluded, we were able to produce a set of 12,706 papers from the guidelines matched in Web of Science. This is a low proportion of the total references extracted, but reflects a high level of policy documents and other non-journal material referenced in the guidelines.

For each of these papers matched in WoS, we were able to analyse the institutional affiliations of the contributing authors, based on the address information provided. We used this to look at the countries in which the research was conducted. We used a full counting approach, to allow for comparisons with previous studies in the area.

We also have the date of publication of each of the articles matched in WoS. By comparing this to the publication date of the relevant guideline, we are able to produce an estimate of the ‘knowledge cycle time’ – that is, the time between the publication of a paper and its citation on the guideline.

Finally, for a number of the publications, we were also able to extract the names of the funders supporting the research from the funding body acknowledgements (FBAs) on the publications. This is only available in WoS for publications from 2008, and even since 2008 the data is not complete, partly because authors do not always cite their funding sources completely and accurately (and sometimes not at all). These data were available for 15% of the papers matched in WoS.

Using these data sets we were able to analyse the contributions and levels of collaboration across funding bodies, countries and between institutions on the publications cited on public health guidelines, by area of public health and as a whole. Analysis was carried out using Microsoft Excel and R.

Interviews

Interviews were conducted with eleven stakeholders engaged in public health-related decision-making and public health research in the UK. Interviewees were selected to ensure a mix of geographical settings and experiences. At the time of the interview, seven of the interview participants were directors of public health at the local authority level, tasked with responsibility for delivering public health interventions in the UK since 2013⁴. Four interview participants were either senior academics or researchers with significant experience of public health in the UK. In accordance with the terms of interview participation, any quotes or attributions have been anonymised where applicable.

Interviews were typically 30–60 minutes long and were conducted by telephone in a semi-structured format, based on the interview protocol provided in Table 2. An anonymised list of interviewees, along with their designation and expertise, is included in Table 3. Interviews were recorded, subject to the consent of the individuals, and summary notes written up after each interview (not full transcripts). Interviews were conducted under the principle of informed consent with interviewees informed that the recordings and notes would only be used by the study team for the purposes of the study. These files are stored on secure servers and recordings will be destroyed one year after study completion. The interview notes were analysed, and key themes and common messages extracted to generate some of the insights and the underlying context described, taking a thematic analysis approach. The emergent themes from the interviews are covered in more detail in the Discussion section of this paper.

Contribution of the UK as a provider of research

The first level of analysis was to assess the distribution and co-occurrence of institutional affiliations from publications cited in the guidelines. This allows us to look at the contribution of the UK and other countries to the research conducted that underpins this set of guidelines.

We found that 28% (2,627/9,391) of the papers matched include at least one UK affiliation. Of the 2,627 papers with UK institutions, 2,114 papers (80%) contained only UK based affiliations, suggesting the proportion of international collaboration to be around 20% with non-UK based institutions. This is below the general average for international collaboration in research across the UK^5,6.

We can also look at the contribution from other countries. Across the 9,391 papers, we identified 5,617 institutions across 93 countries. Figure 1 shows the distribution of institutions affiliated with the papers.

Figure 1. Number of papers by country institutional affiliation associated with publications in NICE public health guidelines.

However, though many countries make some contribution, the majority of the research comes from a more limited set of countries. As shown in Figure 1, there are four countries (UK, US, Canada and Australia) who have contributed to over 500 papers, and collectively, these four countries have contributed to 90% of the papers. The largest contributor is the US - 47% of papers have a US institution affiliation (4,419). This is likely to reflect the scale of public health research conducted in the US, relative to other countries^7,8.

Dividing the papers by subject area, we see that the contribution of UK research to the evidence used in guidelines is broadly similar for each subject area. Figure 2 shows the total number of papers for UK, US, Canada and Australia by public health subject area. The US accounts for the majority of papers in each subject area, with the exception of diabetes and oral health, where the UK accounts for the majority of papers (40% and 29% respectively). Papers from non-Anglophone countries account for less than 4% of papers in each subject area with the exception of: oral health, where 9% of papers are from Sweden; mental health, where 6% of papers are from the Netherlands; cardiovascular, where 4% of papers are from Finland.

Figure 2. Number of papers by NICE public health guideline subject area, across four Anglophone countries.

We can also look at the contribution of specific institutions, which provides some scope to identify ‘centres of excellence’ in public health research. In the UK, 1,181 institutions were identified, featuring on 2,627 papers. Table 4 shows the top ten institutions from the UK in the dataset, also mapped in Figure 3.

Figure 3. Map of top 10 UK institutions in the dataset.

Figure 4 outlines the network of institutional affiliations on papers with a UK address. In the network, each node represents an institution and the connecting lines indicate these institutions’ co-acknowledgement on papers (a thicker line indicates a greater number of co-acknowledgements). The size of a node is proportional to the number of papers with a UK corresponding address affiliated to the institution. Institutions with fewer than 4 papers with an associated UK address are not included in the network. Nodes are coloured by the location of the institutions.

Figure 4. Co-acknowledgement network – institutions affiliated with papers with a UK address (n=2,627).

Each node represents an institution and the connecting lines indicate these institutions’ co-acknowledgement on papers (a thicker line indicates a greater number of co-acknowledgements). The size of a node is proportional to the number of papers with a UK corresponding address affiliated to the institution. Institutions with fewer than 4 papers with an associated UK address are not included in the network. Nodes are coloured by the location of the institutions.

Aside from the top UK institutions in the network (see Table 4), institutions from other Anglophone countries, such as Australia, US, New Zealand and Canada, tend to be the most prominent non-UK institutions in the network.

Overall, the UK is an important contributor of research to the evidence base supporting NICE public health guidelines, contributing to 28% of papers, alongside other Anglophone countries. This may partly reflect biases in the WoS towards English language publications but may also reflect the strength of these countries in the field. Clarke et al.’s bibliometric overview of public health research in Europe⁷ found that only 3.5% of research papers were published in non-English languages. This does not differ significantly by field of research within public health.

Contribution of the UK as a funder of public health research

The second level of analysis was to assess the distribution and co-occurrence of recorded funding body acknowledgements (FBAs) from publications cited in the guidelines. This bottom-up approach has been used in various studies to map the funding landscape of a particular field, explore interactions among research funders and assess trends in research funding^9–12.

1,420 papers contain FBAs with an average of 2.98 acknowledgements per paper. This represents 15% of the dataset, which is lower than other studies¹² but may reflect the fact that 67% of the papers in the dataset were published before 2008 – the year WoS began collecting data on FBAs.

Across these papers, 165 unique funders were identified with more than three acknowledgements, based in 20 countries. Table 5 shows the total number of papers and FBAs by funder location.

While there are significantly more US-based institutions affiliated with papers in the dataset, there are more papers with UK-based funders acknowledged than the US. NIHR is the most acknowledged funder in the dataset, accounting for 11% of papers with FBAs. Taken together with the UK Department of Health, it accounts for 226 papers (16% of papers with FBAs). Table 5 shows the top ten most acknowledged funders in the dataset, which account for 60% of the papers with FBAs.

Along with the UK Department of Health, research councils (such as MRC and ESRC) and disease-specific charities working in public health (such as British Heart Foundation and Cancer Research UK) are the most prominent UK funders in the dataset. In addition to UK funders, we see two US funders and one Australian funder in the top ten.

Government funders make up the majority of the 165 funders in our data set, accounting for eight of the top ten funders and around 72% of all funding acknowledgements (Figure 5). While our previous study on the FBAs associated with global mental health research¹² found a similar distribution in the overall number of acknowledgements (with government funding accounting for 68%), the study found a significantly larger proportion of charities, foundations and non-profits in relation to the total number of funders.

Figure 5. Proportion of funding body acknowledgements and funders by sector (n=1,420).

Figure 6 outlines the network of funders acknowledged in papers across the dataset. In the network, each node represents a funder and the connecting lines indicate these funders’ co-acknowledgement on papers (a thicker line indicates a greater number of co-acknowledgements). The size of a node is proportional to the number of papers in the network. Funders with fewer than ten papers are not included in the network. Nodes are coloured by the location of the funder.

Figure 6. Co-acknowledgement network of funders acknowledged on all papers cited in NICE guidelines (n=1,420).

Each node represents a funder and the connecting lines indicate these funders’ co-acknowledgement on papers (a thicker line indicates a greater number of co-acknowledgements). The size of a node is proportional to the number of papers in the network. Funders with fewer than 10 ten papers are not included in the network. Nodes are coloured by the location of the funder.

The most prominent funders in the network are the most frequently-acknowledged US and UK funders (see Table 5). There is also a relatively large contingent from Australia, multilateral funders such as the EU and the WHO, as well as a number of pharmaceutical companies, such as Pfizer, Merck, GSK and Sanofi. Government funders in Canada, the Netherlands and Scandinavia are also visible.

Of the 165 unique funders identified with more than three acknowledgements, 43 funders (26%) are based in the UK. We have split UK government funding, which make up 42% of the UK funders, into five funder groups for the purposes of analysis.

Figure 7 outlines the network of funders acknowledged on papers with an associated UK address. In the network, each node represents a funder and the connecting lines indicate these funders’ co-acknowledgement on papers (a thicker line indicates a greater number of co-acknowledgements). The size of a node is proportional to the number of papers in the network. Funders with fewer than 3 papers with an associated UK address are not included in the network. Nodes are coloured by the location of the funder.

Figure 7. Co-acknowledgement network of funders acknowledged on UK papers cited in NICE guidelines (n=1,420).

Each node represents a funder and the connecting lines indicate these funders’ co-acknowledgement on papers (a thicker line indicates a greater number of co-acknowledgements). The size of a node is proportional to the number of papers in the network. Funders with fewer than 3 papers with an associated UK address are not included in the network. Nodes are coloured by the location of the funder.

Breaking this down by subject area, we see that the contribution of research funders varies according to the subject area of the guideline. Figure 8 highlights the number of papers cited on the four main subject areas of NICE public health guidelines across the top ten research funders identified. For guidelines on addiction and substance abuse (of which the majority are primarily focussed on smoking), the US National Institutes of Health is acknowledged the most, followed by Cancer Research UK and NIHR. The US National Institutes of Health is also the most frequently acknowledged funder on mental health and wellbeing guidelines. However, for guidelines focussed on obesity and nutrition, the Australian National Health and Medical Research Council (NHMRC) is the most frequently acknowledged funder and for guidelines on physical activity, NIHR is the most acknowledged funder, with the top four funders all based in the UK. However, it is important to reiterate in interpreting this data that only 15% of papers in the dataset contain FBAs.

Figure 8. Breakdown of papers by NICE public health guideline subject area funded by top ten funders.

NCI, National Cancer Institute (US); CIHR, Canadian Institutes of Health Research; CRUK, Cancer Research UK; ESRC, Economic and Social Research Council (UK); BHF, British Heart Foundation; NHMRC, National Health and Medical Research Council (Australia); DH, Department of Health (England); MRC, Medical Research Council (UK); NIH, National Institutes of Health (US); NIHR – National Institute for Health Research (England).

Limitations of the study

As noted above, a key limitation of this analysis is the extent to which we can link citation on NICE guidelines to changes in practice. In some areas of health research there are reasonable grounds to assume that practice should, at least to some extent, reflect, or be moving towards, best practice, as set out in guidelines. However, this is less clear in public health, where implementation can take multiple routes and is very much local-context dependent. In addition, we can also add the usual caveats around the use of bibliometrics, some of which are clearly emphasised here. Firstly, only a relatively small proportion of the publications from the guidelines were matched in Web of Science, which may be because much of what is cited on guidelines are not journal publications. This means the analysis here is only a reflection of that portion of the evidence used to underpin guidelines. Secondly, there are a number of known biases in the coverage of bibliometric databases – notably in relation to language, where English language publications are more covered than other languages, which may bias towards publications from the UK and other Anglophone countries. Finally, it should be noted that a full counting approach is used in the analysis here, meaning we look at the number of papers to which researchers or funders from a particular country has contributed. For many of these, there will also be contributions from other countries, and this analysis cannot account for the extent of the contribution of the different authors and/or funders, which may differ substantially.

In terms of the interviews conducted, we are of course limited to the range of viewpoints provided by the set of individuals to whom we spoke. There may be perspectives which we did not capture, and there may be biases inherent of the sample of individuals willing to speak about their experience and knowledge of evidence supporting practice in public health. However, we conducted sufficient interviews that we reached ‘saturation point’, that is, additional interviews did not seem to be adding substantially to the range and nature of information captured. We also compared the interview findings to wider published evidence on the topic.

Underlying data

Interviews were conducted under the principle of informed consent. Interview participants were informed that recordings, transcripts and notes from the interviews were solely for the use of the project team and therefore these cannot be made publicly available, even in an anonymised form. If researchers wish to access these notes for the purposes of further research, they may contact the corresponding study author, Dr Susan Guthrie, by email (sguthrie@rand.org), providing details of the information required and the intended use of the data. Subject to approval by the RAND Human Subjects Protection Committee, we will then contact the interview participant(s) in question to seek their permission to share the interview notes for the purpose specified.

Figshare: Papers from NICE public health guidelines matched in Web of Science.csv. https://doi.org/10.6084/m9.figshare.8053187.v1¹³

This project contains the following underlying data:

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).