How an international research funder’s forum developed guiding principles to ensure value and reduce waste in research

Introduction

The impact of health-related research, and the difference research makes for patients and the public, is the main focus of research funding agencies. It is the reason health-related research maintains the support of our health, public health and social care services and society more generally. But it is also essential to consider and acknowledge how funders deliver research systems and whether there are opportunities to enhance the efficiency and effectiveness of funding organisations.

Research funding agencies’ (RFAs) practice improvements are often set in the context of current global debates, specifically focusing on the purpose, accountability, and quality of research. These include issues raised about research integrity (World Conference on Research Integrity, https://wcrif.org/); the “crisis” of reproducibility in research (Ioannidis, 2016); avoidable waste across the research enterprise (Chalmers and Glasziou, 2009, Lancet Series Research: increasing value, reducing waste (https://www.thelancet.com/series/research)); the impact, openness and transparency of specific health-related research activities (AllTrials, All Trials Campaign, http://www.alltrials.net/, Chalmers, Glasziou and Godlee, 2013); and the rise (and threat) of populism vis-a-vis science (Grant, 2017). Mostly these are well-meaning attempts to improve science; in some cases, they are biased attempts to undermine public confidence (Ignore the public at your peril, Times Higher Education (THE), https://www.timeshighereducation.com/news/ignore-public-at-your-peril).

Health-related research funding is provided through a range of sources such as the commercial sector, public resources (e.g. taxation or government borrowing) and philanthropic funding agencies, whose source funding comes from charitable activity such as raising donations or through a charitable trust. RFAs have mandated (e.g. through legislation) as well as internally developed policies and procedures that guide how they decide what research to fund and how funded research should be conducted and communicated. Research funders operate within a specific context, what the World Health Organisation would describe as an “integrated health research system” (Pang et al. 2003). Although RFAs operate within different integrated research systems, defined by geographic, political, scientific and societal contexts, funding organisations share similar challenges. Whilst research funders can influence the environment in which they operate, they are one part of the wider research ecosystem.

Over the last 20 years, there have been a growing number of initiatives to improve health-related research practices across the spectrum of activities required to support research development, conduct and communication. These initiatives are often related to specific sectors of the research system (e.g. regulators or journal editors) or specific issues in the research process. Examples include the James Lind Alliance (https://www.jla.nihr.ac.uk/), public involvement in research identification and prioritisation, research integrity, evidence synthesis methodology such as Cochrane’s innovative methods development and evidence synthesis for pre-clinical research via SYRCLE.

The Research Waste and Rewarding Diligence Alliance (REWARD) is a significant initiative in the world-wide efforts for quality improvement in research. The REWARD Alliance came from an initial paper on avoidable research waste in research in 2009 (Chalmers and Glasziou, 2009), which was later followed up with a series (Lancet Series Research: increasing value, reducing waste, (https://www.thelancet.com/series/research)), detailing expert consensus recommendations for all sectors of the research ecosystem. Both the REWARD Alliance and the 2014 Lancet series recommended aligned and collective action by different stakeholder groups – one such group suggested was research funders. The formation of the Ensuring Value in Research (EViR) Funders’ Forum, whilst not formally part of the REWARD Alliance, was strongly influenced by the work of the REWARD Alliance.

This paper sets out ten Guiding Principles, and a conceptual model, developed by the EViR Funders’ Forum in an attempt to contextualise and address these issues for funders. It will be of interest to RFAs considering their own policies and practice and researchers seeking to understand funders’ perspectives on these issues.

Methods

To address the challenges facing health-related research, a group of health-related research funders, organisations that represent health-related research funders and those that inform and set funding policy from around the world, have come together to discuss these challenges, to share learnings, and to explore the potential for collaboration. The group started planning the Forum in 2016; the first meeting was held in January 2017. The group meets under the name of the Ensuring Value in Research Funders’ Collaboration and Development Forum (EViR Funders’ Forum, for short) (Chinnery et al. 2018).

The Forum’s focus is on health-related research for two reasons: firstly, the initial members are health-related research funders; secondly, the debates on research waste and integrity, whilst relevant to all areas of research, are more established and have greater resonance given the implications, risks and opportunities associated with health-related research.

The National Institute for Health Research (NIHR) in the UK, the Patient-Centered Outcomes Research Institute (PCORI) in the US and the Netherlands Organisation for Health Research and Development (ZonMw) in the Netherlands planned and co-hosted the initial meetings. This steering group has grown since. Since its inception, the Forum has met twelve times, typically hosted by a different health-related funding organisation, or lately online due to travel restrictions:

Through meetings, webinars (added in 2018), and surveys of current practice, the EViR Funders’ Forum has interacted with 53 organisations that either fund health-related research, represent funders or are active contributors to this agenda. Figure 1 lists the organisations from whom at least one individual attended at least one meeting, webinar or responded to electronic surveys.

Figure 1. EViR engagement across the world.

The Forum operates an open invitation policy whereby any public or philanthropic health-related research funding organisations, organisations that represent funders, or organisations that set health-related research funding policy, regardless of location, context, funding type, size or where on the research continuum (from basic discovery through to public health) may participate. The Forum also welcomes other non-funding related organisations to discuss the wider research system and specific topics of conversation. By encouraging diversity across a wide range of contexts, experiences and practices, the Forum provides a greater learning environment to start to address some of the uncertainties currently facing the whole research ecosystem. These practices also contribute to the broader purpose and aim of the EViR Funders’ Forum - to share experiences and to learn on what funders can do to maximise the probability of impact within and across respective research funding parameters.

Results

The development of the EViR Consensus Statement (see Figure 2) was an important step to ensure commitment, common purpose and shared understanding from funding organisations (Chinnery et al. 2018).

Figure 2. EViR Consensus Statement.

Following the Consensus Statement, the Funders’ Forum began to develop the EViR Guiding Principles. These principles are an extension of the Consensus Statement to help guide funders to ensure their work delivers the greatest value. The NIHR’s Adding Value in Research Framework (AViR, https://www.nihr.ac.uk/about-us/our-contribution-to-research/how-we-are-improving-research/adding-value-in-research.htm) was used to help shape the Guiding Principles. The NIHRs AViR Framework was initiated in response to the research waste, research integrity and research transparency literature [https://www.nihr.ac.uk/about-us/our-contribution-to-research/how-we-are-improving-research/adding-value-in-research.htm, and as a result of Lancet avoidable waste series (2014)]. The Guiding Principles were iterated and agreed by consensus during the first three EViR Funders’ Forum meetings. The ten Principles are published on the EViR website (https://evir.org/our-principles/) and reproduced in Table 1.

Tying it all together: A conceptual model

Beyond statements of intent and guiding principles, funders must consider how well their policies and practices align and support their mission of funding research that is credible and beneficial to the public. The ten Guiding Principles provide the practical link between the conceptual domains in the Consensus Statement to the actual policies, procedures and activities delivered by funders. The resulting conceptual model underpinning work of the EViR Funders’ Forum is shown in Figure 3.

Figure 3. Summary of Funders’ Forum Guiding Principles and conceptual model.

Many funders will describe the outcomes they are trying to achieve as impact. The EViR Funders’ Forum has deliberately not defined impact, as this is a much broader topic whose importance will be specific to an individual funder and context. However, the Forum does recognise that the probability of positive impact (benefit) should be enhanced through the application of these principles and practices designed to maximise value (see Figure 4). The model also draws on a tacit understanding that impacts are in relation to benefits accrued to society rather than impacts in relation to surrogate outcomes, such as academic publications and their related metrics. This would also include the impact research has on research itself; for example, by changing research agendas, opening up new avenues of exploration that could lead to further benefits, and closing down others that could not.

Figure 4. EViR “value” definition.

Funders are seeking to maximise the benefits their research delivers and ensure these are relative to the research costs incurred. Cost is interpreted broadly to include tangible costs such as time and money and intangible costs such as opportunity costs, political support, public support, participant and stakeholder experience and enthusiasm, patient and public participation.

To maximise the probability of benefit (impact) relative to the tangible and intangible costs incurred, funders’ policies and procedures should aim to ensure the research they support is:

• - Relevant to the intended end user of the research (e.g. patients and clinicians for clinical research; the next research community in the translational pathway for earlier phase research).

• - High quality and with minimal bias.

• - Transparent and open. This should also apply to funders’ activities.

These high-level aims are achieved by ensuring that:

• - Research priorities are justifiable.

• - Design, conduct and analysis are robust and appropriate.

• - Regulation and management are proportionate to risks.

• - Complete information on research methods and findings are publicly accessible and in usable formats.

• - Findings are appropriately and effectively disseminated.

An integrated conceptual model, as outlined in Figure 1, allows funding organisations to think coherently and comprehensively about their research practices to identify key areas for improvement. Policies and practices can be investigated and evaluated to determine their alignment with all or some of the Guiding Principles outlined in this paper. Conducting performance audits of specific policies can then help to guide quality improvement across a range of funders and within different contexts (e.g. Boutron et al. 2016, Nasser et al. 2017, Whitlock et al. 2018, Cody et al. 2021). The Forum is currently piloting an audit tool for members to track their performance against the conceptual model and Guiding Principles.

Discussion

For funders of health-related research, this work provides a logic model of the indicators of a high value research funder. The Guiding Principles are an aspiration we work to, rather than a checklist or a target to be achieved. As research policy and practice evolve, the interpretation of each Principle will accrue additional facets. For example, what constitutes a robust research design will change over time as new methods become available, and the ways in which research protocols and findings can be shared continue to increase. As such, the Guiding Principles give structure and focus to continual improvement strategies, first with policies and procedures and then with monitoring and measurement.

Despite the increased attention since the original paper on avoidable waste in research, Chalmers and Glasziou lamented that “research waste is still a scandal” but also recognised that the establishment of the EViR Funders’ Forum is “… perhaps the most notable and potentially influential development …” in addressing avoidable waste in research (Glasziou and Chalmers, 2018).

Conclusions

A research funder’s policies and procedures are aimed at ensuring the relevance, quality and transparency of the research process. When RFAs do this well, they will maximise the probability of benefits to society from the research they support - when they do this badly, they passively allow or actively contribute to research waste. This requires continuous, committed, concerted and collaborative effort. The funders within the EViR Funders’ Forum do just that – they work together to effect change within our own organisations and within the wider global research system.

We hope other organisations want to be involved in the Forum, so if you would like any further information about our work, or if your organisation is interested in joining the Forum, please contact evirfundersforum@gmail.com.