How an international research funder’s forum developed guiding principles to ensure value and reduce waste in research

Background: When health-related research funding agencies choose to fund research, they balance a number of competing issues: costs, stakeholder views and potential benefits. The REWARD Alliance, and the related Lancet-REWARD Campaign, question whether those decisions are yielding all the value they could. Methods: A group of health-related research funding agencies, organisations that represent health-related research funding agencies and those that inform and set health-related-research funding policy from around the world have come together since 2016 to share, learn, collaborate and influence emerging practice. This group meets under the name of the Ensuring Value in Research Funders’ Forum (EViR Funders’ Forum). The EViR Funders’ Forum worked together to develop a set of ten Guiding Principles, that if funders adhered to would reduce research waste and ensure value in research. Results: The EViR Funders’ Forum has previously agreed and published a Consensus Statement. The Forum has agreed on a set of ten Guiding Principles to help health-research funders to maximise the value of research by ensuring that: research priorities are justifiable; the design, conduct and analysis of research minimise bias; regulation and management are proportionate to risks; methods and findings are accessible in full; and findings are appropriately and effectively disseminated and used. Conclusions: When setting research funding policy, we must balance multiple stakeholders’ needs and expectations. When funders do this well, they maximise the probability of benefits to society from the research they support - when funders do this badly, they passively allow or actively contribute to research waste. These challenges must be resolved by funders either working together or in conjunction with other actors in the research ecosystem.


Introduction
The impact of health-related research, and the difference research makes for patients and the public, is the main focus of research funding agencies.It is the reason health-related research maintains the support of our health, public health and social care services and society more generally.But it is also essential to consider and acknowledge how funders deliver research systems and whether there are opportunities to enhance the efficiency and effectiveness of funding organisations.
Research funding agencies' (RFAs) practice improvements are often set in the context of current global debates, specifically focusing on the purpose, accountability, and quality of research.These include issues raised about research integrity (World Conference on Research Integrity, https://wcrif.org/); the "crisis" of reproducibility in research (Ioannidis, 2016); avoidable waste across the research enterprise (Chalmers and Glasziou, 2009, Lancet Series Research: increasing value, reducing waste (https://www.thelancet.com/series/research)); the impact, openness and transparency of specific health-related research activities (AllTrials, All Trials Campaign, http://www.alltrials.net/,Chalmers, Glasziou and Godlee, 2013); and the rise (and threat) of populism vis-a-vis science (Grant, 2017).Mostly these are well-meaning attempts to improve science; in some cases, they are biased attempts to undermine public confidence (Ignore the public at your peril, Times Higher Education (THE), https://www.timeshighereducation.com/news/ignore-public-at-your-peril).
Health-related research funding is provided through a range of sources such as the commercial sector, public resources (e.g.taxation or government borrowing) and philanthropic funding agencies, whose source funding comes from charitable activity such as raising donations or through a charitable trust.RFAs have mandated (e.g. through legislation) as well as internally developed policies and procedures that guide how they decide what research to fund and how funded research should be conducted and communicated.Research funders operate within a specific context, what the World Health Organisation would describe as an "integrated health research system" (Pang et al. 2003).Although RFAs operate within different integrated research systems, defined by geographic, political, scientific and societal contexts, funding organisations share similar challenges.Whilst research funders can influence the environment in which they operate, they are one part of the wider research ecosystem.
Over the last 20 years, there have been a growing number of initiatives to improve health-related research practices across the spectrum of activities required to support research development, conduct and communication.These initiatives are often related to specific sectors of the research system (e.g.regulators or journal editors) or specific issues in the research process.Examples include the James Lind Alliance (https://www.jla.nihr.ac.uk/), public involvement in research identification and prioritisation, research integrity, evidence synthesis methodology such as Cochrane's innovative methods development and evidence synthesis for pre-clinical research via SYRCLE.
The Research Waste and Rewarding Diligence Alliance (REWARD) is a significant initiative in the world-wide efforts for quality improvement in research.The REWARD Alliance came from an initial paper on avoidable research waste in research in 2009 (Chalmers and Glasziou, 2009), which was later followed up with a series (Lancet Series Research: increasing value, reducing waste, (https://www.thelancet.com/series/research)),detailing expert consensus recommendations for all sectors of the research ecosystem.Both the REWARD Alliance and the 2014 Lancet series recommended aligned and collective action by different stakeholder groupsone such group suggested was research funders.The formation of the Ensuring Value in Research (EViR) Funders' Forum, whilst not formally part of the REWARD Alliance, was strongly influenced by the work of the REWARD Alliance.This paper sets out ten Guiding Principles, and a conceptual model, developed by the EViR Funders' Forum in an attempt to contextualise and address these issues for funders.It will be of interest to RFAs considering their own policies and practice and researchers seeking to understand funders' perspectives on these issues.

REVISED Amendments from Version 1
We are particularly grateful for the reviewers' insightful comments, which we fully agree with, even if some we were unable to amend the paper directly.We would like to acknowledge one of the reviewer's comments about including further details on why each guiding principle is important and some practical examples of how to deliver them.While we were not able to include more detail in the article, the information is available on our website -Applying the principles -EViR (https://evir.org/our-principles/applying-theprinciples/)Minor Comments: On page 8, the figure number referred to was wrongly numbered and has been updated to Figure 3.

Methods
To address the challenges facing health-related research, a group of health-related research funders, organisations that represent health-related research funders and those that inform and set funding policy from around the world, have come together to discuss these challenges, to share learnings, and to explore the potential for collaboration.The group started planning the Forum in 2016; the first meeting was held in January 2017.The group meets under the name of the Ensuring Value in Research Funders' Collaboration and Development Forum (EViR Funders' Forum, for short) (Chinnery et al. 2018).The Forum's focus is on health-related research for two reasons: firstly, the initial members are health-related research funders; secondly, the debates on research waste and integrity, whilst relevant to all areas of research, are more established and have greater resonance given the implications, risks and opportunities associated with health-related research.
The National Institute for Health Research (NIHR) in the UK, the Patient-Centered Outcomes Research Institute (PCORI) in the US and the Netherlands Organisation for Health Research and Development (ZonMw) in the Netherlands planned and co-hosted the initial meetings.This steering group has grown since.Since its inception, the Forum has met twelve times, typically hosted by a different health-related funding organisation, or lately online due to travel restrictions: 1. London, England.NIHR -January 2017 Through meetings, webinars (added in 2018), and surveys of current practice, the EViR Funders' Forum has interacted with 53 organisations that either fund health-related research, represent funders or are active contributors to this agenda.Figure 1 lists the organisations from whom at least one individual attended at least one meeting, webinar or responded to electronic surveys.
The Forum operates an open invitation policy whereby any public or philanthropic health-related research funding organisations, organisations that represent funders, or organisations that set health-related research funding policy, regardless of location, context, funding type, size or where on the research continuum (from basic discovery through to public health) may participate.The Forum also welcomes other non-funding related organisations to discuss the wider research system and specific topics of conversation.By encouraging diversity across a wide range of contexts, experiences and practices, the Forum provides a greater learning environment to start to address some of the uncertainties currently facing the whole research ecosystem.These practices also contribute to the broader purpose and aim of the EViR Funders' Forum -to share experiences and to learn on what funders can do to maximise the probability of impact within and across respective research funding parameters.

Initial activities
The initial activities of the EViR Funders' Forum focused on developing a Consensus Statement and a set of Guiding Principles to serve as the backbone of our collaboration (Chinnery et al. 2018).Surveys and sharing current practices through the surveys and open discussions were carried out to obtain and increase wider perspectives from other funding organisations.

Results
The development of the EViR Consensus Statement (see Figure 2) was an important step to ensure commitment, common purpose and shared understanding from funding organisations (Chinnery et al. 2018).
Following the Consensus Statement, the Funders' Forum began to develop the EViR Guiding Principles.These principles are an extension of the Consensus Statement to help guide funders to ensure their work delivers the greatest value.The NIHR's Adding Value in Research Framework (AViR, https://www.nihr.ac.uk/about-us/our-contributionto-research/how-we-are-improving-research/adding-value-in-research.htm) was used to help shape the Guiding Principles.The NIHRs AViR Framework was initiated in response to the research waste, research integrity and research transparency literature [https://www.nihr.ac.uk/about-us/our-contribution-to-research/how-we-are-improving-research/ adding-value-in-research.htm, and as a result of Lancet avoidable waste series ( 2014)].The Guiding Principles were iterated and agreed by consensus during the first three EViR Funders' Forum meetings.The ten Principles are published on the EViR website (https://evir.org/our-principles/) and reproduced in Table 1.
Tying it all together: A conceptual model Beyond statements of intent and guiding principles, funders must consider how well their policies and practices align and support their mission of funding research that is credible and beneficial to the public.The ten Guiding Principles provide the practical link between the conceptual domains in the Consensus Statement to the actual policies, procedures and activities delivered by funders.The resulting conceptual model underpinning work of the EViR Funders' Forum is shown in Figure 3.Many funders will describe the outcomes they are trying to achieve as impact.The EViR Funders' Forum has deliberately not defined impact, as this is a much broader topic whose importance will be specific to an individual funder and context.However, the Forum does recognise that the probability of positive impact (benefit) should be enhanced through the application of these principles and practices designed to maximise value (see Figure 4).The model also draws on a tacit understanding that impacts are in relation to benefits accrued to society rather than impacts in relation to surrogate outcomes, such as academic publications and their related metrics.This would also include the impact research has on research itself; for example, by changing research agendas, opening up new avenues of exploration that could lead to further benefits, and closing down others that could not.
Funders are seeking to maximise the benefits their research delivers and ensure these are relative to the research costs incurred.Cost is interpreted broadly to include tangible costs such as time and money and intangible costs such as opportunity costs, political support, public support, participant and stakeholder experience and enthusiasm, patient and public participation.
To maximise the probability of benefit (impact) relative to the tangible and intangible costs incurred, funders' policies and procedures should aim to ensure the research they support is: -Relevant to the intended end user of the research (e.g.patients and clinicians for clinical research; the next research community in the translational pathway for earlier phase research).
-High quality and with minimal bias.

Principle 1
Health-related research agendas and priorities should be set with the meaningful involvement of those who will use and be affected by health-related research.
Ensuring robust research design, conduct and analysis.

Principle 2
Research should only be funded if set in the context of one or more existing systematic reviews of what is already known (or an otherwise robust demonstration of a research gap).

Principle 3
Funders should take into account advances in research methodology and fund new research only if adequate steps have been taken to reduce bias.
Ensuring regulation and management of research conduct are proportionate to the costs and risks of research.

Principle 4
Selection and conduct of research should be actively managed in a risk proportionate way, consistent with applicable human participant research laws, regulations, and ethical guidance.
Principle 5 Studies should be registered in an appropriate, design-relevant publicly accessible registry at study inception whenever possible.
Ensuring all information on research methods and findings are accessible and all reports are complete and usable.

Principle 6
Research questions, methods, materials, analysis plans or sequence of analytical choices for all studies should be made available as early as possible and preferably near or before the start of the study or analysis.Any deviation from the original plans should be documented.

Principle 7
All studies should report methods and findings in full, following credible and justifiable reporting guidelines.This applies irrespective of the nature of the findings and whether the study completed as planned.

Principle 8
When appropriate and when it will add value to evidence users, replication, re-analysis, and re-use of data from studies should be supported and facilitated.
Principle 9 New evidence should be placed in the context of existing knowledge to inform appropriate interpretation and use of findings.When appropriate and when it will add value to evidence users, systematic reviews should be updated following primary research.

Principle 10
Research knowledge that can lead to benefit should be effectively disseminated with and to end users.Where appropriate, the usage of new knowledge should be supported and facilitated.
-Transparent and open.This should also apply to funders' activities.
These high-level aims are achieved by ensuring that: -Research priorities are justifiable.
-Design, conduct and analysis are robust and appropriate.
-Regulation and management are proportionate to risks.
-Complete information on research methods and findings are publicly accessible and in usable formats.
-Findings are appropriately and effectively disseminated.
An integrated conceptual model, as outlined in Figure 1, allows funding organisations to think coherently and comprehensively about their research practices to identify key areas for improvement.Policies and practices can be investigated and evaluated to determine their alignment with all or some of the Guiding Principles outlined in this paper.Conducting performance audits of specific policies can then help to guide quality improvement across a range of funders and within different contexts (e.g.Boutron et al. 2016, Nasser et al. 2017, Whitlock et al. 2018, Cody et al. 2021).The Forum is currently piloting an audit tool for members to track their performance against the conceptual model and Guiding Principles.

Discussion
For funders of health-related research, this work provides a logic model of the indicators of a high value research funder.
The Guiding Principles are an aspiration we work to, rather than a checklist or a target to be achieved.As research policy and practice evolve, the interpretation of each Principle will accrue additional facets.For example, what constitutes a robust research design will change over time as new methods become available, and the ways in which research protocols and findings can be shared continue to increase.As such, the Guiding Principles give structure and focus to continual improvement strategies, first with policies and procedures and then with monitoring and measurement.
Despite the increased attention since the original paper on avoidable waste in research, Chalmers and Glasziou lamented that "research waste is still a scandal" but also recognised that the establishment of the EViR Funders' Forum is "… perhaps the most notable and potentially influential development …" in addressing avoidable waste in research (Glasziou and Chalmers, 2018).

Implications
The Forum has used the Guiding Principles to share their experience, learn from each other and collaborate to move its collective understanding forward.Diverse topics have been discussed against the framework of the Guiding Principles: for example, data sharing, automation, priority setting, novel decision-making processes, and transparency.
Longer-term working groups have focussed on implications for preclinical research (Ritskes-Hoitinga et al. 2018); stakeholder engagement; dissemination and implementation; systematic reviews ahead of new primary research; and self-audit against the Guiding Principles as a whole.
To date, funders in the Forum have used the Guiding Principles as a tool for internal measurement, process improvement, and strategy development.Funders in the Forum have also applied the Guiding Principles to standards/requirements for researchers seeking funding.The Scar Free Foundation now requires applicants for funding to embed the EViR Guiding Principles in the study design, management and dissemination of their research.Several funders now require researchers to demonstrate that there is a need for the proposed study that is informed by a systematic review or other robust evaluation of the available evidence (Italian Ministry of Health, NIHR, NHMRC, PCORI, HCRW, HRB and the Stroke Association).While NIHR and PCORI had this requirement prior to the formation of the Funders' Forum, others have been prompted by their participation in the Forum.
One of the authors (MR-H) has led a linked project to highlight ways to apply the Guiding Principles to pre-clinical research by adding pertinent examples.In 2020, a pre-clinical working group was established which will focus on furthering the implementation of the Guiding Principles for pre-clinical research as well.
It is important that funders understand current practice internationally and evaluate progress in improving research systems.The Guiding Principles provide a framework for this discussion.Some principles lend themselves to simple quantitative measures (e.g. percentage of studies registered, Knowles 2020), while others are more complex and qualitative.Even those that may at first appear more straightforward need to be seen in context: for example, whilst regulated clinical trials are required by law to be registered, the same does not apply to other types of health research and practice varies between health services research, biomedical research, and population health research.Funders and the wider community will need to develop methods for monitoring and tracking these process measures.In doing so, it is important to remember that these will inevitably only be indicators.These indicators are useful because they are largely within our collective influence within medium term time frames.However, they should not be confused with the end aim; namely, maximising the benefits to society from research, which is the end state of a complex system.
There are also implications for research-on-research or meta-research.Research-on-research uses a range of research methods to investigate the process of research or of funders' policies and guidance with the aim of improving the planning, conduct and sharing of research as well as its governance and oversite, including how funding decisions are made.The concepts in this paper are grounded in the research-on-research literature but it is fragmented and often descriptive.More research is required under each guiding principle.The Guiding Principles may provide the framework for a research-on-research agenda that can provide evidence regarding funders' policies and practice.This is already beginning to happen; the NIHR has a long running research-on-research programme that has used the concepts and issues discussed here as an organising framework for its research agenda (NIHR research on research (RoR), https://www.southampton.ac.uk/netscc/research/index.page).
This work highlights how funders are placing greater emphasis on research practices and by doing so contribute to the improvements of the research they fund.The Funders' Forum and its Guiding Principles will continue to translate into funder policies and procedures, and to ensure value in research.

Conclusions
A research funder's policies and procedures are aimed at ensuring the relevance, quality and transparency of the research process.When RFAs do this well, they will maximise the probability of benefits to society from the research they support -when they do this badly, they passively allow or actively contribute to research waste.This requires continuous, committed, concerted and collaborative effort.The funders within the EViR Funders' Forum do just thatthey work together to effect change within our own organisations and within the wider global research system.
We hope other organisations want to be involved in the Forum, so if you would like any further information about our work, or if your organisation is interested in joining the Forum, please contact evirfundersforum@gmail.com.

Importance and Originality
This article is both timely and vital, emphasizing the necessity of upholding research integrity to enhance the value delivered in health care and patient outcomes.

Comments
As this manuscript is in its second round of reviews, I have just two minor suggestions.The Comments of Hans Lund have been addressed.

1.
It is recommended that the title explicitly reference its focus on health research to prevent potential misunderstandings among readers.

2.
The manuscript should include an expanded discussion on the geographic distribution of member institutions, all of which are situated in high-income countries.An analysis of how this may shape the formulated principles is crucial, especially considering that the predominant burden of disease lies in lower-income regions.This will provide a more comprehensive understanding of the forum's global impact and relevance.

Hans Lund
Section Evidence-Based Practice, Department of Health and Functioning, Western Norway University of Applied Sciences, Bergen, Norway The authors and the organisation behind should be praised for their initiative and for promoting a Comment 3.This is an interesting view that clearly has merit.The definition of value we used was purposefully centred on real world impact or outcomes and was inspired by Michael Porter's definition of healthcare value in http://www.nejm.org/doi/full/10.1056/NEJMp1011024?viewType=Print& The reason for choosing a definition firmly related to real world impact was to ensure that our focus was on real world benefits rather than outputs or process issues like scientific validity and relevance.This allowed a diverse group of funders, from across the world, with different political, social and scientific contexts, from pre-clinical and early phases to health service research, to align under the idea that we are all interested in delivering benefits to society rather than just good science.As well as creating a sense of alliance, to be candid, this prevented us from discussing different views and definitions of research impact, and meant we could focus on the task at hand, which was to develop useful guidance to change funder practices.
That said, the reviewer's points are entirely correct, which is why we structured our framework (figure 3) under the headings of relevance, quality (akin to validity), and openness.It could be interpreted that we were saying value = (relevance x quality x openness) / costs, which aligns with the reviewer's thinking.We did not articulate it that way in the agreed version, so while we don't disagree with the reviewer, it would not be appropriate to change the text now.
Comment 4. We agree, and the manuscript has been updated.
Comment 5. We agree with this point.Unfortunately, the journal prevented us from including further details on why each guiding principle is important and some practical examples of how to delivery them because we had already placed this material on our website.We can not add this to the paper but the reviewer may be reassured that we agree and have included this on our website -Applying the principles -EViR Competing Interests: No competing interests were disclosed.This paper provides a narrative overview of the creation and evolution of the EViR Funders' Forum and its development of a Consensus Statement and Guiding Principles to help health research funders to maximise the value of research.It is a valuable addition to the literature because it describes an innovative platform that allows and enables diverse funders from different countries to jointly discuss how to address issues of common concern.
One arguable shortcoming of the paper is that it fails to explore the more 'political' (in the broadest sense) aspects of funders' engagement with the issues of research waste and research quality.Funders may face considerable barriers to acting decisively to maximise the value of research.For instance, openly acknowledging that funders may not be making optimal use of public money could invite criticism from political decision-makers and taxpayers and threaten their budgets.Another barrier is that funders often seem determined to maintain positive relationships -including long-standing personal relationships -with the research community, which disincentivises the imposition of sanctions.
For example, the world's largest medical research funder, the National Institutes of Health (NIH), has continued to award new funding to grantees that had violated NIH's own policies and thereby generated research waste (https://www.statnews.com/pharmalot/2022/10/14/clinical-trials-nihtransparency/).In general, there often seems to be a gap between funders' publicly stated aspirations and their subsequent translation into practice.In this context, it is excellent that the authors cite Knowles et al., who focuses on the thorny issue of implementation.
However, given space constraints, omission of these more 'political' aspects from the scope of the article is defensible, especially given that the Consensus Statement and Guiding Principles provide a useful starting point for further debate about funders' upstream aspirations.The Forum's experiences gained during the ongoing piloting of an audit tool might provide a good starting point for a future, separate publication that focuses on the downstream challenges of implementation.
In sum, this paper is a valuable addition to the literature.I look forward to future publications documenting the Forum's further development and experiences over time.The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias • You can publish traditional articles, null/negative results, case reports, data notes and more • The peer review process is transparent and collaborative • Your article is indexed in PubMed after passing peer review • Dedicated customer support at every stage • For pre-submission enquiries, contact research@f1000.com 2. The Hague, Netherlands.ZonMw -June 2017 3. Washington D.C., USA.PCORI -November 2017 4. Cardiff, Wales.Health Care Research Wales (HCRW) -May 2018 5. Canberra, Australia.National Health and Medical Research Council (NHMRC) -November 2018 6. Dublin, Ireland.Health Research Board (HRB) -March 2019 7. Washington, DC., USA.PCORI -September 2019 8. Berlin, Germany.REWARD-EQUATOR Conference 2020 9. Virtual -September 2020 10.Virtual -October 2021 11.Virtual -March 2022 12. Virtual -October 2022

Figure 3 .
Figure 3. Summary of Funders' Forum Guiding Principles and conceptual model.

Reviewer Report 24
March 2023 https://doi.org/10.5256/f1000research.141421.r167144© 2023 Bruckner T. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.Till Bruckner QUEST Center for Transforming Biomedical Research, Berlin Institute of Health (BIH), Charité -Universitätsmedizin Berlin, Berlin, Germany

the work clearly and accurately presented and does it cite the current literature? Yes Is the study design appropriate and is the work technically sound? Yes Are sufficient details of methods and analysis provided to allow replication by others? Yes If applicable, is the statistical analysis and its interpretation appropriate? Not applicable Are all the source data underlying the results available to ensure full reproducibility
?No source data requiredAre the conclusions drawn adequately supported by the results?YesCompeting Interests: No competing interests were disclosed.Reviewer Expertise: Epidemiology, Meta ResearchI confirm that I

have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
QUEST Center for Transforming Biomedical Research, Berlin Institute of Health (BIH), Charité -Universitätsmedizin Berlin, Berlin, Germany I welcome the indexing of this paper and have no further comments to make.
Reviewer Report 28 February 2024 https://doi.org/10.5256/f1000research.160448.r233984©2024 Bruckner T. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.Till Bruckner

the work clearly and accurately presented and does it cite the current literature? Yes Is the study design appropriate and is the work technically sound? Yes Are sufficient details of methods and analysis provided to allow replication by others? Yes If applicable, is the statistical analysis and its interpretation appropriate? Yes Are all the source data underlying the results available to ensure full reproducibility? Yes Are the conclusions drawn adequately supported by the results? Yes Competing Interests:
I have met the lead author in a professional context on several occasions.I have delivered one or two unpaid presentations at EVIR since my initial review of this paper.As the founder of the TranspariMED campaign, my advocacy activities involve engaging with both EVIR and the UK Health Research Authority (the latter is currently chaired by the lead author), and actively participate in an HRA campaign group as an unpaid volunteer.However, I have no employment or financial relationship with any of the authors, nor with EVIR, nor the HRA.I confirm that these potential conflicts of interest did not affect my ability to write an objective and unbiased review of the article.

have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Version 1
This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Is the work clearly and accurately presented and does it cite the current literature? Yes Is the study design appropriate and is the work technically sound? Yes Are sufficient details of methods and analysis provided to allow replication by others? Partly If applicable, is the statistical analysis and its interpretation appropriate? Not applicable Are all the source data underlying the results available to ensure full reproducibility? No source data required Are the conclusions drawn adequately supported by the results? Yes Competing Interests:
No competing interests were disclosed.