Examining the role of funders in ensuring value and reducing waste in research: An organizational case-study of the Patient-Centered Outcomes Research Institute

International experts have recommended actions that funders can take to improve the value of research investments. They state that self-assessment and public sharing are the basis for accountability and improvement. We examined our policies and practice to determine the extent to which the Patient-Centered Outcomes Research Institute’s (PCORI) policies and practices as a research funder align with international best practice recommendations. A self-audit of current policies and practice against 17 recommendations and 35 sub-recommendations representing five major stages of research production, based on adapted methods used for self-assessment by another funder, was performed. Fit of existing PCORI policies and practices with 35 sub-recommendations, qualitative assessment of adequacy (area of strength; area of partial strength; area of growth; not applicable) for 17 recommendations for five stages of research production was assessed. Of the 17 recommendations, 15 were applicable to PCORI’s research mission and focus. PCORI has policies and practices in place for all elements of six recommendations (“area of strength”) and policies that address each element but with some still in active development for three (“area of partial strength”). PCORI is partially addressing six of the 15 relevant recommendations (“area of growth”). Areas for growth include making study protocols publicly available, improving policies on data sharing, and enhancing collaboration with other funders to reduce redundant funding. A voluntary consortium of international funders is underway to encourage further progress, including additional self-assessment and public sharing for accountability. These findings indicate PCORI has undertaken efforts to align its funding practices with international recommendations to ensure the value of public dollars invested in research. Further efforts will likely require additional coordination and collaboration between funders and stakeholders.


Introduction
In 2014, in response to concerns about avoidable waste in research prioritization, conduct, and reporting 1 , The Lancet published a series of articles which identified specific recommendations for the biomedical research community to ensure value and minimize inefficiency in research 2-6 . Research funders were a major target for these recommendations, along with regulators, journals, academic institutions and researchers themselves. Prompted by these and related activities, the biomedical research community around the world has begun considering best practices to ensure value in publicly-funded research. As key contributors 7 , research funders are encouraged to audit and update their own policies and practice, even as external assessments of funders are also undertaken 8,9 .
In light of these trends, the Patient-Centered Outcomes Research Institute (PCORI), undertook an organizational case study of its policies and practices. PCORI was created in 2010 to address research needs of a range of healthcare stakeholders through clinical comparative effectiveness research, and ranks among the top 10 US non-commercial funders of health research (see Healthresearchfunders.org). Our goals were to examine and report how closely PCORI adheres to best practice recommendations for research funders (i.e., to foster transparency), to highlight areas of needed development for PCORI (to foster public accountability), and to consider how other research funders in the US and elsewhere can examine, report, and adopt best practices for supporting value in research (to foster enterprise-wide efficiency).

Methods
To maximize comparability, we adapted another funder's selfassessment methods (M. Westmore, personal communication, June 15, 2016; See Adding Value in Research from the National Institute for Health Research). PCORI staff (KD, LF, EW) examined PCORI's existing policies and initiatives against 17 recommendations for funding agencies from the Lancet series 2-6 ; after initial assessment, we consulted with additional PCORI staff members to confirm accurate interpretation of policies and processes (see PCORI site). Many of the 17 Lancet recommendations include multiple components. To accurately assess and transparently communicate our performance across all intended components of these recommendations, we subdivided some recommendations to capture each dimension within them separately, for a total of 35 sub-recommendations. (Table 1). Four authors (KD, LF, EW, GN) independently categorized fidelity to the 17 recommendations as: 1) "area of strength" -PCORI's practices reasonably address all sub-recommendations; 2) "area of partial strength" -PCORI's practices reasonably or partially address all sub-recommendations; 3) "area of growth" -PCORI's practices do not address all sub-recommendations, either reasonably or partially; or 4) not applicable. We resolved discrepancies through discussion and final ratings reflect consensus. Table 1 represents a detailed summary (through November 2018) of PCORI's policies and practices related to ensuring value in research. Across the 17 recommendations (35 subrecommendations), two recommendations were not applicable (1,8), and one recommendation primarily applies to non-funders (both 9a, 9b). For the 15 relevant recommendations, PCORI at least partially addresses most of the relevant subrecommendations (28/33). Our consensus process categorized PCORI's existing policies and practices as "areas of strength" for 6/15 applicable recommendations, "partial strength" for 3/15. PCORI's authorizing legislation, although preceding the Lancet recommendations by several years, mandated a number of these (indicated in bold in the table).

Discussion
Our consensus process categorized PCORI's existing policies and practices as meeting criteria for "areas of strength" or "partial strength" for many of the recommendations, and we also identified clear areas for growth. Examples of strengths include PCORI's requirements that funded research adhere to methodology standards to minimize bias and that all study results are posted on the PCORI website to enhance public access to findings. On the other hand, PCORI has not yet fully developed its policies and practices related to rewarding research replication and reproducibility (Recommendation 7). Further development of performance metrics, standardized approaches to all studyrelated reporting, and enforcement of key policies (Recommendations 12, 13, 14) offer other areas ripe for growth, particularly if undertaken in coordination with others across the research enterprise. PCORI like many funders, is still actively developing its practices related to publicly sharing information, including raw data, as early as possible from funded research (Recommendations 4, 5). For example, making research protocols publicly available (Recommendation 5a) is required by PCORI's authorizing legislation, but timing and format were not specified, and our current practices may not be ideal. PCORI now requires funded investigators to submit a study protocol and record its details in an appropriate registry but does not yet specify a standard protocol format nor require protocol publication before study completion. To our knowledge, just one funder (NIHR) clearly publishes study protocols at the time of award 10 . Nonetheless, making study protocols available at study inception can benefit the public by providing a detailed record of the planned study, which may help avoid unwitting duplication of research underway and support detection of important study deviations and post-hoc changes.
There is also opportunity for improvement through further development of policies and practices related to research data sharing and re-use. While funders can require awardees to share data from funded research and trial participants are supportive of such sharing 11 , many researchers remain concerned about the impact on their work 12 . PCORI's policy on data sharing 13 was informed by a public comment process as well as pilot work

Amendments from Version 1
We have revised the article and expanded the methods section in response to comments from reviewers (Hans Lund and Mona Nasser). Table 1 • PCORI's Methodology Standards require that any proposed study be justified by evidence gaps identified through gap analysis or systematic review. (3a)

REVISED
• The first of the PCORI application Merit Review Criteria requires that applications demonstrate the potential for the study to fill critical gaps in evidence. (3a; 3b) • PCORI's legislation specifies its use of evidence synthesis to increase quality and relevance of information; programs for funding evidence syntheses have been expanding since 2016 including systematic reviews and updates, individual patient data meta-analysis and other evidence synthesis approaches. (3c) • PCORI has funded studies on improving methods for systematic reviews.  Bolded text indicates process or policy mandated by PCORI's authorizing legislation assessing time and effort required for investigators to prepare their data for sharing and on identifying appropriate repository models. Accelerating the practice of responsible data sharing necessitates broad coordination between journals, academic institutions, and data-repository organizations, alongside consistent requirements and support from funders. PCORI plans to monitor progress in these areas and conduct an updated selfassessment in two years. Limitations: Our methods are limited by self-assessment, but findings are consistent with audit results for PCORI from external assessors 10 . In addition, the availability of policies or current practices represent only the first step, with actual performance measurement needed. Finally, while the Lancet series highlights areas for improvement for funders and others across the research enterprise, the impact of implementing and adhering to these recommendations on research value has yet to be demonstrated.

Data availability
Underlying data All data underlying the results are available as part of the article and no additional source data are required.
addition, as the authors use another funder´s self-assessment method, the possible alterations or adjustments made in the present self-audit should be mentioned. If no alterations were done, this should also be mentioned. Using the same assessment method makes is possible to compare, and this could have been mentioned in the Discussion.
There is an * and a † in Table 1, I can´t find what these refers to.
In conclusion: this is a very important and useful report of a self-audit (see above), and with the minor adjustments mentioned is should be published the sooner the better.

Is the work clearly and accurately presented and does it cite the current literature? Yes
Is the study design appropriate and is the work technically sound? Yes

If applicable, is the statistical analysis and its interpretation appropriate? Not applicable
Are all the source data underlying the results available to ensure full reproducibility? Partly

Are the conclusions drawn adequately supported by the results? Yes
No competing interests were disclosed.

Competing Interests:
Reviewer Expertise: My professional content area is research within rehabilitation. Methodologically, I am using systematic reviews, meta-analyses, and meta-research. As the chair of the "Evidence-Based Research Network" I am fully occupied with issues related to promote ways of thinking and acting to improve the quality of research and to avoid waste in research.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
Author Response 24 May 2019 , Patient-Centered Outcomes Research Institute, Washington, USA

Kelly Dunham
We greatly appreciate the reviewer's comments and suggestions.
While we routinely monitor PCORI practice, we plan to conduct a second self-assessment in two years. We added this information to the discussion section.
We appreciate the reviewer's suggestion regarding the methods section. In the revised article, we explicitly linked to the 17 recommendations from the series. We made clearer in the text Lancet that we broke these into a total of 35 sub-recommendations, since the original 17 sometimes that we broke these into a total of 35 sub-recommendations, since the original 17 sometimes included multiple items against which it would have been difficult to assess and transparently communicate our current activities. In other words, it would have been easy to claim credit for doing one part of a recommendation when we were not addressing another aspect of the recommendation at all. The other funder's self-assessment was not published, so it cannot be linked to directly. We added a reference to personal communication in the revised article. We did not have a copy of their written methods so cannot make the clear comparison that is requested, but have made our approach more transparent to avoid confusion.
Thank you for pointing out the miscellaneous symbols in the table. These were holdovers from a previous version of the table and have been removed.
Thank you for the thoughtful critique. Your comments and questions have helped us to clarify the methods used and greatly improved the article.
No competing interests were disclosed.

Competing Interests:
The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias You can publish traditional articles, null/negative results, case reports, data notes and more The peer review process is transparent and collaborative Your article is indexed in PubMed after passing peer review Dedicated customer support at every stage For pre-submission enquiries, contact research@f1000.com