Examining the role of funders in ensuring value and reducing waste in research: An organizational case-study of the Patient-Centered Outcomes Research Institute

Introduction

In 2014, in response to concerns about avoidable waste in research prioritization, conduct, and reporting¹, The Lancet published a series of articles which identified specific recommendations for the biomedical research community to ensure value and minimize inefficiency in research^2–6. Research funders were a major target for these recommendations, along with regulators, journals, academic institutions and researchers themselves. Prompted by these and related activities, the biomedical research community around the world has begun considering best practices to ensure value in publicly-funded research. As key contributors⁷, research funders are encouraged to audit and update their own policies and practice, even as external assessments of funders are also undertaken^8,9.

In light of these trends, the Patient-Centered Outcomes Research Institute (PCORI), undertook an organizational case study of its policies and practices. PCORI was created in 2010 to address research needs of a range of healthcare stakeholders through clinical comparative effectiveness research, and ranks among the top 10 US non-commercial funders of health research (see Healthresearchfunders.org). Our goals were to examine and report how closely PCORI adheres to best practice recommendations for research funders (i.e., to foster transparency), to highlight areas of needed development for PCORI (to foster public accountability), and to consider how other research funders in the US and elsewhere can examine, report, and adopt best practices for supporting value in research (to foster enterprise-wide efficiency).

Methods

To maximize comparability, we adapted another funder’s self-assessment methods (M. Westmore, personal communication, June 15, 2016; See Adding Value in Research from the National Institute for Health Research). PCORI staff (KD, LF, EW) examined PCORI’s existing policies and initiatives against 17 recommendations for funding agencies from the Lancet series^2–6; after initial assessment, we consulted with additional PCORI staff members to confirm accurate interpretation of policies and processes (see PCORI site). Many of the 17 Lancet recommendations include multiple components. To accurately assess and transparently communicate our performance across all intended components of these recommendations, we subdivided some recommendations to capture each dimension within them separately, for a total of 35 sub-recommendations. (Table 1). Four authors (KD, LF, EW, GN) independently categorized fidelity to the 17 recommendations as: 1) “area of strength” –PCORI’s practices reasonably address all sub-recommendations; 2) “area of partial strength” –PCORI’s practices reasonably or partially address all sub-recommendations; 3) “area of growth” –PCORI’s practices do not address all sub-recommendations, either reasonably or partially; or 4) not applicable. We resolved discrepancies through discussion and final ratings reflect consensus.

Results

Table 1 represents a detailed summary (through November 2018) of PCORI’s policies and practices related to ensuring value in research. Across the 17 recommendations (35 sub-recommendations), two recommendations were not applicable (1, 8), and one recommendation primarily applies to non-funders (both 9a, 9b). For the 15 relevant recommendations, PCORI at least partially addresses most of the relevant sub-recommendations (28/33). Our consensus process categorized PCORI’s existing policies and practices as “areas of strength” for 6/15 applicable recommendations, “partial strength” for 3/15. PCORI’s authorizing legislation, although preceding the Lancet recommendations by several years, mandated a number of these (indicated in bold in the table).

Discussion

Our consensus process categorized PCORI’s existing policies and practices as meeting criteria for “areas of strength” or “partial strength” for many of the recommendations, and we also identified clear areas for growth. Examples of strengths include PCORI’s requirements that funded research adhere to methodology standards to minimize bias and that all study results are posted on the PCORI website to enhance public access to findings. On the other hand, PCORI has not yet fully developed its policies and practices related to rewarding research replication and reproducibility (Recommendation 7). Further development of performance metrics, standardized approaches to all study-related reporting, and enforcement of key policies (Recommendations 12, 13, 14) offer other areas ripe for growth, particularly if undertaken in coordination with others across the research enterprise. PCORI like many funders, is still actively developing its practices related to publicly sharing information, including raw data, as early as possible from funded research (Recommendations 4, 5). For example, making research protocols publicly available (Recommendation 5a) is required by PCORI’s authorizing legislation, but timing and format were not specified, and our current practices may not be ideal. PCORI now requires funded investigators to submit a study protocol and record its details in an appropriate registry but does not yet specify a standard protocol format nor require protocol publication before study completion. To our knowledge, just one funder (NIHR) clearly publishes study protocols at the time of award¹⁰. Nonetheless, making study protocols available at study inception can benefit the public by providing a detailed record of the planned study, which may help avoid unwitting duplication of research underway and support detection of important study deviations and post-hoc changes.

There is also opportunity for improvement through further development of policies and practices related to research data sharing and re-use. While funders can require awardees to share data from funded research and trial participants are supportive of such sharing¹¹, many researchers remain concerned about the impact on their work¹². PCORI’s policy on data sharing¹³ was informed by a public comment process as well as pilot work assessing time and effort required for investigators to prepare their data for sharing and on identifying appropriate repository models. Accelerating the practice of responsible data sharing necessitates broad coordination between journals, academic institutions, and data-repository organizations, alongside consistent requirements and support from funders. PCORI plans to monitor progress in these areas and conduct an updated self-assessment in two years.

Efforts to reduce waste and increase value in research are in alignment with trials transparency¹⁴, research integrity¹⁵, administrative efficiency¹⁶, and other similar initiatives. PCORI and other health research funders are in consortium to encourage further development and voluntary adherence to international best practice recommendations for research funders, (17; see Ensuring Value in Research (EVIR) website). The Ensuring Value in Research Funders’ Forum is exploring other initiatives, such as evaluating and sharing best practices for similar challenges that funders face, and considering what avenues exist to enhance efficiency and value in the full research agenda across funders. Beyond the consortium, greater transparency and coherence between funders and key players producing health research---including journals, research institutions, sponsors, and regulators---remains vital for tangible progress in our shared efforts^7,18.

Limitations: Our methods are limited by self-assessment, but findings are consistent with audit results for PCORI from external assessors¹⁰. In addition, the availability of policies or current practices represent only the first step, with actual performance measurement needed. Finally, while the Lancet series highlights areas for improvement for funders and others across the research enterprise, the impact of implementing and adhering to these recommendations on research value has yet to be demonstrated.

Data availability