Non-commercial pharmaceutical R&D: what do neglected diseases suggest about costs and efficiency?

Participant selection

The study population was selected using the database of pipeline technologies for neglected diseases developed by Duke University and Policy Cures Research¹⁰. The database does not include all non-commercial R&D initiatives – for example, it excludes biodefense projects that are largely publicly-funded – but it is the most comprehensive database of which we are aware focusing on R&D for neglected diseases, which is by nature largely non-commercial. We used a version of the spreadsheet “Candidates in the pipeline for neglected diseases, as of August 31, 2017” sent to us by the authors, which included a categorization of the organizations by developer type. We selected all not-for-profit organizations that were directly involved in conducting R&D, which included product development partnerships (PDPs), academic and research institutions and public research institutes and other public sector organizations. There were a total of 443 candidate products and 285 organizations that fit this initial criterion. All 16 PDPs were included given their organizational focus on non-commercial R&D, and the list of PDPs was complemented by other studies^1,11. In addition to PDPs, we included organizations that had at least one product that had reached Phase 3. Another 32 organizations were included after this second selection, and three additional organizations were included through snowball sampling. The final list of organizations is available in the full research report (extended data; Annex 1, pp. 77–78⁹). We contacted each organization by email, with the initial request addressed to the organization’s most senior executive (e.g. Chief Executive Officer, Executive Director, Managing Director) to ensure leadership was aware of and agreed to our interview request, as recommended by the ethical review process. In specific cases, where we had reason to know another employee would be relevant to or aware of our research project, we copied other individuals on the initial email. The senior executive often delegated the interview to one or more staff, such as the lead staff person responsible for R&D, finance, policy and/or external relations.

Quantitative data – collection and analysis

We developed a questionnaire using MS Excel to collect quantitative data from the organizations in our sample on the costs, timeframes, and attrition rates for a given organization or project (extended data; Annex 2, pp. 79–89⁹). We also included quantitative data on costs, timeframes or attrition rates published in reports or articles prior to the start of this study from four organizations^4–7. For timeframes, we consulted data available on the organizations’ websites and in the clinical trials database ClinicalTrials.gov.

We created a quantitative dataset in Excel combining data provided by respondents with publicly available information pertaining to the costs, timeframes, and attrition rates of non-commercial R&D initiatives. Data was anonymized and combined by product archetype. Due to the limitations of our dataset, we limited our analysis to only two P2I archetypes: simple and complex new chemical entities (NCE-Simple, NCE-Complex)¹². Only one organization provided information about diagnostics and two about vaccines (one only included aggregated totals for one product), and we excluded these from the analysis as it would be impossible to protect the anonymity of the organizations.

Several assumptions were made to standardize the data and allow comparison (see detailed methodology in extended data; pp. 31–32⁹). For costs, we assumed that money was spent at a steady rate across the time period of each phase. For data in currencies other than USD, a yearly exchange rate from the year in which the cost was incurred was used to make the conversion into USD. Totals were calculated in 2017 dollars to facilitate comparison with P2I model figures. Inflation and deflation adjustments used the standard consumer price index.

For timeframes, we calculated total time spent in development as the sum of time spent in Pre-clinical, Phase 1, 2 and 3. Phase 1a and 1b trials were counted as phase 1. Phase 2a, 2b, and 2c trials were counted as phase 2, as were phase 2/3 tests. Phase 3a and 3b are both counted as phase 3. An average of time per trial for each phase was taken to estimate the amount of time required for a given clinical phase. For some data points, early stage testing was aggregated for multiple drug candidates and total time spent in a phase was divided by the number of candidates. Many candidates have multiple trials in each phase and the average of all trials was used.

Qualitative data – collection and analysis

A list of questions for semi-structured interviews was developed to collect the qualitative data (extended data; Annex 3, p. 90⁹). The questions were not pilot tested. Most interviews lasted about one hour and were either conducted in person in Geneva or using videoconferencing software and were recorded upon agreement of the participant. Interviews were held with individuals with a high degree of familiarity about product development from the organizations included in the study. Interviews were held by the three authors (MV- MPH, researcher, female; RK – researcher, male; SM – PhD., director of research, female) and no one else was present besides the participants and the researchers.

Transcription or notes from the interviews were analysed and coded using NVivo 12. Interviews were coded by MV based on themes derived from the data, and reviewed by SM. A description of the coding tree is not available. Interviews were anonymized both at individual and organization level and each was given a number (PO - participant organization) for quotation identification. Given the small sample size, data saturation was not reached.

Parameter of comparison

We used parameters from the Portfolio-to-Impact (P2I) tool v2, which was initially developed by TDR⁸ and updated by Duke University and Policy Cures Research¹⁰. The P2I Model is based on assumptions of costs, timeframes and attrition rates. Assumptions on development costs at each phase were based on clinical trial costs from Parexel’s R&D cost sourcebook, derived from historical data on health product development of more than 25,000 candidates for all diseases, and further refined and validated by interviews. The underlying data used to construct those assumptions is not publicly available and it was not possible to disaggregate costs, timeframes or attrition rates by commercial vs non-commercial developer. Given that non-commercial R&D (in particular, non-commercial late-stage product development) is both relatively recent and small in scale, we assume that the vast majority of the data used to construct the P2I averages comes from commercial R&D. We compared our quantitative data to P2I averages, and our qualitative data compared non-commercial with commercial R&D.

We conducted a literature review on costs, timeframes and attrition rates of biomedical R&D to compare other estimates with those of the P2I Model (available at the Knowledge Portal on Innovation and Access to Medicines). There is a wide range of estimates, most focusing on the development of new chemical entities (NCEs) by pharmaceutical companies (commercial R&D). P2I averages are at the low end of the range of cost estimates available for commercial R&D^13–16, are somewhat higher for phase 1, lower for phase 2 and within the same range for phase 3 timeframes^14,17–19 and success rates are lower for complex NCEs and in the same range for simple NCEs^{17,18,20–23}. In comparison to the few estimates available for non-commercial R&D initiatives^4,24, the P2I Model assumptions estimate lower success rates in all clinical development phases, while preclinical is higher. A more detailed literature review is available in the full research report.

Costs

Quantitative data on non-commercial R&D costs were largely in line with the P2I model estimates, with some variation by phase. For simple NCEs, total costs for non-commercial R&D were 13% higher than the P2I estimates (51.87 million USD for non-commercial vs 45.84 million USD for P2I) (Figure 1). The largest differences were in pre-clinical and phase 1 – where the costs in our sample were more than double the P2I model estimates. Conversely, phase 2 and 3 trials were less expensive for simple NCEs in our data, but by a small margin. The sample size is too small for statistical significance or to generalize to non-commercial R&D more broadly; rather, the findings merely suggest a hypothesis that overall costs to develop simple NCEs are similar between non-commercial R&D initiatives and P2I averages.

Figure 1. Development costs – collected data vs. P2I model (NCE-simple).

The figure compares the development costs collected from the organizations in the study and the averages in the P2I Model for the archetype “new chemical entities – simple”. The comparison is made for different stages of development preclinical, phase 1, phase 2 and phase 3 and total costs. Data shows higher costs for collected data at preclinical, phase 1 and total, and lower costs at phase 2 and phase 3. Collected data for preclinical is based on 4 data points (n=4), for phase 1 n=3, for phase 2 n=3 and for phase 3 n=2.

For complex NCEs, total costs were similar to the P2I model, 8% lower (53.98 million USD for non-commercial vs 58.93 million USD in P2I) (Figure 2). In contrast with simple NCEs, for complex NCEs non-commercial preclinical and phase 1 costs were lower than the P2I model. Notably, phase 2 costs were much higher in our dataset (12.65 million USD vs 6.39 million USD in P2I). This could be in part because of the high proportion of phase 2/3 trials in the dataset, as well as the ratio of phase 2b to 2a tests being higher than the P2I data. Phase 3 costs were substantially lower than the P2I estimates, which may be explained by the fact that many pivotal trials were in phase 2. The opportunity to forgo phase 3 testing would drive up phase 2 costs while lowering phase 3 costs. The proportion of pivotal phase 2 tests may be different between P2I and our dataset. As with simple NCEs, the findings merely suggest a hypothesis that should be tested against a larger dataset – that overall costs to develop complex NCEs are similar between non-commercial R&D initiatives and P2I averages.

Figure 2. Development costs – collected data vs. P2I model (NCE-complex).

The figure compares the development costs collected from the organizations in the study and the averages in the P2I Model for the archetype “new chemical entities – complex”. The comparison is made for different stages of development preclinical, phase 1, phase 2 and phase 3 and total costs. Data shows lower costs for collected data at preclinical, phase 1, phase 3 and total, and higher costs at phase 2. Collected data for preclinical is based on 8 data points (n=8), for phase 1 n=6, for phase 2 n=4 and for phase 3 n=4.

To assess how sensitive our results were to coding by archetypes (i.e. characterizing a product as “simple” or “complex” NCEs), we combined our data from both categories and found that total costs (48.9m USD) lay between P2I’s NCE-simple and NCE-complex estimates (45.8m-59.9m USD) (Figure 3). This sensitivity analysis suggests that total non-commercial costs are largely in line with P2I parameters, even if there are some differences in coding of archetypes.

Figure 3. Development costs – collected data vs. P2I model (combined).

The figure compares the development costs collected from the organizations in the study combining the archetypes “new chemical entities – simple” and “new chemical entities – complex” combined and the averages in the P2I Model for the archetypes “drug repurposed complex”, “new chemical entities – simple” and “new chemical entities – complex”. The comparison is made for different stages of development preclinical, phase 1, phase 2 and phase 3 and total costs.

The qualitative data identified 12 factors that drove costs up or down in the different phases of product development within non-commercial R&D initiatives⁹ (Table 1). Most responses focused on the clinical stages of development rather than pre-clinical or earlier. Three factors pushed costs upward, and five factors pushed costs downward for non-commercial R&D in comparison with commercial. Four factors were categorized as indeterminate as they would affect both non-commercial and commercial R&D in the same way. The table below presents a summary of the factors influencing costs. A description of the factors and sample quotes are available in the full research report.

There were more factors that would push costs for non-commercial R&D down vis-à-vis commercial models. However, as the qualitative data does not tell us about the magnitude of the effects, no firm conclusions can be drawn on whether non-commercial R&D would generally cost the same, less or more than commercial R&D.

Timeframes

The quantitative data showed that for simple NCEs, timeframes between non-commercial R&D and P2I averages were roughly similar. Non-commercial R&D had shorter preclinical times (1.65 years vs 2.49 years in P2I), and longer phase 1 times (2.61 vs 1.80 years in P2I). Non-commercial R&D also had much shorter phase 2 times (1.75 vs 3.38 years in P2I), while phase 3 times were slightly higher (3.67 vs 3.18 years in P2I). Overall, our dataset suggested modestly faster timeframes for non-commercial simple NCE development, taking 9.67 years vs. 10.85 years in the P2I model (Figure 4).

Figure 4. Development times – collected data vs. P2I model (NCE-simple).

The figure compares the development times collected from the organizations in the study and the averages in the P2I Model for the archetype “new chemical entities – simple”. The comparison is made for different stages of development preclinical, phase 1, phase 2 and phase 3 and total costs. Data shows shorter times for collected data at preclinical, phase 2 and total, and longer times at phase 1 and phase 3. Collected data for preclinical is based on 2 data points (n=2), for phase 1 n=3, for phase 2 n=3 and for phase 3 n=2.

For complex NCEs, non-commercial pre-clinical testing was much shorter (1.00 vs 2.87 years in P2I), phase 1 testing slightly shorter (1.67 vs 1.93 years in P2I), phase 2 longer (4.25 vs 3.51 years in P2I), and phase 3 longer (4.0 vs 2.8 years in P2I) (Figure 5). Overall, non-commercial development time was nearly identical, at 10.92 compared to 11.11 years for the P2I model. Possible explanations for these differences could be a result of the ratio of phase 2a to phase 2b tests included in phase 2. The P2I model does not suggest what proportion of its phase 2 tests are 2a compared to 2b, but for our data set, there were many phase 2b tests, which may have increased the amount of time in this phase. Pre-clinical time may have been shorter due to our decision to divide the time in this stage of development among multiple candidates, as that is how some data was shared with us. It is unclear how many preclinical studies in the P2I dataset would have been calculated in this way.

Figure 5. Development times – collected data vs. P2I model (NCE-complex).

The figure compares the development times collected from the organizations in the study and the averages in the P2I Model for the archetype “new chemical entities – complex”. The comparison is made for different stages of development preclinical, phase 1, phase 2 and phase 3 and total costs. Data shows shorter times for collected data at preclinical, phase 1 and total, and longer times at phase 2 and phase 3. Collected data for preclinical is based on 1 data point (n=1), for phase 1 n=6, for phase 2 n=4 and for phase 3 n=1.

The qualitative data identified 12 factors influencing timeframes for non-commercial R&D (Table 2). As with costs, the identified factors were categorized by their potential to push timeframes up or down for non-commercial R&D in comparison to commercial R&D. Seven factors were likely to lengthen timeframes for non-commercial R&D, no factors were likely to shorten timeframes and five factors were categorized as indeterminate. The table below presents a summary of the factors influencing timeframes. A longer description of the factors and sample quotes are available in the full research report⁹.

There were a number of factors that would lengthen timeframes for non-commercial R&D vis-à-vis commercial models, or that were indeterminate (Table 2). Notably, in none of the interviews did a respondent argue that non-commercial R&D would move faster. As the qualitative data does not tell us about the magnitude of the effect, no firm conclusions can be drawn on whether non-commercial R&D would take generally the same amount of time or more than commercial R&D.

Attrition rates

The quantitative data on attrition rates was the most difficult to obtain, and there did not appear to be a standard methodology nor practice of calculating such rates within participating organizations. As all non-commercial initiatives in our sample had relatively small portfolios (compared to large commercial firms), attrition rates might not be meaningful. We judged that the data we received could not be aggregated across organizations, nor was it adequate for hypothesis generation. Further research is needed in this area.

Interviewees were also asked about the main factors that drive attrition rates up or down in the different phases of product development. This question generated a wide range of responses, and different organizations took quite different approaches to conceptualizing – let alone calculating – attrition rates. There was also reasonable disagreement as to whether or when a higher attrition rate is undesirable. Some interviewees argued that it is beneficial for an organization to “fail early and fail fast” – that is, to have a high(er) attrition rate in pre-clinical or Phase 1. Too low of an attrition rate could also suggest an organization is not taking enough risk, particularly in the earlier and lower-cost phases of R&D.

The qualitative data identified nine factors influencing attrition rates for non-commercial R&D (Table 3). As with costs and timeframes, the identified factors were categorized as likely to drive attrition rates higher or lower for non-commercial R&D in comparison to commercial R&D. Three factors were identified as pushing attrition rates higher for non-commercial R&D, one factor as pushing attrition rates lower and five factors were categorized as indeterminate. The table below presents a summary of the factors influencing attrition rates. A longer description of the factors and sample quotes are available in the full research report⁹.

There were more factors that would raise attrition rates for non-commercial R&D vis-à-vis commercial models, than would lower them, but most of the factors raised by respondents were indeterminate. As the qualitative data does not tell us about the magnitude of the effect, no firm conclusions can be drawn on whether non-commercial R&D would be characterized by higher, lower or equivalent attrition rates as commercial R&D.

Underling quantitative data

Zenodo: Quantitative data: costs and timeframes_ non-commercial pharmaceutical R&D. https://doi.org/10.5281/zenodo.4519709¹².

This project contains the following underlying data:

Underlying qualitative data

The qualitative data is confidential to protect participant confidentiality as required by the Ethics Review Committee given the small number of organizations active in the field. Selected quotes from the interviews are available in the full research report at the Graduate Institute Institutional Repository at https://repository.graduateinstitute.ch/record/298834⁹.

Extended data

The Graduate Institute Geneva Institutional Repository: Do costs, timeframes and attrition rates differ between non-commercial and commercial biomedical R&D ? A study of neglected diseases R&D and the P2I model. https://repository.graduateinstitute.ch/record/298834⁹.

This report contains the following extended data:

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).