Pipeline analysis of a vaccine candidate portfolio for diseases of poverty using the Portfolio-To-Impact modelling tool

Introduction

The Special Programme for Research and Training in Tropical Diseases (TDR) recently developed a new modelling tool, called Portfolio-to-Impact (P2I), that allows users to model the impact of different research portfolios¹. The P2I tool can be used to estimate the costs of moving a portfolio of candidate products for poverty-related and neglected diseases (PRNDs) through the pipeline and the likely launches that would result. It can also help to identify potential funding bottlenecks and operational challenges. The modelling tool, which is deterministic, uses Excel based software to estimate the “minimum funding needs to accelerate health product development from late stage preclinical study to phase III clinical trials” and to “visualize potential product launches over time¹.”

As a financial forecasting tool that estimates the funding needs for pharmaceutical product development, the tool and its outputs are of value to funders of product development, product development partnerships, and other stakeholders involved in research and development (R&D) policy. Terry and colleagues, who developed the P2I model, note that “its real utility lies in its predictive value for modelling the impact of different funding strategies at the portfolio level”¹.

To the best of our knowledge, the first published use of the tool was by Young and colleagues². These researchers first conducted a pipeline portfolio review to identify current candidates in the pipeline for 35 PRNDs. They then used the P2I tool to estimate (a) the costs to move these candidates through the pipeline, (b) the likely launches, and (c) the highly needed products that would still be “missing” at the end. As of August 31, 2017, they found 685 PRND product candidates, of which 538 candidates met inclusion criteria for input into the model. Their modelling estimated that it would cost about US$ 16.3 billion (range $13.4–19.8B) to move these candidates along the pipeline, resulting in about 128 (89–160) expected product launches. The study found that “there would be few launches of complex new chemical entities; launches of highly efficacious HIV, tuberculosis, or malaria vaccines would be unlikely².”

The European Vaccine Initiative (EVI), established in 1998 as the European Malaria Vaccine Initiative (EMVI), is a not-for-profit organization that supports the development of effective, affordable and accessible vaccines against diseases of poverty and emerging infectious diseases. To achieve this goal, EVI supports translational vaccine research and development (R&D) spanning from preclinical development through to the establishment of a clinical proof of concept. To date, EVI has supported the development of about 40 vaccine candidates through to early- and mid-stages of clinical development. Initially focussing only on malaria vaccines, in 2009 in the context of a strategic revision EVI broadened its scope and since has built a vaccine portfolio that addresses critical challenges and opportunities in vaccine R&D for a variety of diseases of poverty and emerging infectious diseases.

Currently EVI´s vaccine portfolio comprises around 20 vaccine candidates at different stages of development between late preclinical and mid-stage clinical development. In order to estimate future financing needs required for delivering the EVI portfolio and the potential public health impact of product launches, we conducted an analysis of EVI´s vaccine candidate portfolio using the P2I tool.

Together with similar pipeline portfolio reviews using the P2I tool that are currently being conducted by other product development partnerships (PDPs), the results will inform product developers as well as funders and policy makers regarding future funding needs. The results may also guide future investment priorities to maximise the chances of developing products for diseases that are missing urgently needed health products.

We used the P2I tool because, to the best of our knowledge, it is the first publicly available comprehensive portfolio model that includes data on cost, success rate, and cycle time per phase for various product types based on data from a very large number of previous product development candidates (over 25,000)¹. The P2I tool is thus complementary to other available tools that can help guide prioritization in vaccine development, such as the multi-stakeholder Vaccine Innovation Prioritization Strategy¹ and Total Systems Effectiveness Framework².

Methods

In this section, we begin by summarizing how the Microsoft Excel-based P2I tool was developed, which phases of product development are included in the tool, and which costs are excluded. After this summary, we then describe the four key steps in our analysis of EVI’s vaccine portfolio, which are summarized in Figure 1.

Figure 1. Four key phases in the analysis.

A summary of how the P2I tool was developed, and which phases and costs are included

A detailed research paper describing the development of the first version of the P2I tool itself (P2I version 1, or P2I v1) has been previously published¹. The development of a second version of the P2I tool, called P2I version 2 (P2I v.2), has also been previously described². As summarized below, the model is based on assumptions for costs per phase, attrition rates (probability of success) per phase, and cycle times per phase for four development phases (preclinical to phase III, see Figure 2) for a number of different kinds of medical products, called archetypes.

Figure 2. Development phases included in the P2I model.

ND: investigational new drug application; NDA: new drug application; m: million; PK: pharmacokinetics (absorption, distribution, metabolism, excretion); PD: pharmacodynamics; pop.: population; P2I: Portfolio-to-Impact (adapted from Health Product Research and Development Fund: A Proposal for Financing and Operation³).

The P2I v.1 model has 11 archetypes: simple or complex vaccines; simple, innovative, or complex new chemical entities (NCEs); simple or complex repurposed drugs; simple or complex diagnostics; simple or complex biologics; and two diagnostic archetypes, assay development or simple technical platform development. Descriptions and examples of each are described elsewhere¹. For each of these 11 different archetypes, the model has in-built assumptions on costs, attrition rates, and cycle time per phase.

One valuable feature of the model is that it is highly adaptable, users can input additional archetypes into the Excel tool. In the P2I v.2 model, additional archetypes include “unprecedented vaccines” (for candidate vaccines for HIV, TB, and malaria to, which are “considered as unprecedented as current platforms have not led to suitable vaccines”²) and vector control products.

As described elsewhere¹, assumptions on development costs at each phase of product development for the 11 archetypes included in the P2I.v1 model were initially based on an analysis of clinical trial costs from Parexel’s R&D cost sourcebook. The assumptions on attrition rates and cycle times at each phase were initially based on the historical attrition rates and cycle times of more than 25,000 development candidates. All of these assumptions were further refined and validated based on (i) academic literature, (ii) industry publications and databases, and (iii) 133 stakeholder interviews with a wide variety of product development partnerships (PDPs), pharmaceutical companies, and major funders of global health R&D. As described elsewhere², additional sources of assumptions for the new archetypes in P2I.v2 were the McKinsey Risk-Adjusted Portfolio (RAP) Model and clinical trial data shared by the Bill & Melinda Gates Foundation.

As described below, the three archetypes of relevance for our analysis of the EVI portfolio were simple, complex, and unprecedented vaccines (reference 2 describes these three different archetypes in more detail). After classifying each EVI candidate into its archetype and phase, we then ran the model. There are two main model outputs. The first is “launches”; in this paper, the term launch refers to a candidate making it through phase III and thus being ready for the next steps, e.g. the regulatory and manufacturing steps. The second is the total costs to move all candidates through the pipeline from their current phase from now to 2031 (the model also gives a breakdown of these total costs into annual costs by year, from 2019 to 2031).

The model includes only advanced preclinical to phase III, and thus the cost estimates are an under-estimate of the full costs of product development. In particular, the model excludes all costs related to basic research through lead optimization; chemistry, manufacturing, and controls; good manufacturing practice; manufacturing build up and scale-up costs; regulatory or registration fees (post-phase III); and all post-market commitments (e.g., phase IV pharmacovigilance studies).

Results

Conclusions and discussion

The mission of EVI is to accelerate the development of vaccines for diseases of poverty and emerging infectious diseases. Compared to other PDPs with a narrower focus, for example on a single disease, EVI has a broader scope and consequently a more heterogeneous portfolio, currently comprising 18 active vaccine candidates covering five different diseases/pathogens (malaria, including placental malaria, leishmaniasis, shigella/ETEC, Nipah and Zika viruses). An analysis of EVI´s current vaccine portfolio, providing estimations for future vaccine development costs and expected product launches, was considered important to inform future decision-making and priority setting at EVI, as well as to provide valuable information to global health funders and policy makers. At EVI, decision regarding which vaccine candidates to include into the organisation’s portfolio, and which ones to move forward, are based on a rigorous selection of candidates made by the EVI Board, based on input from an independent scientific advisory committee. For selecting vaccine candidates and advancing their development, EVI employs a portfolio management approach that has defined gating criteria (Go/No-Go criteria), ensuring that only the best leads are fed in and candidates that do not meet the criteria set are weeded out early on, thereby balancing the number of projects supported with available resources⁴. Results delivered by the portfolio analysis using the P2I tool, in particular the total estimated costs and expected success rates, therefore provide valuable information that informs this selection and decision-making process.

Overall, using the pre-defined assumptions established in the P2I tool, our modelling resulted in a total estimated cost of about US $470 million for moving all 18 candidates included in the analysis along the pipeline until launch. Of this total amount, just over one third (35%) of the expected costs would be incurred by the development of the eight vaccine candidates for malaria (excluding those for placental malaria). Development of the candidates for placental malaria, Nipah virus, and Zika virus would account for about 16% each of the total costs. The remaining financing would be required for the development of candidates for leishmaniasis and shigellosis (10% and 7% of total costs, respectively).

The re-run (second run) of the P2I model using modified assumptions for phase costs and length based on EVI’s internal data increased the projected portfolio costs by US $46 million up to a total cost of US$ 517 million for all 18 candidates. The main driver of this change in the estimated cost is the increase in expected costs for the unprecedented vaccine candidates (i.e. the eight malaria vaccine candidates, five placental malaria vaccine candidates and two leishmaniasis vaccine candidates). Results for the vaccine candidates for Nipah virus, Zika virus, and shigellosis, ETEC were not affected by the change in these parameters.

The costs that we estimated using the P2I tool are likely to be an underestimate of the true costs. Vaccine development is a reiterative process, meaning that many steps, such as clinical trial phases, will be conducted several times. This process is in contrast to the rationale of the P2I tool, which assumes a straightforward development of product candidates without the reiteration of any particular development steps. For example, very often several phase I trials for a particular antigen are conducted, i.e. multiple trials in which different formulations, different routes of administration or different technology platforms for antigen presentation are being tested and compared. Also, very often several phase I clinical trials are conducted in which a vaccine´s safety is assessed in various age groups in an age-deescalating manner (i.e. the vaccine is tested consecutively in volunteer groups with decreasing age). It is therefore rather unlikely that any vaccine candidate would immediately advance to a phase II clinical trial after the conduct of a single phase I clinical trial only. Consequently, the total costs for the development of the different vaccine candidates in EVI´s portfolio are likely to be significantly higher than those estimated by the P2I tool. In addition, as several phase I or phase II clinical trials are likely to be conducted for the same vaccine (as explained above), due to such “reiterated” phases the overall timelines to reach the market are expected to be longer.

With regards to the estimated future launches, for all 18 candidates under development, the P2I model estimates that there would be 0.69 expected launches across all six diseases combined. In the re-run of the model using EVI‘s own internal parameters, thanks to higher success rates at EVI as compared to the P2I tool´s predefined parameters, the launch probability increased for malaria, placental malaria, and leishmaniasis vaccines (from 0.098 to 0.11, from 0.05 to 0.07, and from 0.024 to 0.03, respectively). Directly related to this increased launch probability, the total estimated costs for moving these vaccine candidates through the pipeline increased accordingly. This difference between the estimated future product launches emphasizes that, in order to increase the chances of ultimate success with this kind of product development, it is important to make the product development process technically as efficient as possible, i.e. reducing the attrition rates as much as possible. If attrition rates during product development cannot be reduced, the only other chance of achieving ultimate product launches is to boost the overall number of product candidates in the pipeline, obviously in the end resulting in higher total costs linked to the launch of a single product due to the high costs linked to failed product candidates.

However, rather than looking at the isolated results on likely launches from the analysis of a single organization’s portfolio, as has been done in this particular study, more meaningful results will be obtained from such simulations using the P2I tool by conducting a much wider portfolio analysis in which the launch estimation results of the entire global vaccine candidate portfolio are estimated in an integrated, combined manner. Only this kind of “full global portfolio” study can provide a reliable prediction of the product success rates on a global level for the next few decades.

The P2I tool has a number of other limitations, which were described in detail by Young et al.² We highlight six specific limitations. First, as a static, deterministic model, it does not take into account possible improvements in product development techniques over time (e.g. R&D efficiencies that reduce costs). Similarly, as a static model, it does not take into account the possibility that candidates may sometimes have to go “backwards” to an earlier phase. For example, once a candidate enters into phase I there may be bottlenecks that require that the candidate return to a preclinical evaluation stage (e.g., if different formulations need to be re-evaluated or alternate adjuvants need to be tested).

Second, the model’s assumptions for costs, attrition rates, and cycle times for phase are based on product development data from across multiple diseases (including non-communicable diseases—the model does not reflect possible differences in R&D parameters between different diseases). Although the assumptions were based on a very large number of data points (from 25,000 development candidates) and validated with experts, it is unclear how many of these data points came from vaccine development for neglected and emerging infectious diseases. Thus there is some uncertainty as to how accurate the assumptions are for the costs, attrition rates, and cycle times per phase for the three vaccine archetypes used in our study.

Third, the model does not include all phases of development (e.g. it excludes drug discovery, basic research, and regulatory review). The exclusion of phase IV studies, also known as post-marketing surveillance, is a major limitation—determining long-term safety and effectiveness is critical, yet it can be a lengthy, expensive process. We acknowledge that using the P2I tool, which only includes advanced pre-clinical to phase III, will always lead to an under-estimate of the total costs, since the costs of early pre-clinical research and post-phase III research can both be substantial. For example, based on data from a sample of 106 NCEs, DiMasi et al.⁵ estimate that developing an NCE to the point of marketing approval costs $2.6 billion; this includes $1.2 billion in “time costs” (the expected returns that private investors forgo while a drug is in development)⁵. Of this $2.6 billion, $1.1 billion is in pre-clinical development costs and $1.5 billion in clinical development costs. Previous research has suggested that the cost of regulatory approval stage may represent up to 5.7% of the total R&D cost⁶.

By only including advanced pre-clinical to phase III, the model also provides no insight into the costs and complexity of the array of activities that need to happen after phase III for a new product to have a public health impact. The phase after phase III is often considered as a “valley of death” for product development—a product may pass successfully through phase III but then there may be no concerted, strategic plan for large-scale manufacturing or scale-up. Demand forecasting, developing a long-term business case, understanding the public health value of new products, and analysing the delivery system and scale-up approach are all critical components in determining the ultimate public health utility of a new health technology. We have previously noted that the P2I model “is “agnostic” when it comes to the public health value of the estimated launches—it cannot judge their clinical utility”².

Fourth, accurate classification of candidates into their archetypes can be challenging. As we previously noted, “the P2I v.2 model requires users to classify every candidate into an archetype, but categorizing candidates based on the archetype definitions was challenging—especially determining a candidate’s complexity. It will be helpful for future iterations of the model to include more fine-grained, detailed descriptions”².

Fifth, the model assumes that the costs, attrition rates, and cycle times per phase for vaccine development would be the same regardless of the setting where the study is done. Yet in reality, it is likely that these parameters would be different if a study were conducted in a high-, middle-, or low-income country. It would be helpful for future iterations of the P2I tool to incorporate these differences across study settings.

Sixth, the model also assumes that the costs of vaccine development per phase do not vary and are predictable. Yet there can be substantial variation and unpredictability in items such as the cost of manufacturing the product candidates and adjuvants, or the maintenance and quality control of clinical trial sites.

In this study, several adaptations to the P2I tool initially were considered with the aim of improving the tool´s usefulness and reliability. First, we considered making adaptations of the assumptions for success rates, costs and cycle times based on EVI’s longstanding experience with conducting studies in resource-limited, low-income settings. Second, we considered making adaptations for the same parameters based on (a) the fast-track clinical development strategy often used by EVI (consisting of a strategy in which the first-in-human evaluation involves a staggered multi-centre phase Ia/b clinical trial resulting in shorter timelines^7,8), and (b) the inclusion of accelerated clinical testing based on controlled human infection models available, for example, for malaria^9,10. Third, in the original proposal we considered including adaptations of the assumptions for costs and cycle times for vaccines that might be eligible for accelerated approval by regulatory authorities, for example the “Expedited Programs for Serious Conditions––Drugs and Biologics” from the US Food and Drug Administration¹¹.

In the end, we were able to do one re-run of the P2I model using EVI’s own parameters for success rates and cycle times for phase I clinical trials for unprecedented vaccines (70% and 17.4 months, respectively, compared to 50% and 24 months defined in the original P2I tool for unprecedented vaccines). However, although to date EVI has been involved in the conduct of over 30 clinical trials, data from only a limited number of studies could be used in the analyses conducted. When it came to estimating clinical trial costs, for example, for several trials it was not possible to extract the specific costs for preclinical or clinical trial activities out of the overall study costs.

Concerning the proposed modification of P2I parameters based on accelerated clinical testing using controlled human challenge models and on accelerated approval by regulators, we realized that although these two issues are likely to speed up vaccine development, currently there is not enough evidence/data based on which the P2I parameters could be adapted and analyses be run to assess their impact.

In conclusion, we found that despite the limitations discussed above, the P2I tool was flexible and adaptable enough to be used to study EVI’s portfolio. We believe that the P2I model represents a useful tool to analyze the portfolio of global health products under development. Findings from the analysis of the overall EVI vaccine portfolio using the P2I tool will be taken into consideration in the next revision of the EVI Strategic Plan, and estimations for individual vaccine candidates will inform decisions regarding whether or not to continue with the development of individual vaccine candidates once they reach major milestone or stage gating criteria. We expect that studies like ours will inform future updates of the model that will further increase its value for product developers, R&D funders, and decision makers.

Data availability

All data underlying the results are available as part of the article and no additional source data are required.

The particular vaccine candidates included in this study have been anonymized for intellectual property reasons.