Analysis of diagnostic product portfolios using the Portfolio- To-Impact modelling tool [version 1; peer review: awaiting peer review]

Background: The Portfolio-To-Impact version 2 (P2I v.2) financial forecasting tool estimates funding requirements for development of portfolios of candidate health products (drugs, biologics, vaccines or diagnostics). The assumptions and archetypes relating to diagnostics in P2I v.2 are based on limited data and may not accurately describe research and development costs, timelines and probability of success. This study aimed to revise the P2I v.2 tool by modifying the diagnostic assumptions to improve accuracy of predictions for diagnostic portfolios. Methods: Data from expert interviews and historical information on development of 26 existing diagnostics were used to determine approximate research and development costs, timelines and probability of success for development of diagnostics, and to revise diagnostic archetypes and development phases. To compare the revised tool with P2I v.2, data on 27 candidates from the Foundation for Innovative New Diagnostics (FIND) tuberculosis and pandemic preparedness portfolios were input into both versions. Results: The number of diagnostic archetypes increased from two in P2I v.2 to three in the revised tool. Total estimated costs to move the 27 candidates along the pipeline to launch were US$641.62 million with P2I v.2 and US$274.00 million with the revised model. The number of expected launches was 21.65 over five years with P2I v.2 and 11.48 over eight years with the revised model. Development timelines were extended and probability of success was lower with the revised model compared with P2I v.2. Conclusions: Outputs from the revised tool were in line with expert experience, suggesting that the proposed revisions improve the accuracy of the tool for estimating research and development costs, timelines and probability of success relating to diagnostic portfolios. Additional improvements to the tool could include further refinement Open Peer Review Reviewer Status AWAITING PEER REVIEW Any reports and responses or comments on the article can be found at the end of the article. Page 1 of 16 F1000Research 2021, 10:116 Last updated: 04 AUG 2021


Introduction
The Portfolio-To-Impact (P2I) model is a financial forecasting tool developed by the Special Programme for Research and Training in Tropical Diseases (TDR) 1 . The model enables users to estimate funding requirements for development of a portfolio of candidate health products, from late stage preclinical studies to phase III clinical trials, and estimates the number of potential product launches over time, as well as the probability of success of health products. The P2I v.2 model allows a range of different types of medical products to be assessed, including drugs, biologics, vaccines, and diagnostics. These are described as 'archetypes' and have further subdivisions within the model depending on the complexity of the product 1,2 . The model is intended to be continually refined and improved, and users are encouraged to provide inputs to update key parameters and assumptions. In line with this, a second iteration of the model (P2I v.2) that includes additional archetypes and has modified cost assumptions was subsequently developed and used to estimate research and development costs associated with a pipeline of candidates for 35 neglected diseases 2 . The model has also been used to assess the vaccine portfolio of the European Vaccine Initiative (EVI), which includes vaccine candidates for various diseases of poverty and emerging infectious diseases 3 .
The development pathway for diagnostics differs considerably from that of pharmaceuticals and vaccines, as they do not follow the traditional pathway from preclinical to phase III studies. Rather, after initial concept and research and feasibility planning, diagnostics move from the design and development phase directly to clinical validation, approval and launch 4,5 . However, there is a lack of academic literature available on development characteristics for diagnostics, thus the archetypes and assumptions used within the P2I v.2 model were based solely on interviews with diagnostics developers and other diagnostic experts. Based on the experience of the Foundation for Innovative New Diagnostics (FIND), a product development partnership (PDP) focusing on diagnostics for diseases of poverty 6 , the diagnostic assumptions in the P2I v.2 model tend to underestimate the research and development costs and timelines for diagnostic discovery and early stage development. Additionally, due to the complex nature of the diagnostic landscape, the archetypes and assumptions in P2I v.2 may not accurately describe all diagnostics. A range of factors can impact diagnostic research and development costs, timelines and probability of success, including the underlying technology of an assay, the maturity of the manufacturing company and their expertise and experience (in clinical, regulatory, quality management and infectious disease areas, as well as in assay development), the disease area, ease of obtaining samples for validation, and the Stringent Regulatory Authority (SRA) being pursued.
FIND has a large diagnostics pipeline across six disease areas (antimicrobial resistance, hepatitis C and human immunodeficiency virus [HIV], malaria and fever, neglected tropical diseases, tuberculosis [TB] and pandemic preparedness), and plans to use the P2I v.2 model as a complementary tool to existing portfolio management methods for prospective scenario planning, to inform funding decisions, refine technology assessments, and predict product launches and estimated research and development costs. The objective of this study was to propose revisions to the P2I v.2 diagnostic archetypes and assumptions, to improve accuracy of predictions for diagnostic portfolios, and to compare outputs of the revised model with the original P2I v.2 tool.

Revision of the P2I v.2 model
This work was carried out between April and December 2019. Data to inform the refinement of the P2I v.2 tool were collected through two methods: a) expert interviews and b) collection of historical data on development of existing diagnostics. The expert interviews were performed by the FIND project team. Seven experts with experience in diagnostic development in both high-income countries (HICs) and in low-and middle-income countries (LMICs) were asked if they were willing to be interviewed; all experts were either employees of FIND, consultants, or FIND external partners (Table 1). We obtained consent orally before the beginning of the interviews. Interviewees were provided with background information about the P2I v.2 tool, and were asked to give feedback on the tool's diagnostic assumptions in relation to at least one product that they had supported through development. Data ranges for research and development costs, development timelines, and probability of success per product development phase were collected.
The historical data for existing diagnostics (ideally commercially available tests) were retrieved from the database housed by Halteres Associates (San Ramon, CA, USA), a consulting company with extensive experience in diagnostic development.
A set of 26 diagnostic products for which sufficient data were available were selected from a broad group of product types and product development stages (Table 2). Research and development costs were approximate, and were based either on information from the companies themselves or from persons with knowledge of the companies; additionally, a number of the products were not yet marketed, thus final numbers for research and development costs and timelines were not available. As such, ranges for research and development costs and timelines were used except where otherwise stated.
Diagnostic archetypes were determined by identifying clusters among the 26 products with similar estimates for research and development costs, timelines and probability of success. This was done by calculating a composite 'level of risk' score that comprised factors thought to have most impact on these three assumptions. Initially, the following factors were included: target product profile, biomarker status, assay technology, instrumentation, per assay development costs, system development costs, regulatory requirements, preclinical/clinical studies, team competence, financial strength, payment/reimbursement model, product line extension, voice of customer, partner options, market development and commercialization plan/budget, entity stability, number of tests needed, and market demand/development. From this list, 15 factors were extracted that were believed to have the most impact (resulting in the removal of entity stability, number of tests needed, and market demand/development from the list), and the list was then further reduced for practicality of scoring, to obtain a set of eight criteria. Each of the 26 products was then assessed using the eight criteria, scored on a scale of 0 to 3 (Table 3). Approximate groupings were then determined based on the level of risk score.

Comparison of P2I v.2 and revised models
In order to compare the two models, data on the products from two of FIND's portfolios, TB and pandemic preparedness, were input into the P2I v.2 model, using the original diagnostic assumptions and archetypes (Table 4 and Table 5) and using the revised diagnostic assumptions and archetypes collected from the expert interviews and the historical database of 26 existing diagnostics. The research and development cost and timelines to move the candidates from the two FIND portfolios through the pipeline, and to estimate the likely number of launches per disease, was estimated. The model assumed a 2019 start, with all diagnostic candidates inputted based on their development phase at the time of review.

Potential impact
It can be challenging to compare diagnostic opportunities when they have comparable levels of risk. In these cases, the potential public health impact of a diagnostic can be a valuable differentiating factor. Factors that can influence the potential public health impact of a diagnostic product were identified from the historical database of 26 diagnostic products. Ten factors were identified; these were: increase in number of patients/year receiving appropriate treatment, cost per result, improvement in guideline adherence, time to intervention, improved access to critical data, support for diagnostic, surveillance or outbreak programmes (as intended), impact metrics, change in disease incidence or prevalence, reduction in loss to follow-up, and change in morbidity or mortality. The list was reduced to seven factors with the most influence, and a potential impact score for ranking was developed, scored on a scale of 0 to 3 ( Table 6). Each of the 26 products from the historical database was scored according to this system.

Results
Refining the P2I v.2 model Diagnostic archetypes. To determine the optimal diagnostic archetypes, clusters of diagnostic products that depicted similar estimates for research and development cost, timelines and probability of success were identified from the historical dataset of 26 diagnostic products ( Figure 1). Approximate groupings of products that aligned with three archetypes emerged: 1. "New or existing entity, discovery" archetype: a research or commercial stage entity in the discovery phase of a new technology 2. "Existing entity, new product/technology" archetype: an established commercial company that is creating a new product that is not a product line extension 3. "Existing entity, product extension" archetype: an established commercial company that is creating a new product as an extension of an existing product line Diagnostic assumptions. The expert interviewees proposed three development phases for diagnostics: "discovery", "design and development" and "clinical validation and launch readiness". These were fitted to the phases in the P2I v.2 model 2 by encompassing both "concept and research" and "feasibility and planning" from the original P2I v.2 tool into the "discovery" phase. The remaining two development phases were the same as those in the P2I v.2 tool ("design and development" and "clinical validation and launch readiness"). A fourth phase, "commercialization", was recommended by the interviewees, as this can be costly for diagnostics. Data for this phase are presented, but it was not included in the comparison analysis with the P2I v.2 tool.
Data from the expert interviews and the historical database of 26 existing diagnostics were used to identify the revised research and development cost, timeline and probability of success assumptions per product phase. Where the database NAT PC 5-10 3-5 5-10 2-5 10-25 2-3 5-10

Score Definition
Comments TPP 0 The TPP is missing or incomplete A complete, properly prepared and vetted TPP is essential to ensure the unmet needs and intended use for the product are properly understood and that the specified product requirements will meet the unmet needs

Comments
Financial strength 0 The entity lacks a sustainable financial model A key indicator of sustainable success is the ability to maintain sufficient gross and operating margins to support continued product/market and commercial development programmes. A financial plan alone is generally not sufficient if the entity has not also prepared an associated strategic operating plan or business model (ideally both) 1 The entity has a financial model pertinent to the product in development but has not completed a strategic operating plan or business plan that should lead to a sustainable entity 2 The entity has both a financial model pertinent to the product in development and a strategic operating plan or business model that should lead to sustainability Voice of customer 0 The entity has no expertise with the intended customer base and/or in the required (initial) market channels; they have limited understanding of use cases or work flows Access to internal or external resources to properly inform the TPP requirements, the market development plans, the competition, the key stakeholders and the customer access and support strategies is essential for success The entity has been primarily focused on technology development and appears to lack sufficient understanding of, plans for and/or sufficient budget for market development and commercialization activities Once technology and product development activities have been successfully navigated, many companies face a second chasm related to market development and commercialization. Often these activities are too challenging, specialized and/or resource demanding for an individual entity to manage on their own. Recognizing the necessary activities required for market development and sustainable commercialization takes knowledge, planning and networking to be successful. An entity that scores high here will have thought through these challenges 1 The entity has prepared an initial plan and budget for market development and commercialization activities, but significant gaps remain and/or budget is insufficient 2 The entity has developed a comprehensive set of market development and commercialization plans with appropriate budget TPP, target product profile.   Having a properly designed TPP is essential for achieving intended impact and long-term sustainability (but not necessarily sufficient). It is incumbent on the entity to produce data showing that it can address the unmet need in the target population with accepted and understood strong impact measures 1 The entity has reasonably described the impact measures necessary to support the implementation of their product and has a reasonable plan for generating and reporting the required metrics 2 The entity has already generated strong data in support of the desired impact measures and has reasonable plans for completing any remaining gaps Change in disease incidence or prevalence 0 <5% change in target disease incidence or prevalence is anticipated following introduction of the product or technology All products should have a demonstratable impact, either directly or indirectly, on the incidence or prevalence of disease. This may be one of the impact measures highlighted in the impact metrics criterion, but is sufficiently important to highlight separately 1 5-25% change in target disease incidence or prevalence is anticipated following introduction of the product or technology 2 >25% change in target disease incidence or prevalence is anticipated following introduction of the product or technology Change in morbidity or mortality 0 <5% change in target disease morbidity or mortality per year is anticipated following introduction of the product or technology All products should have a demonstratable impact on the morbidity (e.g. River Blindness) or mortality (e.g. Cryptococcal meningitis in Advanced HIV Disease) of disease. This may be one of the impact measures highlighted in the impact metrics criterion, but is sufficiently important to highlight separately 1 5-25% change in target disease morbidity or mortality per year is anticipated following introduction of the product tor technology 2 >25% change in target disease morbidity or mortality per year is anticipated following introduction of the product tor technology  included data ranges, averages were calculated for the purposes of inputting into the P2I model. Data were weighted at 0.67 for "concept and research" and 0.33 for "feasibility and planning", based on expert experience suggesting that greater time and cost is placed at the concept stage. Table 7 lists the revised diagnostic assumptions. The average costs per phase were higher in the updated assumptions compared with the original P2I v.2 tool assumptions ("discovery": US$1.1-24.6 million versus 3.0 million (combined "concept and research" and "feasibility and planning"); "clinical validation and launch readiness": US$6.9-10.0 million versus 3.5 million), with the exception of the cost of the "design and development" phase, which was significantly lower than the initial assumptions (US$3.3-6.9 million versus US$2.0 to 100.0 million). As it was not possible to calculate specific probability of success scores for each archetype with the available data, an average across all archetypes was used for each phase. The probability of success was significantly lower in the updated diagnostic assumptions compared with the initial assumptions.
Comparison of P2I v.2 and revised models FIND's current TB and pandemic preparedness portfolios contained a total of 27 diagnostic candidates under development (Table 8). This included three products for Lassa fever, two on multi-disease fever-causing pathogens, one on yellow fever, and 12 products for TB. Using the P2I v.2 model, 12 products were classified as the "simple technical platform development" archetype and 15 were categorized as the "assay development" archetype. All four development phases were covered. Using the revised model, nine products were classified as the "existing entity, new product/technology" archetype; 10 as the "new or existing entity, discovery" archetype; and eight as the "existing entity, product extension" archetype.
Using the P2I v.2 model, the total estimated cost to move the 27 candidates along the pipeline to launch was US$641.62 million, with the majority of costs (77%) incurred by the development of the TB diagnostic candidates, followed by the multi-disease fever candidates (20%) (Figure 2A). Using the revised model, total estimated costs to move the 27 candidates along the pipeline to expected launch were around US$274.00 million, substantially lower than the cost estimate from the initial model. However, the costs of developing the yellow fever and Lassa fever diagnostics were higher compared with the initial model due to increased costs during early development ( Figure 2B).
The total expected number of launches across all 27 candidates using the P2I v.2 model was 21.65, with the majority of launches for the TB candidates (17.82), followed by the Lassa fever candidates (2.42) ( Figure 3A). Using the revised model, the total launch probability for all 27 candidates was reduced

Cost per phase (US$ million)
New or existing entity, discovery Phase based on recommendation from experts and combines "concept and research" and "feasibility and planning" in P2I v2 model. b Total up to "clinical validation and launch readiness". c Probability of success in "concept and research": 30%; probability of success in "feasibility and planning": 60%.  to 11.48 ( Figure 3B), a little over 50% lower than the P2I v.2 model projected probability, most likely due to the lower probability of success rates.

Disease
The P2I v.2 model predicted that the majority of costs would fall in the first three years of development; development time for all diagnostic candidates was predicted to be completed by 2024 (Figures 4A and 4B). Using the revised model, development timelines were extended, with development time for all diagnostic candidates predicted to be completed by 2028 ( Figures 4C and 4D).

Potential impact criteria
Level of risk scores versus potential impact scores for each product are shown in Figure 5. A number of products had the same level of risk score but substantially different potential impact scores.

Discussion
The P2I v.2 tool predicted that we would require US$642 million to move the 27 products from the FIND TB and pandemic preparedness portfolios forward through the pipeline, with a launch probability of 21.65 launches over five years. Using our revised P2I model with updated diagnostic archetypes and assumptions, costs were reduced to US$274 million, with a launch probability of 11.48 launches over eight years. While the diagnostic archetypes and assumptions in P2I v.2 were solely based on interview feedback, our revised archetypes and assumptions were partially based on an analysis of the development history for 26 existing diagnostics, thus our revised tool  incorporated actual data as well as expert feedback. The revised outputs are more in line with FIND's experience compared with the outputs of the P2I v.2 model, especially for early stage products. The considerable differences between the outputs of the two models suggest that the current P2I v.2 tool may not accurately capture the research and development costs, timelines, and probability of success for diagnostics.
We found that the previous P2I diagnostic archetypes, "assay development" and "simple technical platform development", were too confining given the breadth of assay and platform technologies used for the 26 products we analysed from the historical database. For example, some products were being developed on existing platforms and others on new platforms, but both would have been categorized under the "assay development" archetype even though the costs and development times differed significantly. While some products that improve upon current technologies with simple modifications, such as readers for lateral flow assays, fit the "simple technical platform" archetype, others were new systems with novel  disposables, chemistry and instruments, which were too complex in terms of the technology used to fit under this heading. However, while the updated diagnostic archetypes may better capture the complexity of diagnostics product development, they do not necessarily represent the optimal and complete view of the reality. We recommend continued collection of additional data in order to identify more precisely the archetypes and factors that have the most impact on the assumptions examined, and allow more effective prioritization of investment in diagnostic companies and products.
The reduction in cost using the revised model compared with the P2I v.2 tool is most likely due to the reduced cost for clinical validation in the updated assumptions, as well as the lower probability of success rates compared with the original assumptions. Notably, the lower probability of success assumptions are in line with previous work on this subject 7 . As expected, our analysis and the feedback from the expert interviewees showed that developing newer products tends to require more investment and development time compared with menu expansion and product extension. This reflects FIND's experience in diagnostic development for LMIC settings. In addition, there is a significant development burden in the discovery stage, particularly under the "concept and research" phase, as successful products must not only be technically sound, but must reflect their intended use.
Evaluating the public health impact of a product in addition to the probability of success is valuable, as products with a high public health impact but low probability of success may still be worth investing in; although riskier than products with higher success probability, they are likely to improve the health of a greater number of people if successful. We assessed the potential impact of a new diagnostic product at the same time as assessing its level of risk, as a way to prioritize investment in diagnostic products or technologies that, on the surface, display similar cost requirements. We believe that it is important to incorporate this notion into decision-making when estimating funding needs to move candidate health products through the pipeline. Understanding the potential impact of a product can inform decisions on whether to fund or include a high-risk product in a portfolio, as well as supporting advocacy and fundraising efforts.
The interviewees recommended the addition of average research and development costs, timelines and probability of success of commercialization to the model, by inserting a new development phase. Once a product has launched, understanding the costs associated with access and sustainability, especially in LMIC markets, is important, as the probability of success of a new diagnostic test depends not only on the development of the product itself but also on the way it is delivered to the target patient populations. Even when they are available, diagnostic tests are not necessarily affordable, appropriate for use in LMICs, and/or adopted in these settings. To maximize the access to a diagnostic product, a sizeable level of human and financial resources needs to be invested in marketing, sales, manufacturing, distribution and in-country registration, which can lead to higher costs. For more accurate estimates of overall funding needs for diagnostics, future iterations of the tool could incorporate the commercialization phase.
In addition to the limitations of the P2I model already described 1,2 , there are some further limitations specific to our revised model that should be noted. First, the research and development costs and timelines for product development phase used in the revised P2I model correspond to averages of the historical data collected, as we were only able to obtain ranges of data rather than unique data points, from which averages were calculated to allow for inclusion into the tool. The averages may not effectively capture the large differences that were observed between products within the same archetype, which stem from a complex and multifaceted environment of the diagnostics industry. Secondly, like the developers of the original P2I tool, we faced challenges in data collection for diagnostic development, as the information needed was either not publicly available or not systematically recorded. For FIND and other PDPs with interest in diagnostics, we recommend improving organizational processes and systems for tracking, storing and sharing this type of information and/or dedicating human resources, such as a Portfolio Manager, to this task. Similarly, the number of diagnostic products used to generate new archetypes and development assumptions was limited, thus more data are required to further validate the proposed archetypes and assumptions.
In summary, we believe that our proposed revisions to the archetypes and assumptions for diagnostic products of the P2I model improve the accuracy of the tool for estimating costs and timelines relating to diagnostic portfolios. During the preparation of this article, a corrected version of the P2I v.2. was published 8 . We hope the data and recommendations presented here are considered for inclusion in a future iteration of this tool, particularly with regards to the inclusion of commercialization and public health impact, which may be applicable to other health products.

Data availability
All data underlying the results are available as part of the article and no additional source data are required. Feedback from expert interviews is not provided due to data protection/ confidentiality considerations. The diagnostic products used to inform revised assumptions are not ascribed to a specific company due to confidentiality agreements.
The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias • You can publish traditional articles, null/negative results, case reports, data notes and more • The peer review process is transparent and collaborative • Your article is indexed in PubMed after passing peer review • Dedicated customer support at every stage • For pre-submission enquiries, contact research@f1000.com