The application of multi-criteria decision analysis to inform in resource allocation

Abbreviations

HTA, health technology assessment; MCDA, multi-criteria decision analysis; QALY, quality-adjusted life-year.

Introduction

The increasing demand on healthcare resources has created the need to minimize costs, resulting in more rigorous pricing and reimbursement pathways in most of Europe, potentially causing delays for patients to receive valuable treatments¹. As the pressure on health technology assessment (HTA) bodies increases, so does the need to demonstrate the cost-effectiveness of orphan drugs. There is a perception among payers that orphan drugs are over-priced and therefore not cost-effective². Patients have been refused access to drugs based on cost-effectiveness perceptions that might possibly be linked to the application of cost-effectiveness thresholds³. The pharmaceutical media go to great lengths to highlight that orphan drugs are expensive interventions with costs being raised by the involvement of “big pharma”^4,5. Therefore, orphan drugs are increasingly under scrutiny by health authorities as part of cost-containment initiatives and are frequently only reimbursed when negotiated under a patient access scheme^6,7. The HTA processes are aimed at bridging the gap between evidence and healthcare policy and reimbursement decisions⁸. The current HTA systems adopted in many European countries may facilitate financial decisions that largely disregard disease impact on patients and a host of other disease-related attributes, and potentially ignore some unique features of innovative orphan drugs, thereby delaying or denying patients access to much needed treatments^9–11. While the cost and budget impact of orphan drugs are important in relation to affordability, cost should be considered in relation to a broader range of drug- and disease-related attributes that are not necessarily considered in the usual HTA processes for most non-orphan drugs. This is increasingly problematic for markets linking cost-effectiveness analysis to reimbursement decisions as the cost per quality-adjusted life-year (QALY) approach is not necessarily sensitive enough to capture the broader attributes of the therapy such as unmet need, disease severity, patient and societal preferences and other disease-related elements, possibly including disease rarity¹².

Disease rarity implies that the level of clinical evidence and the uncertainty surrounding the treatment effectiveness generated in clinical trials is likely to differ from that of conventional diseases due to several key factors¹³. In particular, small heterogenous populations increase the level of uncertainty of the outcomes^14,15. In some rare diseases, there is limited natural history data and often a lack of consensus on the choice of treatment comparator and clinical endpoints^16,17. Ethical constraints regarding placebo treatments in the control group when no other treatment options are available may arise^15,18. Consequently, these factors might preclude a thorough analysis. Therefore, the likelihood of orphan drugs achieving the expected robust cost per QALY levels is limited¹⁹. Thus, the cost per QALY approach might not be optimal to assess the real value and benefits of orphan drugs and should, at least, not be the dominant tool to establish reimbursement.

In response to criticism regarding the variation in reimbursement decisions and potential inconsistent patient access to orphan drugs^20–22, the use of multi-criteria decision analysis (MCDA) has been proposed as a viable method for orphan drug assessments. The MCDA framework provides a consistent, transparent and accountable approach for the decision-making process for orphan drugs. The MCDA approach assumes that the value framework adopts clear criteria and that the criteria are weighted appropriately^20,23. The premise is that because the framework is based on multiple criteria, it will provide a result that reflects not only the efficacy and cost of the drug, but also wider aspects such as disease severity, unmet need, patient’s health-related quality of life and target patient age groups. Techniques for MCDA enable expert panels to perform “trade-offs” between multiple aspects and outcomes of a product against the product’s cost as well as a combination of different criteria²⁴. The Evidence and Value: Impact on Decision Making MCDA framework was developed to support the prioritization of a broad range of healthcare interventions, with priority given to the intervention that obtains the highest rank. It has evolved and is now in its 10th edition²⁵, yet it is not designed specifically for assessing the value of orphan drugs. MCDA frameworks could also inform on societal preferences, if designed to capture such information. The International Society for Pharmacoeconomics and Outcomes Research Taskforce in multi-criteria decision analysis published recommendations on MCDA, highlighting that MCDA is a tool to support decision-making by HTA bodies and payers^26,27.

Based on our earlier research²⁸, the aim of this study was to further review the literature to establish what criteria are most frequently discussed in the context of assessing orphan drugs with a view to re-testing these criteria on a group of drugs used in the treatment of rare endocrine disorders and to analyze the possible association between the key criteria and the average annual cost of each of the treatments. All the endocrine diseases included in this study are rare (maximum prevalence of 50/100,000 population in Europe²⁹ diseases and the drugs included in the study have orphan drug designation although some have lost marketing exclusivity.

Methods

Results

Identification of articles and criteria

Even though this was a targeted literature review, we followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses method⁶⁵. The literature review identified 161 publications (after removal of duplicates), of which 15 were relevant to our study^{21,23,25,66–77} (Figure 1). A bibliometric analysis described the publication by origin of the primary author (e.g. academic, industry) as shown in Figure 2. First authors were from academia in 53% of the publications^{23,66–68,70,73,75,77}. The first authors in 13% of the publications were each from consultancy backgrounds^72,74, government or parastatal origins^25,76, and healthcare positions^69,71. A final publication (7%) was first-authored from industry²¹.

Figure 1. Targeted literature review results.

Figure 2. Bibliometric analysis of primary authorship of the studies included for analysis.

A total of 45 criteria were identified (Table 2). Geographic variation in the importance of the different criteria was observed. For example, in publications in The Netherlands and Germany, the most important criteria in MCDA for orphan drugs were Life-threatening nature of disease and Evidence of clinical efficacy and patient outcome, respectively⁶⁸. As anticipated when focusing on rare diseases, the most frequently cited criterion is Disease severity. This is inevitable, given that many of the rare diseases are life-threatening and result in significant impact on quality of life⁷⁸. The same applies to Unmet Need. By contrast, and rather surprisingly, Treatment convenience was only reported on by 7% of the studies despite the fact that some drugs may need to be administered regularly by non-oral routes, i.e. injection or intravenously and may impact on overall treatment costs and quality of life. In a landscape where increasingly Innovation (of new treatments) is of significant interest to HTA bodies, it’s a criterion that was only reported on by 20% of the studies.

Table 2. The most cited criteria in the literature.

Criterion	Percent reported in the literature
Disease severity	93%
Unmet Need (or availability of therapeutic alternatives)	87%
Comparative effectiveness or efficacy	73%
Quality of evidence	73%
Safety & tolerability	67%
Cost	60%
Cost-effectiveness	60%
HRQoL	53%
Disease rarity (Prevalence)	47%
Size of affected population	40%
Treatment impact on condition	33%
Expert consensus or clinical guidelines	33%
Complexity of manufacturing	27%
Level of research undertaken	27%
Disease burden	27%
Budget impact	27%
Type of therapeutic benefit	27%
Population priority (rare vs non-rare)	20%
Level of innovation	20%
Change in life expectancy	20%
Uniqueness or number of indications	20%
Life-saving	20%
Type of preventive benefit	20%
Age of patient population	13%
Societal need or perspective	13%
Disease impact on family/carers' lives	13%
Environmental impact	13%
Type of benefit of intervention	13%
Level of uncertainty of effectiveness	13%
Follow-up measures	13%
Comparative non-medical costs	13%
Treatment convenience (route of administration)	7%
Willingness to pay	7%
System capacity	7%
Cure	7%
Comparative other medical costs	7%
Mortality	7%
Progression rate	7%
Productivity	7%
Implementation complexity in existing healthcare services	7%
Local priorities	7%
Politics or political focus on disease	7%
Disease morbidity and patient clinical disability with current standard of care	7%
Spin-off (potential for additional indications)	7%
Duration of treatment	7%

MCDA framework: Criteria per average annual cost per patient

The average annual per patient cost in relation to the total drug score is shown in Figure 3 and the comparison of Disease severity score to the average annual cost per patient (Figure 4) highlights the cost differences for diseases with the same severity.

Figure 3. Comparison of the total drug score with the average annual per-patient drug treatment cost.

Figure 4. Disease severity compared to average annual per-patient cost.

The Unmet Need is a criterion that featured frequently in the literature as would be anticipated in relation to treatments for rare diseases, yet a plot of Unmet Need and the average annual cost suggests there is no discerning relationship based on this single criterion. In contrast, the three products used to treat acromegaly all address the same unmet need, yet considerable variation in average annual cost is observed.

The analysis reviewed the relationship between average annual cost and the number of licensed indications that each respective product has. Based on the results of this small sample size of products, the results do not suggest that the average annual cost decreases as a product is licensed for additional indications.

Discussion

This study sought to assess the key criteria used in MCDA frameworks for orphan drugs and having given each criteria a score based on an ordinal scoring system, to compare the aggregated outcomes for each drug with its average annual per-patient cost without applying different weights (importance) to any criteria. We used the criteria that appeared most in the literature, although because all the diseases studied are rare diseases, the criteria Disease rarity (Prevalence) which was cited by 47% of the studies, and Size of the affected population (40%) were not included in this MCDA framework. We observed that there is room for inclusion of others. Notably, the Treatment convenience was not included in the criteria for orphan drugs. In the context of some complex orphan drug treatments that may be used for the duration of life, such as in the lysosomal storage disorders, the cost and management implications of a drug that has to be administered intravenously on a very regular basis are greater than those of a drug injected subcutaneously once a month (e.g. treatment for acromegaly) or an of a drug taken orally. Therefore, we believe that it is a criterion worth including in MCDA frameworks for orphan drugs.

The Target age (of the treatment population) is a criterion that featured only in two of the studies we reviewed. Indeed, this could be regarded to impact on the assessment of the drug in so much as it may result in absenteeism from school for children and work for parent(s) caring for ill children. Access to orphan drug treatment could have significant impact on education attainment and productivity.

The Level of research undertaken prior to licensing of a drug is likely to establish the drug’s efficacy and safety with a view to reducing the level of uncertainty. However, we question the comparability of the level of research between products. To our knowledge, no studies have suggested a methodology to score the duration of studies versus the population size of a study. For example, would a study performed in a small population sample over several years earn a higher score than one conducted on several thousand patients over less than a year? In the scope of drugs used for rare diseases, the population size of a study is frequently very small due to the low prevalence of the disease. In diseases where there is no other treatment option, it may be necessary to reduce the impact or weight of this criterion in order to facilitate access to the medication. Irrespective of the level of research, it was seen to have little correlation with the average annual per-patient cost of each drug.

While some other studies have shown an inverse relationship between Disease rarity and price^79,80, this study was not designed to analyze this relationship. However, it provided the opportunity for a within therapy comparison of treatments for acromegaly. One might expect treatments for the same disease to have a narrow average annual per-patient cost range. Nevertheless, the average annual prices range from €617 to €20,520 demonstrating that whilst disease severity is the same for two or more drugs for the same indication, their costs may vary widely.

The criteria can be measured on various scales: binary, nominal, ordinal, cardinal or ratio⁸¹. For simplicity and transparency, we chose a numerical scoring system whereby the data for each criterion in the framework are converted into consistent numerical values from which an overall score is calculated. In their pilot study to test a MCDA for orphan drugs, Sussex et al.⁷⁴ used a numerical rating scale from 1 (worst score) to 7 (best score). Their rationale was that it permitted sufficient discrimination between levels for each criterion. By contrast, Hughes-Wilson et al.²¹ suggested a simplistic three-level scale. One of the limitations of the model might be the application of the scores to the criteria, and the subjectivity of those scores. However, as each criterion was treated equally and scored by the same team, the differences are likely to be minimal on the overall results.

The literature review highlights some of the limitations of MCDAs. Defining the criteria at the outset is crucial to ensure that overlap between criteria are avoided. As in this study, we chose to avoid confounding the overall score by excluding Disease rarity and Size of the population. It is essential that the criteria are not selected merely to favor a preferred outcome. Ultimately, disease rarity in itself should not be the reason for paying premium prices. Other factors, such as severity of the disease, may be more important⁸². Furthermore, it is unlikely that all criteria should carry the same importance (weight)²³, and as yet, very little work has been done to elucidate weighting preferences in a way that could best inform in healthcare reimbursement decisions. Weighting the criteria may be complicated, and dependent on the perspective of the assessment²¹.

Given that the criteria used might vary from one health economy to another, and similarly the importance of these criteria perceived differently, the MCDA framework will not necessarily lead to the same reimbursement decisions across different countries. Choices of criteria and weighting might be influenced by the health policy focus and the inclusion of the societal perspective⁸³.

Since regulatory organizations, such as the Haute Autorité de Santé⁸⁴ in France, have indicated that drug innovation is valued in assessing drugs, it is interesting to observe that Level of innovation did not feature more frequently in the literature. It should serve as a powerful criterion in the judgement of drugs, especially in the current climate of priority setting, not only for orphan drugs but equally for all other drugs too.

In their work on MCDAs, Hughes-Wilson et al.²¹ highlighted the need for several key criteria that should form part of an MCDA tool for orphan drugs. One such criterion is the Manufacturing complexity. Our study found this criterion to be reported by 27% of the literature review results. In the landscape of plasma-derived medicines (clotting factors, immunoglobulins) that are reputed to be more complex in their manufacturing processes^85,86 it is likely that this criterion will feature more prominently if MCDAs are to become useful assessment tools in drug assessments.

The scoring systems for each of the criteria that were adopted in this study were of the numerical form, with equal importance between grades. Despite potential criticism about the imprecision of a simple numerical scoring system, the rationale for its use is due to its simplicity and that it does not require an expert panel to adjudicate the value of one criterion against another, as would be the case in outranking methods⁸⁷, satisficing methods²⁰ and value measurement methods²⁰. Furthermore, it does not require the use of special computer software. Similarly, the same rationale applies to lack of using any weighting or preferences for a particular criterion, since the primary aim was to apply the key identified criteria as a tool for within therapy and across therapy comparisons. Since the methodology was applied uniformly across the criteria and drugs in the study, the outcomes provide valuable tools for comparison. However, the aim of the study was to demonstrate the trends and above all, to highlight the need for a different approach to assessing orphan drugs.

Conclusion

To date much work has been done in describing MCDAs and the criteria which could be considered core to value despite the evident lack of relationship between Disease severity and the overall drug score derived in the MCDA framework. Whilst no consensus on their design and applicability has yet been reached, MCDAs are frameworks that are worth adopting in the assessment of orphan drugs, specifically because they include disease-related and drug-related criteria that are likely to impact on patients and the healthcare system. They are not intended to replace the current HTA methodologies, but should certainly be used in conjunction to assist in decision making processes and prioritizing allocation of healthcare resources.

Data availability

All data underlying the results are available as part of the article and no additional source data are required.