A systematic review and critical analysis of cost-effectiveness studies for coronary artery disease treatment

Background: Cardiovascular disease remains the primary cause of death among Australians, despite dramatic improvements in overall cardiovascular health since the 1980s. Treating cardiovascular disease continues to place a significant economic strain on the Australian health care system, with direct healthcare costs exceeding those of any other disease. Coronary artery disease accounts for nearly one third of these costs and spending continues to rise. A range of treatments is available for coronary artery disease yet evidence of cost-effectiveness is missing, particularly for the Australian context. Cost-effectiveness evidence can signal waste and inefficiency and so is essential for an efficient allocation of healthcare resources. Methods: We used systematic review methods to search the literature across several electronic databases for economic evaluations of treatments for stable coronary artery disease. We critically appraised the literature found in searches, both against the CHEERS statement for quality reporting of economic evaluations and in terms of its usefulness for policy and decision-makers. Results: We retrieved a total of 308 references, 229 once duplicates were removed. Of these, 26 were excluded as they were not full papers (letters, editorials etc.), 55 were review papers, 50 were not cost-effectiveness analyses and 93 related to a highly specific patient sub-group or did not consider all treatment options. This left five papers to be reviewed in full. Conclusions: The current cost-effectiveness evidence does not support the increased use of PCI that has been seen in Australia and internationally. Due to problems with accessibility, clarity and relevance to policy and decision-makers, some otherwise very scientifically rigorous analyses have failed to generate any policy changes.


Introduction
Cardiovascular disease is the primary cause of death for Australians and places enormous strain on the health care system. Treatments for cardiovascular disease consume 12% of Australian health care spending, AUD $7 billion annually, with coronary artery disease responsible for 27% of the cost 1 . Australian health services continue to increase spending in this area, with cardiovascular disease treatment costs doubling between 2000-01 and 2008-09 1 . This increase in spending is occurring despite improvements in the cardiovascular health of Australians, resulting from improved lifestyle factors, most importantly reduced rates of tobacco smoking 2 .
There have been changes in the preferences for different treatments. Since 1998, percutaneous coronary intervention (PCI) has overtaken coronary artery bypass graft (CABG) as the most common revascularisation procedure in Australia 3 . Between 2000-01 and 2007-08 the number of PCIs performed increased by 57% 4 . Much of the increase is relates to patients treated for acute myocardial infarction, however there was a 21% increase in PCI performed in patients without acute myocardial infarction 5 . In 2012-13, 93% of PCIs involved the insertion of one or more stents 4 . Accompanying the increase in PCIs, there was a 19% reduction in the number of CABGs performed 6 . Since then, rates of PCI have remained high 4 . The increase in PCI also suggests that more patients who in the past would have been treated conservatively, with medical therapy only, are now also undergoing PCI.
Invasive treatments for coronary artery disease are costly, involving surgery, expensive equipment and consumables, yet there is no adequate assessment of the cost-effectiveness of the treatments provided. An important question is whether the extra costs incurred are adequately compensated by gains to health.
In an era of non-increasing health budgets, changes to practice should be accompanied by improvements in health outcomes, particularly if increased costs are involved. Cost-effectiveness evidence should be used to assess whether changes in costs are justified by changes to health outcomes associated with new services or changes in practice.
In the case of treatment for stable coronary artery disease, it is not clear if there is sufficient evidence to support the recent changes in treatment preferences from CABG to PCI, or a move from medical therapy alone to more invasive treatment such as PCI. While many economic evaluations have been undertaken, much of the literature assessing the cost-effectiveness of coronary artery disease treatments compares only two options at a time, with a large focus on the differences between drug-eluting stents (DES) and bare metal stents (BMS). However, comparing only two treatments at a time is limited. It assumes the chosen baseline comparator is a good quality service, and omits other available treatment options. It is sub-optimal for high-level budgetary decisions to be made without more comprehensive information about all competing treatment choices. An analysis comparing BMS only with DES fails to consider treatments other than stents and may therefore overestimate the cost-effectiveness of one type of stent, over other treatments.
Clinical trial evidence has failed to show a mortality benefit of PCI over medical therapy in the treatment of stable disease, but there is some evidence of greater symptom relief 7-10 . Coronary artery bypass graft surgery on the other hand has been shown to provide mortality benefit in some circumstances, and more prolonged symptom relief compared with PCI 9,11 . However, PCI is an expensive procedure, and CABG even more so. The question, therefore, is whether the additional costs are sufficiently offset by the greater symptom relief afforded by PCI and mortality benefit and symptom relief of CABG, when compared with medical therapy alone.
The aims of this review are to evaluate the literature that describes the cost-effectiveness of all treatment options for stable coronary artery disease: PCI including stent insertion, CABG and medical therapy, and then to critique the literature based on the quality of the cost-effectiveness evaluations and usefulness of the findings of the research for real-world applications. Usefulness for real-world applications includes the applicability of the outcomes for informing decisions about the allocation of resources, particularly in the Australian context, and the ability to translate the findings into practice.
Prior to undertaking any new research, it is important to undertake a review of the literature, to reduce the chance of duplication of effort, and to avoid tackling questions that have already been answered 12 . This review is designed to identify gaps in the knowledge about the cost-effectiveness of treatments for coronary artery disease, and therefore to inform future research in this area. Our goal is to provide insights useful for clinicians, healthcare service budget holders, and policy-makers about the best use of scarce resources for the treatment of coronary artery disease.

Methods
The literature published between January 1995 (after the use of stents was approved in the United States in 1994) and May 2017 was searched in PubMed, Embase, Scopus, CINAHL (via Ebscohost) and EconLit (via ProQuest). The searches focussed on extracting papers that examined the cost-effectiveness of PCI (including stent insertion), CABG and medical therapy together. Due to the large volume of research on coronary artery disease, searches were limited to subject headings where possible. A slightly

Amendments from Version 1
In response to the reviewers' comments we have updated the text to make it clear that the review relates to stable coronary artery disease only. Related to that, we have updated information about the increase in PCI in non-AMI patients. We added text to clarify that direct costs were used in all papers examined, and, that the analyses would be relevant to the Australian context where Australian costs could be applied. We have included some further comments regarding the changing clinical evidence in this field and that this will add to the cost-effectiveness evidence in the future. We have also updated Table 4 and Table 5 so that the studies are in the same (alphabetical) order and included a traffic light system regarding overall usefulness. The references have also been updated. Search results were imported into EndNote X7 software, duplicates were removed and articles then reviewed according to theinclusion and exclusion criteria outlined in Table 1.
We used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement 13 and checklist in retrieving and reviewing articles for inclusion in this review. Due to the nature of this review, not all items were relevant. Our completed checklist is available in Supplementary File 2. Titles of all papers were reviewed and the abstract or full text examined in detail where required to assess against inclusion and exclusion criteria.
We used the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) Statement 14 to assess the quality of reporting of the economic analyses (Supplementary File 3). The CHEERS Statement and checklist was developed to improve and promote quality of reporting health economic evaluations 15 . Economic evaluations are designed to assist in health service decision-making and resource allocation. Therefore, due to the opportunity costs of acting on poor-quality evidence, it is particularly important to ensure high-quality reporting in economic evaluations 15 . The CHEERS Statement checklist consists of 24 items which should be reported and guidance regarding the specific details required. We extracted information from each of the included studies foreach item on the checklist, to assess the quality of reporting.
In addition to high-quality reporting, economic evaluations need to be useful to decision-makers, as their purpose is to provide evidence to improve the efficiency of use of healthcare resources. Decision-makers need to understand the potential impact of acting on cost-effectiveness evidence and making changes to healthcare services; most notably what will be the effect on health outcomes and costs, and how certain are these projections? To assess the usefulness of the evaluations for decision-makers, we also extracted data on the interventions compared, the effectiveness measures, whether the analysis applied to a specific patient group, the structure or type of analysis used and the time period of the analysis. We then assessed usefulness of the reporting by rating the reporting of outcomes, costs, uncertainty as: useful, not useful, or partly useful, based on whether the results could be used by a decision-maker. We also looked for a clear statement about the policy implications or direction that should follow based on the outcomes, and gave each paper an overall usefulness rating of low, medium or high, depending on the other elements assessed. We acknowledge that these ratings are subjective and have not been validated, nevertheless we think they are practically useful.

Results
Searches in all databases, except EconLit, revealed potentially relevant articles. A total of 308 results were retrieved, and 229 remained after duplicates were removed. The numbers of papers retrieved from each database are in Figure 1.
Many articles tagged under the cost-benefit or cost-effectiveness subject headings were not cost-effectiveness analyses but simply mentioned cost-effectiveness as a factor for consideration. In addition, most of the cost-effectiveness papers did not examine medical therapy, percutaneous coronary intervention and bypass graft surgery together, but focussed on only two treatments. Papers were excluded for other reasons including a focus on rehabilitation following cardiac procedures, screening of cardiac patients, being  commentary only or reviews. However, we checked for papers included in reviews that had not been found in our searches. The results of the review process are shown in Figure 1.
The results of the review process left only five papers for consideration. Table 2 provides a summary of the papers included in the full review. The results varied across the five studies, but across the scenarios analysed most concluded that medical therapy was the cost-effective treatment, and three concluded that CABG was cost-effective. In no scenario was PCI reported as being cost-effective compared with the alternatives (Table 2). Three studies included quality of life measures in at least part of their analyses and only two studies undertook projections over the lifetime of patients ( Table 2).

Quality of reporting
A summary of the assessment against the CHEERS Statement are shown in Table 4. Overall, the quality of reporting was high, with studies adequately reporting against 50 to 100% of relevant items on the checklist (Table 3). A more detailed table is available in Supplementary File 3.
All studies adequately reported on the CHEERS Statement items relating: to model/analysis description, background and reasons for undertaking economic evaluations, relevant patient groups and sub-groups, comparators, time horizons and choice of health outcomes included in their analyses. The most poorly reported element related to reasons for choice of model (part of item 15). Only three studies did this. The others described the analysis undertaken, but did not give reasons for the chosen strategy.

Usefulness of evaluations
While for the most part the studies reported their analyses and findings to a good standard when assessed against the CHEERS Statement, their usefulness to decision-makers is arguably of greater importance. The summary data extracted in relation to usefulness of each paper is shown in Table 4. Our assessment of the usefulness of the reporting for decision-makers is in Table 5.
Of the five papers reviewed in full, three were trial-based analyses, one used a cohort modelling approach and there was one metaanalysis (Table 4). The timeframes analysed ranged from 1 year post-intervention to a lifetime horizon. The studies came from a wide range of countries. Three of the five studies used quality of life measures in their analyses, one of which only considered QALYs as part of sensitivity analyses. The others used length of life measures to assess cost-effectiveness and one also used the  In assessing the usefulness of reporting, we found that while most studies reported on various items, reporting was not always easy to interpret in the context of decision-making. In judging the reporting we were looking for a clear direction or suggestion about how the results of the analysis could be used to improve the efficiency of healthcare resource use. Only two studies made a clear statement about changing the allocation of resources or how the outcomes are relevant to policy 17,18 . We rated two studies as low usefulness, two medium and only one highly useful for decisionmakers.
Caruba et al. 16 carried out a meta-analysis of cost-effectiveness studies. After concluding that there was no statistically significant difference between treatment strategies on clinical endpoints of myocardial infarction or death, the analysis was conducted on costs only, over 1 and 3 years 16 . As a result, the analysis focusses primarily on cost differences across the treatments. They estimated that substantial cost savings could be made through the management of patients with stable angina, using medical therapy 16 .
A more detailed examination of the outcomes reported by Caruba et al. 16 revealed that while no statistically significant differences were found, there appears to be some clinically significant difference in treatment effectiveness. The confidence intervals of hazard ratios reported for both death and myocardial infarction are very wide. For example, at three years follow-up, confidence intervals related to estimates of risk of death range from a halving to a doubling of risk for all comparator treatments 16 . Similarly, the probabilities of being the best treatment vary widely; from 0.49 for drug-eluting stents to 0.05 for percutaneous transluminal coronary angioplasty (for risk of death at three years follow-up) 16 . These results suggest both that there is a high degree of uncertainty regarding estimates of effectiveness, and therefore that a clinically significant difference between the treatments is possible. This could greatly affect estimates of cost-effectiveness. In addition to   We rated this meta-analysis as of low use to decision-makers due to the level of uncertainty described, making it difficult to interpret how the findings might be used to direct policy or practice. While a high level of uncertainty in results should not disqualify an analysis from being useful, the authors did not make any statements which assist in determining how the findings might be used. In addition, the difference in effectiveness of treatments was not fully explored, adding even more uncertainty to the findings.
An analysis by Fidan et al. 17 modelled the life years gained for 36 condition-treatment scenarios for coronary artery disease. These included everything from acute myocardial infarction to primary prevention using statins 17 . They used the IMPACT model; a large cell-based mortality model of coronary heart disease risk and treatment 17,21 . Cost-effectiveness ratios were reported, however these are not presented as incrementally, which makes it difficult to do head-to-head treatment comparisons. All treatments were examined against the baseline mortality rates, and they found that medical and surgical treatments prevented or postponed over 25,000 deaths in patients with coronary artery disease 17 . The approach ranked different interventions, showing a 100-fold difference in cost-effectiveness across all treatments, but it does not provide insight into incremental costs associated with new technology or interventions 10 . Again, the analysis only considers length of life, not quality of life. While this provides useful information about length of life it does not consider the full effect of different treatments. Included in this assessment are treatments for chronic angina. It is a somewhat counterintuitive approach to only examine length of life gains from a treatment that targets symptom relief.
While the reporting in this study is clear, we rated it as of medium usefulness for decision-makers because it does not include an assessment of the full effect of treatments on health outcomes (i.e. it only assessed length of life). The authors did, however, present a general policy suggestion, stating that investment in secondary prevention was likely to produce gains in length of life for lower costs.
Prior to the analysis of the MASS-II trial, there had been no costeffectiveness analysis based on a trial comparing percutaneous intervention, surgery and medical therapy together 20 . To address this gap, Griffin et al. 18 conducted an economic analysis using the Appropriateness of Coronary REvascularisation (ACRE) study cohort. The ACRE study rated patients as appropriate for percutaneous coronary intervention and/or coronary artery bypass grafting but followed them according to the treatment they actually received 18 . Economic analysis of the ACRE study data concluded that coronary artery bypass grafting was cost-effective compared with percutaneous coronary intervention in patients classified as appropriate for bypass grafting only or for both bypass grafting and percutaneous intervention 18 . The analysis also found that percutaneous coronary intervention was not cost-effective when compared with medical therapy for patients classified as appropriate for percutaneous coronary intervention only 18 . The results of this analysis are useful because they include quality of life outcomes. However, the approach used averaged quality of life over the 6-year period using a regression model 18 . This gives some good information about the average quality of life of patients receiving different treatments over the time period, but does not account for events during which patients might experience reduced quality of life, such as a period of hospitalisation for a subsequent procedure.
We rated this analysis as of high usefulness to decision-makers. While the estimates of quality of life could be improved, the authors make clear statements about the changes in costs and health outcomes achieved through different treatments. They also make a clear statement of how to make changes to resource allocation based on their results, which could benefit the health service. One limitation to the usefulness of the outcomes presented is that they only cover the six-year trial period. This may not be long enough to see the full cost-effectiveness of treatment for a chronic disease and the authors foresee extending the model over a lifetime horizon in future work 18 .
The analysis by Hlatky et al. in 2009 examined the cost-effectiveness of revascularisation procedures in patients with type-2 diabetes, using data from the Bypass Angioplasty Revascularization Investigation 2 Diabetes trial (BARI 2D) 19 . The BARI 2D study randomised patients with type 2 diabetes to medical therapy alone or medical therapy with immediate revascularisation (either PCI or CABG) 22 . While the effectiveness of treatment for coronary artery disease has been shown to be affected by the presence of diabetes 23-26 , due to the high prevalence of type 2 diabetes in this patient population and more generally, this analysis was not excluded on grounds of being relevant only to a specific group. The rates of diabetes in other included studies are 36% in MASS II 20 , 15% in the ACRE study 18 , and 9 to 33% in the studies included in the meta-analysis by Caruba et al. 16 . The economic evaluation of the BARI 2D study outcomes concluded that medical therapy was cost-effective compared with revascularisation (PCI or CABG), in the short-term (4 years) 19 . When using lifetime projections of cost-effectiveness, however, medical therapy was cost-effective compared with PCI, and CABG was cost-effective compared with medical therapy 15 .
The BARI 2D trial used a pragmatic approach which reflects the realities of clinical practice; patients undergoing revascularisation were not randomised to a particular revascularisation strategy (i.e. PCI or CABG); this was directed by clinicians ahead of randomisation to either prompt revascularisation or medical therapy 22 . The effect of this is that patients were stratified into groups based on clinical markers of disease severity. The results of the study are therefore useful for choosing between medical therapy and PCI in patients with less severe disease, or between medical therapy and CABG in patients with severe disease. They are also only relevant to diabetic patients, however, as prevalence of type 2 diabetes is increasing globally, this is relevant to an increasing number of patients.
The overall results in the BARI 2D trial are based on length of life measures; quality of life measures were only used in sensitivity analyses. It was concluded that the quality of life measures did not affect the estimates of cost-effectiveness, based on life-years only. It is unclear why this choice was made, when quality of life measures provide a more comprehensive assessment of treatment effect. We rated this study as of medium use for decision-makers as it presents an analysis of real-world practice, but does not account for the full effect of treatment on patients' health, by all but ignoring quality of life measures. Decision-makers wishing to know the full effect of different treatments on patient health outcomes need information beyond length of life.
Vieira et al. also conducted a trial-based analysis using data from the MASS II Trial (Medical Angioplasty or Surgery Study) 20 . This was the only trial revealed in searches which randomised patients to each of the three treatment options. Its major conclusions were that medical therapy was cost-effective compared to CABG, and CABG was cost-effective compared to PCI 20 . While this analysis did use QALYs they were not calculated using conventional health related quality of life surveys, but estimated based on the average time to event and angina free proportion of the population in each group 20 . This is unlikely to provide good estimates of quality of life in these patients as the measurement assumes that in the period between events the patient has full quality of life and that those with angina have no quality of life. These estimates produced We rated the analysis by Vieira et al. 20 as of low usefulness to decision-makers because although quality of life was included in the analysis, it was not done in a way that makes it comparable to other studies. In addition to these novel methods of QALY estimation, the authors did not conduct incremental analyses, nor did they discuss any uncertainty in their findings.
It is worth noting that further research has been undertaken using the MASS II trial data and a validated quality of life instrument 32 . Unfortunately, only a conference abstract was available and it was therefore not included in the analysis. The results available in that abstract show much higher average health utility weights of 0.77 to 0.81 32 , aligning them with the values seen in other analyses of coronary artery disease 27-31 . When published, the full analysis will add greatly to the current knowledge. Data related to this review are available in an EndNote Library, containing all references retrieved using the search terms described. This library also contains subfolders used to categorise papers during the review process.

Discussion
Challenge for health service decision-makers When operating under conditions of scarce resources there is responsibility to promote cost-effective care, achieving larger health gains from available resources. In an ideal world, decisionmakers would have information about the long-term costs and health outcomes achievable through different configurations of health services and be able to invest accordingly. However, without good evidence of cost-effectiveness, it is impossible for decisionmakers to fulfil this responsibility with any confidence.

Current evidence and value of evidence for coronary artery disease
In the case of stable coronary artery disease, we have some information about the comparative cost-effectiveness of optimal medical therapy, PCI and CABG, but it is difficult to interpret in the context of healthcare resource allocation. Overall, the results of cost-effectiveness analyses suggest that in most scenarios optimal medical therapy is cost-effective compared with alternatives and CABG is cost-effective for certain patient groups.
However, only three of the five studies included in this review used quality of life as an effectiveness measure. This is a key outcome measure for cost-effectiveness and good decision making. For chronic diseases, improvements in quality of life are equally as relevant as improvements in length of life. In the case of coronary artery disease, relief from chest-pain is a key objective of treatment. If quality of life is not measured, two treatments affording a patient equal length of life are valued equally even where one restored the patient to better health than the other. However, if given the choice, patients and health service providers would choose the option most likely to provide the best improvements to quality of life. Therefore, analyses based solely on length of life measures do not provide a full picture of the effectiveness of each treatment.
Another omission from the literature is to neglect the lifetime costs and health outcomes. Important information about the longevity of treatment effect may be overlooked. This is particularly important for chronic diseases such as coronary artery disease, where important costs and health consequences are missed when they occur beyond the timeframe of a clinical trial.

Confidence in changing services based on current evidence
The current information about cost-effectiveness of treatments for stable coronary artery disease suggests that either optimal medical therapy or CABG could be cost-effective, over a 1-year to lifetime timeframe. There is no evidence from the papers included in this review that PCI is cost-effective when compared with other competing treatment options. Therefore the current cost-effectiveness evidence does not support the increased use of PCI that has been seen in Australia and internationally, and there is increasingly reduced evidence of clinical effectiveness 33 .
However, it is unlikely that healthcare decision-makers would be confident making changes to the allocation of resources based on the economic evidence outlined in this review. Our evaluation of the relevant cost-effectiveness evidence showed that overall, information is not presented in a way useful for decision-making. We found only one study to be of high usefulness in this context (Table 5).
Our assessment of the usefulness of the cost-effectiveness studies examined suggests that poor reporting may contribute to the problem. We rated two out of the five studies as 'low usefulness' for decision-makers. Reporting was either too complex, making interpretation challenging, or uncertainty was not reported in way that made clear the effect of acting on the evidence. It is also apparent from our analysis, that the CHEERS Statement, while encouraging comprehensive reporting, is not sufficient alone, to assess the usefulness of economic evaluations.
Others have explored barriers to the use of economic evaluation by decision-makers 34 . A review by Merlo et al. 34 used an accessibility and acceptability framework developed by Williams and Bryan 35 to categorise barriers to use of economic evaluations. Accessibility refers to the ability of decision-makers to interpret and use economic evidence, and includes issues of complexity andtimeliness of economic evaluations 34,35 . Acceptability includes factors associated with scientific rigor, applicability to the institution in which decisions are to be made, and ethical considerations such as equity 34,35 .
While some studies we examined included comprehensive reporting, they did not always present results in a way conducive to decisionmaking. For example reporting large results tables covering many different clinical scenarios, as seen in the paper by Fidan et al. 17 , demonstrates the complexity but does nothing to assist those wanting to make higher-level resource allocation decisions. While clinical complexity is inherently important in many contexts, for it to be of use for decision-making, it needs to be summarised in a way which makes clear the likely impact of acting on the evidence presented. In fact, presenting all clinical complexities can mean the overall message is lost in the details, making research less accessible to decision-makers and decreasing the chance that any improvement to health services will follow. This is unfortunate, considering the purpose of economic evaluation is to provide evidence for resource allocation decisions to improve service delivery.
Most of the economic evaluations we have assessed suffer from a number of accessibility problems which prevent them from being useful for decision-making; the interpretation of results and their applicability to policy are generally lacking. The information in Table 5 19 and Vieira et al. 20 expressed a direction but less-clearly). The result of this highly complex reporting and lack of clear policy direction to follow, is that some otherwise very scientifically rigorous analyses have failed to generate any policy changes or perhaps even reach their intended audience.
In addition to these complexities, while the patient populations may be similar to those in Australia, none of these studies have been carried out in the Australian context. For the purposes of resource allocation decisions in Australia, an analysis based at least on Australian costs, is required.

Conclusion
The evaluation of the studies in this review highlights a lack of information useful for making decisions about the allocation of resources for coronary artery disease. It is concerning that over $2 billion of Australia's annual healthcare budget is being spent on coronary artery disease, with inadequate economic evidence.
Since the mid to late 1990s, increased spending has been directed towards PCI with stenting, over coronary artery bypass grafting, but there is insufficient economic evidence to support this transition.
Compounding this, the current cost-effectiveness evidence which would suggest a move away from PCI and stents remains too unclear and uncertain for policy-makers to be confident in making changes. In a time of increased pressure on health budgets, economic evidence should be fundamental to resource allocation decisions.
For those wishing to make resource allocation decisions to improve the efficiency in treatment of coronary artery disease, the current evidence is insufficient. A transparent, structured, lifetime analysis of all competing treatments, incorporating quality of life measures, would be valuable for decision makers. The analysis should account for fluctuations in the quality of life of patients over their lifetimes, related to symptom relief, repeat procedures and acute events.
The findings of recent trials, in particular ORBITA 33,36 and ISCHEMIA 37 , will strengthen the evidence about the effectiveness of conservative versus invasive therapy for stable coronary artery disease. In turn these data will enable improved cost-effectiveness analyses.
To be of use to decision-makers and have a better chance of generating policy change, analyses must be accessible; economic evaluations should include a clear indication for the direction of policy or changes to practice that should follow and a statement of the probability that such changes will the generate the predicted improvements. In their systematic review of barriers and facilitators to use of evidence by policymakers, Oliver et al. named clarity, relevance and reliability as some of the top barriers to use of evidence 38 . For decision-makers to be able to act on the economic evidence, the expected effect of making changes based on the results needs to be clear. In cases where there is too much uncertainty, a strategy to improve the analysis should be outlined. We suggest that to improve reporting of economic evaluations for decision-making, an additional item could be included in the CHEERS Statement, relating to implications for policy and practice. Ideally, this would be a statement describing the implications of acting on the evidence presented; encompassing both expected improvements to health outcomes and confidence in the effect.

Data availability
Dataset 1: Endnote library of retrieved references -Data related to this review are available in an EndNote Library, containing all references retrieved using the search terms described. This library also contains subfolders used to categorise papers during the review process. 10.5256/f1000research.13616. d190562 39

Competing interests
No competing interests were disclosed.

Grant information
This work forms part of the PhD Candidature of Victoria McCreanor, funded by the Capital Markets Cooperative Research Centre and supported by the Queensland University of Technology.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Supplementary material
Supplementary file 1: Search terms used in each database searched.
Click here to access the data.
Click here to access the data.
Supplementary file 3: CHEERS Statement Checklist -completed for reviewed studies.
Click here to access the data.

James P Howard
International Centre for Circulatory Health, National Heart and Lung Institute, Imperial College London and Imperial College Healthcare NHS Trust, London, UK This is an excellent review of cost-effectiveness studies comparing percutaneous coronary intervention (PCI), coronary artery bypass grafting (CABG) and optimal medical (OMT) therapy for stable coronary artery disease (CAD). It is well-written, easy to follow, and the interpretations are sound.
My only query about the design of the study is I'm not entirely sure why studies comparing only PCI vs OMT were excluded, given the authors go on to rightly raise concerns about a lack of high-quality data. We must remember that the majority of of patients with stable CAD are not candidates under current guidelines for CABG (e.g. if they lack 3 vessel or proximal LAD disease) and so to argue a cost-effectiveness study should use data where all three are compared because CABG could be a superior options to PCI seems strange to me. Even if CABG was deemed more effective than PCI in a study, the finding would not be applicable to most of our patients, as only a small subset of our patients would be eligible for randomisation in whatever study that data was based on.
I also wonder whether the authors think this data will change in the next few years? The two most high I also wonder whether the authors think this data will change in the next few years? The two most high profile studies of stable CAD in the last few years are probably FAME-2 and ORBITA. The former has shown improvements when measured as a reduction in a composite of death, myocardial infarction and urgent revascularisation (although this latter largely powers this and there have been significant concerns about this endpoint in unblinded settings). ORBITA initially showed no benefits in improvements in exercise time from PCI versus optimal medical therapy, but the recent secondary analysis in Circulation showed a significant reduction in the number of patients reporting angina following PCI versus placebo (when all patients were on OMT). I appreciate that the authors are reviewing cost-effectiveness studies, and it may be several years before these new data are included in such studies, but it appears that under this current design (including only cost-effectiveness studies involve CABG) would mean the only blinded trial of PCI versus OMT that I am aware of would be ineligible for analysis? Some minor points are there isn't discussion about the role of direct versus indirect cost measurements in the different studies. This is crucial when considering the applicability of results to different healthcare settings, where sometimes indirect costs are borne by providers but sometimes not.
Also, the authors do query about the applicability of data to Australia. Whilst this may be an issue, it seems likely that evidence from Europe and USA are of relatively-good applicability to Australia as similarly developed nations with, in some cases, similar healthcare systems.
Finally, a tiny point is I think it would be useful to order tables 4 and 5 the same way; I foolishly initially assumed that the top study in both (the most recent study using meta-analysis in table 4 and the high quality study in table 5) were the same study, when actually the most recent study was deemed of 'low' usefulness). Indeed, table 5 may be better as a traffic light system with colour coding to make things more clear, but I will leave that up to the authors and editorial team.

Are the conclusions drawn adequately supported by the results presented in the review? Yes
No competing interests were disclosed.

Competing Interests:
I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
Author Response 26 Jun 2018 , Queensland University of Technology, Australia

Victoria McCreanor
We agree with the comments regarding appropriateness of treatment in different patient groups. However, the problem with reviewing papers comparing two treatments is complexity. This would introduce three additional reviews (PCI v CABG, OMT v CABG and OMT v PCI) -All are relevant However, the problem with reviewing papers comparing two treatments is complexity. This would introduce three additional reviews (PCI v CABG, OMT v CABG and OMT v PCI) -All are relevant but of course, by necessity, only pertaining to more and more selected groups. Thus, comparisons would be hard to interpret, if not, impossible.
Regarding the comments about data changing in the coming years, in short, yes, we think data are likely to change. Invasive procedures for stable disease are currently being subject to increasing scrutiny, particularly in the wake of ORBITA, as you suggest. The secondary analysis of ORBITA provides a well-needed and important contribution to the field and it will be interesting to see how those results affect cost-effectiveness estimates. We have included a comment in the discussions to that effect.
Other important data will come from the ISCHEMIA trial, which randomises patients with stable disease to conservative or invasive strategies, particularly in relation to long-term outcomes. In response to the other points, all studies used direct costs and we have updated the text accordingly.
We agree that evidence from Europe and USA is of relatively-good applicability to Australia and have updated the text to include a comment that existing analyses could be applied to the Australian context if Australian costs were used.
We thank you for the useful suggestion to make the tables clearer. We have updated tables 4 and 5 so the studies are in the same order and included a traffic light system for overall usefulness.
No competing interests were disclosed. This paper presents a valuable critical analysis of cost-effectiveness studies and highlights serious issues in relation to potential inefficiencies in funding for Coronary Artery Disease (CAD) treatment. As identified by the authors, heart disease is a significant area of expenditure for health in Australia and this paper contributes important information for policy consideration. The paper is well-written and raises a number of recommendations useful for general presentation of cost-effectiveness analyses in the literature. In particular, these recommendations are useful in terms of the quality of life measurement and the application of the CHEERS checklist. One interesting issue raised here -which could potentially be further explored -is in terms of high-level policy changes. All new procedures presented for listing on Medicare or the PBS necessarily undergo a rigorous process of evaluation, including assessment of cost-effectiveness evidence, which often includes commissioned modelling studies to apply international clinical findings for Australian scenarios. The economic evaluation requires comparison of the new technology or drug with the status quo (or best current alternative) to appraise the incremental cost-effectiveness for the specific population group defined in the submission. However the point raised here is an important one -that incremental benefits and costs in cost-effectiveness may not reflect the bigger picture if the underlying comparator is flawed. Nonetheless it is a much greater undertaking to address wider policy ramifications, particularly where evidence continues to accrue. (Notably the current MBS taskforce is endeavouring to address such evidence continues to accrue. (Notably the current MBS taskforce is endeavouring to address such challenges.) One major criticism however is that it is not made clear to the reader that the analysis specifically targets stable angina, which is a subgroup of CAD patients. In particular, the abstract suggests that the results apply to treatment of all CAD patients and hence the results are potentially misleading when considered across all patients with CAD. It is important to know what proportion of CAD patients present with stable disease (and hence are eligible for all three treatments (CABG, PCI, MT). The information is implicit since it is clear from the population group of each of the included trials that patients with acute disease have been excluded. The fairly recent RCT which provides the most enlightening data, has quite strict exclusion criteria, which lists a number of presentations. The conclusions of the review are valid, but the subgroup requires transparency. This leads to a second point, which questions the importance of the clinical diagnosis. NICE clinical guidelines recommend drug treatment for stable angina unless symptoms are not satisfactorily controlled, in which case interventional procedures should be considered. This suggests a value judgement in selecting patients for interventional care. Does this potentially present a continuum that has shifted over time? It is important to assess what proportion of the cited 57% increase in PCI has been for the treatment of stable CAD. A related query is whether the non-RCT studies were subject to possible selection bias. A fundamental question then arises as to whether the cost-effectiveness studies should stand apart from the clinical data. That is, if the clinical evidence is changing (and clearly the controversy around interventional procedures for stable disease reflects the growing evidence) -it may be most useful for the cost-effectiveness to be presented for the specific sub groups that are most controversial -for example based on the differential burden of ischaemia. Queries and comments: · Reviews were excluded from consideration -were they considered for comparison of findings and were they scrutinised for potentially missed references? · '..most of the .. papers did not examine medical therapy.' Was this because they pertained to acute presentations or was medical therapy overlooked for stable presentation?
I disagree with some of the statements made around usefulness for policy. For example the statement that presenting clinical complexities 'does nothing to assist those wanting to make higher level resource allocation decisions'. Complexity may limit relevance in some cases, but if cost-effectiveness varies according to clinical diagnosis (and evidence supports this), then the heterogeneity may be an important consideration. Also, the statement referring to publications that consider only 2 treatments as not being useful to policy -if the policy decision relates to critical cases such as myocardial infarction or if it relates to a minor change to policy already in existence, then comparison of more than 2 options may not be relevant. Thirdly, non-Australian evidence may be appropriate if the clinical findings are relevant to an Australian population and Australian cost data can be applied. It may be in the wording of some of these statements, given the separation of higher level allocation decisions and those relating to small changes in resource allocation. However most policy decisions assess a well-defined intervention, compared to a current alternative, for a clearly defined patient group (which is necessary to prevent leakage), so commonly the detailed breakdown of cost-effectiveness provides critical information for those decisions. The paper provides an interesting supplement to the current clinical literature on the controversy around treatment of stable coronary artery disease. The discussion of the importance of policy-relevant evidence will inform future cost-effectiveness analyses of trials currently in progress.
Are the rationale for, and objectives of, the Systematic Review clearly stated?

Are sufficient details of the methods and analysis provided to allow replication by others? Yes
Is the statistical analysis and its interpretation appropriate? Not applicable We agree with these comments regarding the Australian regulatory system, however, we don't feel this paper is the right place for a discussion of Australian regulatory processes. We also agree that economic evidence may change as new evidence accrues, however, cost-effectiveness analysis based on imperfect information should still be used as long the caveats are noted and the decision makers are fully informed about the problems with any analysis. Using imperfect information to make decisions is better than making them with no information. As with clinical evidence and practice, economic evaluations can be updated as new evidence becomes available.
We have updated the text to make it clear that the review relates to patients with stable disease. We agree with the comments regarding the importance of clinical diagnosis, however there is mounting evidence, for example from the first results from the ORBITA trial published in the Lancet, that in stable patients, interventional treatment may not improve patient outcomes to the extent originally expected.
Regarding the change in preference towards PCI, we have updated the text to include the proportion increase in PCI related to non-AMI patients.
The point raised about non-RCT studies being subject to selection bias is valid, however we would suggest that perhaps these better reflect real-world practice and therefore may be more useful for cost-effectiveness analysis and policy than results from highly-controlled clinical trials. RCTs aim for high internal validity, which comes at the expense of generalisability, and generalisability is more important for economic evaluation and policy decisions.
We agree that where clinical evidence changes, economic analyses may also need to be updated. However, we think it is important to use the evidence currently available, rather than waiting for perfect information.
We agree that where the effectiveness is affected by particular clinical sub-groups, it will likely be important to present the cost-effectiveness evidence accordingly. In this paper we only attempted to review cost-effectiveness analyses which compared PCI, CABG and OMT in groups of patients to review cost-effectiveness analyses which compared PCI, CABG and OMT in groups of patients where all three were considered appropriate treatment options.
The other reviewer also raised queries about relevance across different patient sub-groups and we noted the following: This would introduce three additional reviews (PCI v CABG, OMT v CABG and OMT v PCI) -All are relevant but of course, by necessity, only pertaining to more and more selected groups. Thus, comparisons would be hard to interpret, if not, impossible. In response to the queries and comments: We have updated the text to note that we examined review papers for potentially missed references.
The sentence referred to, '..
, meant that most of the .. papers did not examine medical therapy.' most studies did not examine all three treatment groups together. However, most compared interventional treatments with each other and did not consider medical therapy as an alternative treatment on its own. Some pertained to acute presentations and others to stable, however, as noted this paper relates only to stable disease.
In response to comments about statements regarding usefulness for policy: We agree that clinical complexity is important in many instances. In the comment highlighted, we refer specifically to the Fidan et al paper, which presents 34 scenarios together, making it difficult to interpret in the context of resource allocation decision-making. We have updated the text to clarify this.
Regarding comments about two-treatment comparisons not being useful for policy, we agree that if only two treatments are relevant to a certain cohort, then an analysis of only those two would be useful. We have also addressed in this in an earlier comment.
Perhaps it is our wording, but we do not mean to imply that two-treatment comparisons are irrelevant in all contexts. We do state that if a two-treatment analysis omits another relevant treatment, then the analysis is limited.
We agree with the comments regarding the Australian context and have updated the text to note that Australian costs could be applied to an existing analysis.
We agree that detailed breakdown of cost-effectiveness analysis and caveats is important for policy decisions and have discussed this in some of our earlier comments. Our main point, however, is that presenting too much detail can detract from the overall message.
No competing interests were disclosed. Competing Interests: