ALL Metrics
-
Views
-
Downloads
Get PDF
Get XML
Cite
Export
Track
Research Article

Scepticaemia: The impact on the health system and patients of delaying new treatments with uncertain evidence; a case study of the sepsis bundle

[version 1; peer review: 2 approved]
PUBLISHED 26 Apr 2018
Author details Author details
OPEN PEER REVIEW
REVIEWER STATUS

Abstract

Background: A sepsis care bundle of intravenous vitamin C, thiamine, and hydrocortisone was reported to improve treatment outcomes. The data to support it are uncertain and decision makers are likely to be cautious about adopting it. The objective of this study was to model the opportunity costs in dollars and lives of waiting for better information before adopting the bundle.
Methods: A decision tree was built using information from the literature. We modelled the impact of bundle adoption under three scenarios using a simulation in which the bundle was effective as reported in the primary trial, less effective based on other information, and ineffective.
The measurements were health services costs, quality-adjusted life years, and transition probabilities.
Results: If the bundle proves to be effective under either scenario, it will save billions of dollars and millions of life-years in the United States. This is the opportunity cost of delaying an adoption decision and waiting for better quality evidence. We suggest that hospital decision-makers consider implementing the bundle on a trial basis while monitoring costs and outcomes data even while the evidence base is uncertain.
Conclusions: If the decision maker is unwilling to use the best available evidence now, but rather wishes to wait for definitive evidence they are risking incurring large costs for health care systems and for the patients they serve. An explicit analysis of uncertain clinical outcomes is a useful adjunct to guide decision making where there is clinical ambiguity. This approach offers a valid alternative to the default of clinical inactivity when faced with uncertainty.

Keywords

Sepsis, Costs and Cost Analysis, Evidence-based practice

Introduction

Sepsis arises frequently among patients admitted to hospitals in the US and elsewhere. It is often fatal, accounting for 30% to 50% of inpatient deaths1. Those who survive incur large costs from increased risk of organ damage2. A recent paper by Marik et al. (2017) described the effectiveness of a treatment bundle made up of intravenous vitamin C, thiamine and hydrocortisone3. Because of small sample size, non-randomised design, large observed treatment effects and the simple and low cost characteristics of the bundled intervention, there was scepticism among clinicians, administrators and payers regarding adoption46. Rapid adoption without good evidence has risks and costs, yet delaying an effective treatment imposes larger costs to patients and the health system. Waiting for conclusive evidence might be a poor and costly strategy and should be balanced against the likely costs of rapid adoption with uncertain and low quality evidence.

The aim of this paper is to estimate the economic consequences of a decision to adopt the Marik sepsis bundle early, under conditions of large uncertainty. This is compared to the alternative, which is to wait one year; the time period between publication of the first paper in June 20173 and the completion of the National Institutes of Health funded trial due in mid-2018. The trial will ideally reduce the uncertainty in an adoption decision. This paper predicts the cost savings and health benefits of adopting the Marik bundle immediately with uncertain evidence for the entire United States health system, compared to waiting for better quality evidence. Our estimates examine likely changes to costs and health outcomes if the treatment is found to be effective, less effective and ineffective.

Methods

An incremental cost-effectiveness analysis was conducted on the change to costs and Quality-Adjusted Life Years (QALYs) of standard sepsis care compared to the early adoption of the Marik bundle3. Values for expected costs and health utilities were taken from the literature, with probabilities of treatment outcomes taken from Marik’s paper3. The time horizon was 5 years in order to sufficiently measure the long-term outcomes of acute kidney injury (AKI), an important and common result of severe septic shock2. A decision tree was programmed in TreeAge Pro 2017 R2.1 (Williamstown, MA2017) and prior statistical distributions of costs, outcomes and probabilities used to include uncertainties in the data, see Figure 1. Patients receiving either modality progress through a chance node to death or survival. Surviving patients progress through a second chance node, where they would either recover or recover with organ damage. AKI is a common outcome in cases of sepsis, and typically require renal replacement therapy, which can take the form of either dialysis or organ transplantation2.

0925e6f2-e9a9-488a-b666-16e0be673087_figure1.gif

Figure 1. Sepsis decision tree model using TreeAge Pro 2017.

Costs

The costs of care include the hospital cost of a sepsis episode, the additional cost of the Marik treatment bundle and the annual ongoing cost of renal replacement therapy (RRT). Hospital costs were taken from a systematic review by Arefian et al. (2017) for a US perspective with a mean of $32,421 and standard deviation of $15,051 per stay7. For the Marik bundle, we used an estimate of $528 and a standard deviation of 5% at $26.50 per patient per course. The annual cost of RRT for end stage renal disease varies depending on treatment modality, such as dialysis or transplant, and a weighted average of $76,936, with a standard deviation of $15,387 was taken from the 2017 USRDS Annual Data Report8.

Health utilities

The health utility score of patients reflects the value of their heath state and is a measure of health related quality of life. It is expressed on a range between zero, the worst health state, and one, the best possible health state9. The scores for patients who recover from septic shock depend on whether or not they suffered AKI. Patients who did not suffer AKI still underwent the stress of intensive care unit recovery, and their mean health utility scores and standard deviations were taken from Cuthbertson et al. (2010) at 1, 2.5, and 5 years as 0.666 (0.280), 0.701 (0.281), and 0.677 (0.301), respectively10. Survivors of AKI were found to have a utility of 0.40 with a standard deviation of 0.37 at 60 days. We did not find evidence of improved health utilities over the 5 year period11. The health utility of death was valued at 09.

Transition probabilities

The transition probabilities of patients moving through the decision tree were informed by data from Marik’s study. This includes the probabilities of survival and AKI, see Table 13. In Marik’s study, the bundle’s probability of death and AKI were 0.085 and 0.097, with standard deviations of 0.041 and 0.043 respectively. These compared to the standard care mortality and AKI probabilities of 0.404 and 0.367, with standard deviations of 0.071 and 0.070 respectively.

Table 1. Transition probabilities from bundle decision to survival and recovery.

Mean Standard
Care (SD)
Mean Standard Care,
Scenario 1 (SD)
Mean Marik
Bundle (SD)
Mortality0.404 (0.071)0.212 (0.064)0.085 (0.041)
AKI0.367 (0.070)0.220 (0.062)0.097 (0.043)

Modelling additional scenarios

Scenario 1 uses probabilities from the literature for mortality and AKI; costs and utilities were unchanged. The attributable mortality risk and rate of AKI in the general population vary in the literature and are different to those reported in Marik’s control cohort. We used the mortality risk identified in Rhee et al. (2017) of 21.2% and the rate of sepsis-associated AKI in Angus et al. (2001) of 22.0%12,13. These are lower than the rates seen in Marik’s observation group. We estimated large standard deviations for mortality and AKI at 30% and 28% of the means respectively to include Marik’s treatment group results as possible outcomes within 95% confidence intervals.

Scenario 2 represents a worst-case scenario, in which Marik’s bundle is ineffective and there is no difference in probabilities between the bundle and standard care. We modelled scenario 2 using the same transition probabilities found in Rhee et al. and Angus et al. for both arms of the decision tree. As in scenario 1, costs and utilities remained unchanged.

Probabilistic sensitivity analysis

Probabilistic sensitivity analysis was conducted on the decision tree by taking 1000 random resamples of values from the prior statistical distributions of model parameters. This propagates all uncertainty in the data forward to the results and provides useful information for decision making.

Results

The total economic cost of patients under standard care, including 5 year estimations of RRT, was $116,427 compared to the total economic cost of patients receiving the Marik bundle of $66,744, an expected saving of $49,683 per patient. The annual incidence of sepsis in the United States is 1.5m cases suggesting a total expected cost saving of $74.5bn USD. Over the same 5-year period, patients gain an additional 1.26 QALY per case, or 1.9m QALYs per year for the population. Given the reduced costs and improved outcomes, the Marik bundle dominates standard care, as it both saves costs and increase health outcomes at the same time. The probabilistic sensitivity analysis shows this conclusion arises 98.0% of the time.

Scenario 1 reduces the mean expected economic cost of standard care by $17,166 to $100,244 per patient, compared to the Marik bundle at $67,581. The expected cost saving between the bundle and standard care $32,663 per patient, a total saving of $49.0bn USD, down from $74.7bn. Altering the transition probabilities to values from the literature also reduces the QALYs gained, with patients gaining only 0.55 QALY per case, or 825,000 QALY per year over 1.5m cases per year. In this scenario, the Marik bundle still dominates standard care by saving costs and improving health outcomes. The probabilistic sensitivity analysis shows this conclusion arises 90.8% of the time.

Scenario 2 is that there is no change to mortality outcomes and AKI between Marik’s bundle and standard care. The mean cost per patient over 5 years is $98,932USD for standard care and $99,459USD for the Marik bundle, an increase of $527 per patient over a 5 year period. There is no difference in QALYs between treatment alternatives. We were unable to find any evidence of harm to the patient as a result of the bundle. Hydrocortisone and thiamine are already present in routine sepsis care, and the dosage of 6g of vitamin C per day has been shown to be safe unless contraindicated1416. In this scenario, assuming the intervention is universally adopted, no patients will have been harmed, but healthcare payers and providers would have spent an additional $791m USD per year. Outcomes from the scenario analyses are listed in Table 2 below.

Table 2. Scenario analyses for adoption compared to baseline adoption outcomes.

Cost
savings
95% uncertainty
interval
QALYs
gained
95% uncertainty
interval
Baseline$74.5bn[$1.8bn – $166.5bn]1.9m[0.3m – 3.6m]
Scenario 1: less effective$48.5bn[-$22.7bn – $132.4bn]0.8m[-0.2m – 2.1m]
Scenario 2: ineffective-$0.791bn[-$0.9bn – -$0.7bn]00
Table of distributions used in sepsis model
TypeNameDescriptionMeanStandard Deviation
Normald_c_marikCost of Marik bundle52826.4
Normald_c_RRT_annualAnnual cost of renal replacement therapy7693615387.2
TypeNameDescriptionAlphaLambda
Gammad_c_standardCost of a standard sepsis episode in hospital4.6397357490.000143109
TypeNameDescriptionAlphaBeta
Betad_p_marik_mortalityProbability of death under Marik bundle3.91489362242.08510643
Betad_p_RRT_marikProbability of RRT under Marik bundle4.45161263241.54838435
Betad_p_RRT_standardProbability of RRT under standard care16.8666678729.13333542
Betad_p_standard_mortalityProbability of death under standard care18.5957463427.40425778
Betad_u_recover_1yrUtility of recovery from ICU after 1 year1.2236390820.613656837
Betad_u_recover_2point5yearUtility of recovery from ICU after 2.5 years1.1597780930.494684237
Betad_u_recover_5yearUtility of recovery from ICU after 5 years0.9569804970.45658006
Dataset 1.Sepsis Distributions Table.
Data on distributions used in the analysis

Discussion

We found that adopting the Marik bundle has a high likelihood to save billions of dollars and generate millions of extra QALYs under the conditions outlined in Marik’s paper and in an alternate scenario that uses other data. Under the ineffective treatment of scenario 2 costs are increased to health services by $0.791bn. These results reveal substantial opportunity costs in dollars and lives if we fail to implement and the bundle is ultimately found to be effective, even if the treatment effect is lower than purported by Dr Marik. If the bundle does not work then some costs have been incurred by hospitals for no heath gain. Not adopting the bundle because the evidence for effectiveness is currently uncertain could well be a poor strategy.

The scenario analysis uses values at which the Marik bundle is less effective, specifically by aligning observation group figures with the literature. The mortality rate of sepsis under standard care was suggested by Marik to be 40.4%. Sepsis mortality has been declining in the US, from 46.9% in the early 1990s to 21.2% in 2014, declining by about 3% per annum and driven by improved organ support systems and protocoled early recognition and treatment13,17,18. It might be that Marik’s study featured unusually sick patients in the control group. Study participants are often unrepresentative of the general population and the intervention group may have been less likely to suffer an adverse outcome19. This progress in sepsis care management is complicated by the fact that claims data has shown concurrently increasing sepsis incidence and decreasing mortality, so the literature is conflicted20,21. It is possible that the increased incidence from claims data is due to increased reimbursement received by US hospitals for sepsis compared to other diseases, and patients are being misdiagnosed. An increasing rate of misdiagnosis of patients that do not have sepsis increases the denominator of septic cases while mortality stays the same in the numerator, creating the illusion of declining mortality13,21.

Regarding the mortality of Marik’s treatment group, we note the significant improvement in outcomes recommending the use of steroids in adults with septic shock by Annane et al. (2018)22. Patients randomised to the steroid group (n = 614) showed a 6% absolute reduction in 90-day all-cause mortality when compared to placebo (n = 627). Therefore, the figures in Marik’s patient cohort may have been unusual, but they are plausible.

There is no current body of evidence that suggests this bundle is dangerous. Indeed, the combination of vitamin C and thiamine with steroids would have to cause an attributable death once in every 10 sepsis patients - by a hitherto unimagined and novel mechanism - to negate the modelled benefits of its early adoption.

Limitations

Our study excluded the costs of bundle implementation, including training, labour and the potential for high costs of de-implementing an unsuccessful treatment. These would have increased the cost of bundle implementation, so the incremental cost difference of the Marik bundle may be understated. We noted that hydrocortisone, as part of the bundle, was found by Venkatesh et al. (2018) to speed up resolution of shock and reduce the need for blood transfusions, so the in-hospital cost-savings for the treatment group may also be understated16. There is also considerable uncertainty around the parameter estimates that are available. We did not have access to primary data, including clinical data on costs and transition probabilities for each patient, and were reliant on the literature.

Bundle adoption decision

An average hospital in the US may treat around 230 sepsis patients per year12. By implementing the bundle, it will spend an additional $528 per patient, or $121,440 per year. Conservative estimates from the scenario analysis shows cost savings of $32,644 and a gain of 0.55 QALYs per patient. If the treatment was effective for just four patients out of 230, or 1.7%, it will have paid for itself in terms of total economic costs. Comparing implementing the sepsis bundle to other hospital-based treatment studies shows that for 230 patients, the bundle costs less than a tenth of a standard phase I clinical trial, which run from $1.4m-$6.5m23.

The case for not adopting the Marik bundle has several components. Scientific and empirical evidence is thin, and a single-site Vitamin C trial showing remarkable results only to be proven ineffective after a multi-site RCT is a clinical trope. If the bundle was ineffective, health systems will have added an unnecessary load to clinicians and implementers, which would then have to be de-implemented. Administrators and clinicians may be less likely to adopt novel treatments in the future. Introducing the bundle on the current evidence may also set a bad precedent for novel treatments, eroding the authority of the presiding physician and giving more credence to largely unproven interventions. The proposal in this paper does not replace the need for the clarity provided by good science and empirical research, but these are not always immediately available. We provide an approach to explicitly guide the interim decisions that must be made under these circumstances.

The Marik bundle is a somewhat unusual case relative to most ‘miracle’ interventions later found to be ineffective. As the analysis shows, it is an extremely cheap treatment with the potential to reduce rates of mortality and kidney damage at no risk to the patient. The delay between publication of the pilot study and results of the large RCT due in mid-2018 could create substantial opportunity costs in dollars and lives, and while there is not perfect evidence, under the circumstances it might be sufficient for hospitals and health systems to choose whether to conduct their own trials and not only independently verify their results, but also publish their findings and improve the availability of evidence around the treatment bundle.

The implementation decision of the Marik bundle relies upon the willingness of health administrators to use the available evidence to influence policy. We attempted to make the bundle choice as intuitive as possible, using straightforward trade-offs, simple modelling techniques, and a realistic decision process. Merlo et al. (2014) showed that for research to be accessible to decision-makers, it must be contextually relevant, contain little jargon, and put the terms of the implementation decision into terms that specify a trade-off that will be familiar to decision-makers24. If the decision maker is unwilling to use the best available evidence now, but rather wishes to wait for definitive evidence they are risking incurring large costs for health care systems and for the patients it serves.

Conclusion

An explicit analysis of uncertain clinical outcomes is a useful adjunct to guide decision making where there is clinical ambiguity. This approach offers a valid alternative to the default of clinical inactivity when faced with uncertainty.

Data availability

Dataset 1. Sepsis Distributions Table: Data on distributions used in the analysis. 10.5256/f1000research.14619.d20141125

Comments on this article Comments (0)

Version 2
VERSION 2 PUBLISHED 26 Apr 2018
Comment
Author details Author details
Competing interests
Grant information
Copyright
Download
 
Export To
metrics
Views Downloads
F1000Research - -
PubMed Central
Data from PMC are received and updated monthly.
- -
Citations
CITE
how to cite this article
Blythe R, Cook D and Graves N. Scepticaemia: The impact on the health system and patients of delaying new treatments with uncertain evidence; a case study of the sepsis bundle [version 1; peer review: 2 approved]. F1000Research 2018, 7:500 (https://doi.org/10.12688/f1000research.14619.1)
NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.
track
receive updates on this article
Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?
Key to Reviewer Statuses VIEW
ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.
Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions
Version 1
VERSION 1
PUBLISHED 26 Apr 2018
Views
23
Cite
Reviewer Report 04 Jun 2018
Adam Visser, Toowoomba Hospital, Toowoomba, Qld, Australia 
Approved
VIEWS 23
This fascinating paper uses a simple health economics model to argue that delaying the widespread implementation of the Vitamin C protocol as described by Marik and colleagues would lead to the loss of enormous potential savings to the health industry.  ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Visser A. Reviewer Report For: Scepticaemia: The impact on the health system and patients of delaying new treatments with uncertain evidence; a case study of the sepsis bundle [version 1; peer review: 2 approved]. F1000Research 2018, 7:500 (https://doi.org/10.5256/f1000research.15909.r34658)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
Views
27
Cite
Reviewer Report 11 May 2018
Paul E. Marik, Division of Pulmonary and Critical Care Medicine, Eastern Virginia Medical School, Norfolk, VA, USA 
Approved
VIEWS 27
This is an outstanding paper that objectively evaluates the potential benefit of the vitamin C protocol for sepsis.This evaluation was impeccably conducted and reported. The impact of this paper is significant.

A few comments that will improve ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Marik PE. Reviewer Report For: Scepticaemia: The impact on the health system and patients of delaying new treatments with uncertain evidence; a case study of the sepsis bundle [version 1; peer review: 2 approved]. F1000Research 2018, 7:500 (https://doi.org/10.5256/f1000research.15909.r33910)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.

Comments on this article Comments (0)

Version 2
VERSION 2 PUBLISHED 26 Apr 2018
Comment
Alongside their report, reviewers assign a status to the article:
Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.
Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions
Sign In
If you've forgotten your password, please enter your email address below and we'll send you instructions on how to reset your password.

The email address should be the one you originally registered with F1000.

Email address not valid, please try again

You registered with F1000 via Google, so we cannot reset your password.

To sign in, please click here.

If you still need help with your Google account password, please click here.

You registered with F1000 via Facebook, so we cannot reset your password.

To sign in, please click here.

If you still need help with your Facebook account password, please click here.

Code not correct, please try again
Email us for further assistance.
Server error, please try again.