Double blinding requirement for validity claims in cognitive-behavioral therapy intervention trials for major depressive disorder. Analysis of Hollon S, et al., Effect of cognitive therapy with antidepressant medications vs antidepressants alone on the rate of recovery in major depressive disorder: a randomized clinical trial

Directive psychotherapies are not more effective than non-directive psychotherapies when controlled for researcher allegiance.

We noticed that Dr. Hollon is actually one of the authors of the paper we quoted from 2012, that concluded that directive psychotherapies were not more effective than non-directive psychotherapies when controlled for researcher allegiance, and that the majority of the effect of therapy for adult depression is realized by non-specific factors¹.
 
This is the abstract of this paper [our bolding]:
“The effects of non-directive supportive therapy (NDST) for adult depression have been examined in a considerable number of studies, but no meta-analysis of these studies has been conducted. We selected 31 studies on NDST from a comprehensive database of trials, examining psychotherapies for adult depression, and conducted meta-analyses in which NDST was compared with control groups, other psychotherapies and pharmacotherapy. We found that NDST is effective in the treatment of depression in adults (g = 0.58; 95% CI: 0.45–0.72). NDST was less effective than other psychological treatments (differential effect size g = − 0.20; 95% CI: − 0.32 to − 0.08, p < 0.01), but these differences were no longer present after controlling for researcher allegiance. We estimated that extra-therapeutic factors (those processes operating in waiting-list and care-as-usual controls) were responsible for 33.3% of the overall improvement, non-specific factors (the effects of NDST compared with control groups) for 49.6%, and specific factors (the effects of NDST compared with other therapies) for 17.1%. NDST has a considerable effect on symptoms of depression. Most of the effect of therapy for adult depression is realized by non-specific factors, and our results suggest that the contribution of specific effects is limited at best.”
 
Although Hollon and his coauthors are admitting in the above paper that the unblinded nature of therapists who have some allegiance to the therapy they are trained in can be the cause of positive findings, Hollon et al. now opine in their comment to us that blinding is not important for validation of CT efficacy, and that CT is superior to non-directive supportive therapies by referencing papers published the following year in 2013.^2,3

We are not sure why/how Dr. Hollon views these spins of his own conclusions over this short period of one year, nor how Dr. Hollon can state that psychotherapy studies do not require double-blinding to filter out bias.
 
We also noted Parker, et al.⁴ did a systematic literature search and could not find evidence that CBT was distinctively superior than other psychotherapies for major depression.
 
However, regardless of the issue of directiveness of a therapy, there is no way to get around the problem of not being able to blind the subjects or the treaters in any psychotherapy trial, directive or not.
 
That researcher allegiance is a cause of bias as concluded by a paper by Dr. Hollon himself, it is clear that conclusions must be seriously tempered in CBT trials where neither subjects nor treaters can be blinded, and where the majority of treaters trained in CBT would likely believe in CBT if they continued to be providers of this treatment.



References
1. Cuijpers P, Driessen E, Hollon SD, van Oppen P, et al.: The efficacy of non-directive supportive therapy for adult depression: A meta-analysis. Clin Psychol Rev.2012; 32 (4): 280-291 PubMed Abstract | Publisher Full Text | Reference Source
2. Barth J, Munder T, Gerger H, Nüesch E, et al.: Comparative efficacy of seven psychotherapeutic interventions for patients with depression: a network meta-analysis. PLoS Med. 2013; 10 (5): e1001454 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
3. Braun SR, Gregor B, Tran US: Comparing bona fide psychotherapies of depression in adults with two meta-analytical approaches. PLoS One. 2013; 8 (6): e68135 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
4. Parker G, Fletcher K: Treating depression with the evidence-based psychotherapies: a critique of the evidence. Acta Psychiatr Scand. 2007; 115 (5): 352-359 PubMed Abstract | Reference Source

Blinding and blind placebo are clearly crucial to elucidate efficacy in disorders with subjective endpoints.
 
We appreciate that Hollon et al. share our preference for double-blind controls. However, we are concerned that their logic of how to conduct and interpret the results of a clinical trial for an indication with subjective endpoints that can not be single- or double-blinded (and thus can not have a blind placebo group) does not describe the scientific needs of these kinds of trials adequately. We will address some of the most important points they describe here (we use CT (Cognitive Therapy) and CBT interchangeably to mean the same type of psychotherapy as in our article above in this comment; and Hollon et al. to refer to the paper discussed in our article above, and/or the comment to our paper by Hollon et al. above).
 
It is imperative that we repeat that double blinding is crucial for indications with subjective outcomes as in MDD¹. Only interventional studies for indications with objective endpoints can ignore potential bias from lack of blinding. For example, mortality rates, MI incidence, stroke, etc. where random error is small². This study on bias concluded that unblinded randomized clinical trials tended to be biased to positive effects if the outcomes were subjective as opposed to objective³.
 
To add additional important comment to our paper:

1. non-effective drugs and placebo pills can both show an average 30% improvement in depression scores compared to baseline (not due to improvements from waiting) ⁴. Called a, “placebo effect” this is suspected to be due to the subject’s hope and expectation for improvement⁵.
2. So-called “psychological placebos” can be just as effective as standard psychotherapies in MDD⁶.

It is clear that because of the propensity of persons diagnosed with depression to improve with non-specific factors (i.e., receiving some intervention even if placebo pill or therapy, relationship with a caretaker, participation in a trial, etc.), enhanced improvement can be reported by subjects if there is even a small amount of bias to a specific intervention given in an unblinded fashion. Improvement from this bias will then be reported by the subject to the rater and can add up such that they will become significant with a large N as was the case in the study by Hollon et al. discussed in our paper.
 
We are at a loss as to how Hollon et al., can argue their way out of this conundrum and make claims of efficacy without calling for serious hesitation in the interpretation of results from CBT trials such as in Hollon et al.
 
Regarding their points.
 
Point 1: Single-Blinding: definitions and misuse of terms.
 
Hollon et al. continue to call CBT trials “single-blind” in their comment. We noted in our paper that “single-blind” is defined as the condition when subjects are blind, not the treaters or only the raters⁷.
 
See this reference in the American Psychological Association methodology data base, under, “Treatment Outcome/Clinical Trial”: “It is single-blind when the participants do not know which treatment group they have been assigned to”,  (http://www.apa.org/pubs/databases/training/method-values.aspx, Accessed August 29th, 2015).
 
Using the term “single-blind” for studies that are not actually single-blind under this definition is not using the proper definition of this term and seems to be trying to suggest there is more rigor and control in a CBT study than is actually there.
 
The definition for single-blind is not met for a CBT trial as Hollon et al. acknowledge, so we are confused why they continue to use this term for trials with unblinded subjects. None of the trials they reference have a single-blind psychotherapy arm, and thus they can neither have a blind psychotherapy placebo comparison group. All the subjects and therapists know the psychotherapy provided. As we also noted in our paper, you can not logically compare blinded medications with unblinded psychotherapy and make any logical conclusions. The design of these arms are completely different, and as clearly reference in our paper, unblinded subjects in a study with subjective endpoints can not eliminate bias.
 
Even if Hollon et al. wish to redefine “single-blind” as when only raters are blind as in their study (or any study of CBT), this doesn’t change the fact that blind raters only record whatever bias may be in the subjective reports of the subjects that can be swayed by the unblinded conditions. Emphasizing that raters are blind in a CBT study can distract from the fact that this does not change the potential for bias in the subject-treater system.
 
Point 2: Expectancy effects: uncontrolled data, and post hoc analyses.
 
Their counter to our alternative interpretation of their results that , “It strains the imagination to, for example, attempt to come up with an explanation as to why expectancy effects should be limited to the third of the sample that was recurrent and non-chronic, but more severe”, is confusing because it is extremely easy to imagine the logic pitfalls in making associations here.
 
First and foremost is that the data is based on unblinded/non-placebo control conditions in subjective endpoints. Hollon et al. are trying to extract some kind of association from data born from uncontrol so that this is data on shaky ground (see the references in our paper).
 
Next, Hollon et al. have made a post hoc analysis on multiple variables (severity, acute vs chronic, recurrent or not, etc.) and thus have inflated the possibility of a type I error rate⁸.
 
This is because the further one looks for associations the more likely something will be found. In other words, each time an association in the data is considered, a statistical test is effectively performed. This inflates the total number of statistical tests needed and p-values need to be adjusted, which we did not see. Hollon et al. only did an “exploratory analysis” to determine whether severity and chronicity contributed independently to the increments observed. It is necessary to label these analyses as post hoc to avoid misleading readers that they are statistically robust when they are not.
 
The problems in multiple comparisons is common in the use of “data dredging” (also called “data mining”), to uncover patterns in data that can be presented as statistically significant, without first devising a specific hypothesis as to the underlying causality⁹.
 
The authors tried to justify the post hoc analysis problem by using a Bonferroni correction. The traditional Bonferroni tends to lack power, type II error rates are too high for individual tests, and it overcorrects for Type I error. Other modified Bonferroni approaches are available (not specified as done by the authors), but still do not fully get around the post hoc p value problem (Portland University Data Analysis Course, http://www.upa.pdx.edu/IOA/newsom/da1/ho_posthoc.doc, Accessed September 1, 2015).
 
It is hard for us to comprehend how Hollon et al. can call their post hoc subgroup analysis “exploratory” in the body of their text but yet claim these subgroups were clearly part of the efficacy of cognitive therapy in their Abstract conclusion: “Cognitive therapy combined with ADM treatment enhances the rates of recovery from MDD relative to ADMs alone, with the effect limited to patients with severe, nonchronic depression”.

Regardless of statistics, the Hollon et al. data is born from unblinded and thus uncontrolled study design so that we opine this data should not be acceptable to begin with. The FDA requires all antidepressants to have done some double-blind placebo controlled studies for approval for this reason¹⁰.
 
Point 3: Improved care for patients: The risks to claim improved care from unproven interventions are serious.
 
If an intervention is touted as efficacious in spite of having only unblinded and no placebo control, as we have outlined it is for CBT studies of depression with subjective endpoints, it is risky that patients with depression will seek out this intervention in place of treatments that have much more robust double-blind and placebo controlled data with approvals from a regulatory agency such as the FDA. These patients may sustain greater morbidity and mortality, and the financial costs to society can be large.
 
While few U.S. psychiatrists are unfamiliar with Osheroff v. Chestnut Lodge, we feel it is important enough to cite an abstract by Klerman in full here¹¹. Hollon et al., will undoubtedly opine that CBT is a much more rigorously studied psychotherapy than the therapy Osheroff received, however, our paper, and other papers referenced below (references 13, 14, 18 discussed in Point 4 below), have presented strong logic to question the entire basis of CBT efficacy studies. We strongly agree with Klerman below that the, “…right of the patient to effective treatment and that treatments whose efficacy has been demonstrated have priority over treatments whose efficacy has not been established”.
 
Abstract from Klerman¹¹:
 
“Although Osheroff v. Chestnut Lodge never reached final court adjudication, the case generated widespread discussion in psychiatric, legal, and lay circles. The author served as a consultant to Dr. Osheroff and testified that Chestnut Lodge failed to follow through with appropriate biological treatment for its own diagnosis of depression, focusing instead on Dr. Osheroff's presumed personality disorder diagnosis and treating him with intensive long-term individual psychotherapy. The author suggests that this case involves the proposed right of the patient to effective treatment and that treatments whose efficacy has been demonstrated have priority over treatments whose efficacy has not been established.”
 
Point 4: Listing of the many efficacy studies for CBT.
 
Hollon et al. list numerous studies they have done to show that CBT is efficacious. The logic here is, there are many studies with lack of single- or double-blinding control and blind placebo, these studies have shown CBT to be efficacious, thus, the data born from these studies even if not born from double-blinding control and blind placebo is robust; i.e., data from many studies with this design problem makes up for each one of them having this design problem individually. The trouble in the logic here should be obvious.
 
There was no blind psychotherapy placebo, and no double-blind, nor single-blind study (as defined above) in any of the studies listed. Basing conclusions on data born from uncontrolled (unblinded and no placebo) circumstances in subjective endpoints, and then making absolute conclusions from this, and furthermore, comparing this data to medication trials born from double-blind placebo controlled studies is a tortuous argument because the data is born from such widely different level of control.
 
Basing results on data born without single- or double blind controls and blind placebo fits with Ioannidis’ Corollary 4⁹: “The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true.”
 
The subgroup analysis Hollon et al., noted in their reference #7 was also a post hoc analysis¹². The authors in this study even went to great pains to take some “middle ground” analysis in order to avoid conducting “…approximately 150 hypothesis tests, from which we could have expected to find several statistically significant results by chance”. Reference #7 was also an analysis of an unblinded, non-placebo controlled psychotherapy study, and again breaking clinical trial design logic by comparing blinded medication therapy with unblinded CBT. While it is obvious that we must be very careful in concluding much from these kinds of results, Hollon et al., continue to push conclusive remarks in their comment to our article.
 
There is considerable and growing concern that CBT is not an effective treatment for many conditions it touts itself to be. We noted the Lynch et al. meta analysis of CBT trials that controlled for blinding and placebo in our paper¹³. The results of this large meta analysis showed that CBT fared no better than non-specific control interventions in the treatment of schizophrenia and did not improve relapse rates, CBT showed no effect in prevention of bipolar disorder episodes, and only small treatment effects were seen in studies of MDD. We also discuss Parker¹⁴ below who found a low evidence base for the specificity and superiority of CBT over other psychotherapies. 
 
Regarding the point that “cognitive therapy is about as efficacious as medications when adequately implemented in the acute phase of treatment and it has an enduring effect that medications cannot match” (their reference # 8)¹⁵. Long-term or prevention studies of CBT can still only be unblinded studies. CBT more clearly requires subject participation then a non-directive approach and is provided by trained therapists in an open fashion. It is obvious that bias potential is not removed from these studies. We have described above the obvious flaw of logic to compare medication arms or studies carried-out with double-blind placebo with unblinded CBT outcome results because the design of these methods are so different.
 
Regarding Hollon et al.’s comment that , “In this context, findings that CT is superior to non-directive supportive therapies (their reference #s 9,10) suggest the intervention is more than just a placebo.”^[ref-16-17]
 
A meta-analysis found no differences between directive or non-directive psychotherapy intervention when controlled for researcher allegiance, and that most of therapy effects came from by non-specific factors.¹⁸
 
In addition, Parker, et al.¹⁴ did a systematic literature search and concluded, “There is no high level evidence base suggesting or identifying CBT and IPT [Interpersonal Psychotherapy] as distinctively superior than other psychotherapies for major depression. Claiming that they are evidence based, implicating or stating their superiority, and formalizing such differential status in therapeutic guidelines is disingenuous at best and risks distorting the value of an evidence based approach to Psychiatry. The clinical implications of over-selling CBT and IPT as specific and superior treatments for depression impact on both patients and practitioners”.
 
Remember, no study comparing a non-directive therapy to CBT can be blinded. The subjects and CBT practitioners would know they received CBT, and any preference for a directive approach would not be filtered by the unblinded nature of these studies.
 
Point 5: Use of study in Astronomy and Evolutionary Biology as an example that double-blinding and placebo control are unnecessary aspects of study design to claim efficacy in an intervention of a human disorder with subjective endpoints.
 
Hollon et al. have confused the use of data collection to make theories in natural science vs. the making of efficacy claims in a clinical trial of a human condition that has subjective endpoints. The argument about natural science theory making is not applicable to the conduct of clinical trial that is comparing the outcome of an intervention vs. some control group and then making efficacy claims. Natural scientists do not claim that their conclusions or theories should be used to treat a human disorder with subjective endpoints. This is diversionary logic that is extremely hard for us to understand in the setting of a discussion about a clinical trial of a human disorder.
 
Point 6: Competing Interests, potential researcher bias because of grants and academic interests.
 
Regarding potential researcher bias, the paper in question by Hollon et al., and many of the papers referenced in the comment above by Hollon et al. receive research grant monies. These researchers have academic positions and have been researching CBT and comparing CBT to medications under unblinded/no blind placebo control conditions, usually reporting favorable outcomes as referenced in the Hollon et al. comment (see grant information sections in the Hollon et al. comment references, and Collorary 5 in Ioannidis⁹).
 
In Conclusion
 
1. There is really no way to get away from the problem in the inability to single- or double-blind, and have blind placebo control in a CBT clinical trial, whether in the acute, prevention, or long-term treatment phase of these trials.
 
2. Hollon et al. wrongly use the term single-blind, and do not seem to provide an understanding of the bias that can come from the subject-treater system that can not be filtered by blind raters.
 
3. Problems in doing post hoc analyses, and the comparing of unblinded CBT arms to blinded medication arms (the “apples to oranges” comparison of arms of completely differing control levels) run frequently in the trials noted by Hollon et al. who do not provide an understanding of the serious breech of scientific methodology inherent in these issues.
 
Hollon et al. also call their post hoc subgroup analysis “exploratory” in the body of their text but yet claim these subgroups were clearly part of the efficacy of cognitive therapy in their Abstract conclusion.
 
These points are extremely difficult for us to understand the scientific logic of, much less make strong statements of efficacy in the Abstact and Conclusions of Hollon et al. and related papers referenced in their comment.

4. A number of papers have noted problems in blinding of CBT trials as well as the specificity of CBT trials, and directive vs non-directive approaches, referenced in our reply here, and other papers that this comment does not have the space to discuss. Our paper here is not the only concern about the validity and claims of CBT.
 
We confidently stand by our assertion that the authors of these studies should use extreme caution and state clearly the problems with making any conclusions that can be drawn from CBT outcome studies that are unblinded and have no blind placebo control.
 


References
1. Schulz KF, Grimes DA: Blinding in randomised trials: hiding who got what. Lancet. 2002; 359 (9307): 696-700 PubMed Abstract | Publisher Full Text | Reference Source
2. Piantadosi S: Clinical Trials: A Methodologic Perspective, Second Edition. Book. 2005; Wiley Series in Probability and Statistics (New York). Publisher Full Text | Reference Source
3. Wood L, Egger M, Gluud LL, Schulz KF, et al.: Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008; 336: 601-605 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
4. Marchesi C, De Panfilis C, Tonna M, Ossola P: Is placebo useful in the treatment of major depression in clinical practice?. Neuropsychiatr Dis Treat. 2013; 9: 915-920 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
5. Rutherford BR, Mori S, Sneed JR, Pimontel MA, et al.: Contribution of spontaneous improvement to placebo response in depression: a meta-analytic review. J Psychiatr Res. 2012; 46 (6): 697-702 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
6. Wampold BE, Minami T, Tierney SC, Baskin TW, et al.: The placebo is powerful: estimating placebo effects in medicine and psychotherapy from randomized clinical trials. J Clin Psychol. 2005; 61 (7): 835-854 PubMed Abstract | Publisher Full Text | Reference Source
7. Friedman LM, Furgerg CD, DeMets DL: Fundamentals of Clinical Trials, Third Edition. Springer. 1998. Reference Source
8. Jaccard J, Becker MA, Wood G: Pairwise multiple comparison procedures: A review. Psychological Bulletin. 1984; 96 (3): 589-589 Publisher Full Text | Reference Source
9. Ioannidis JPA: Why Most Published Research Findings Are False. PLoS Medicine. 2005; 2 (8): e124 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
10. Hanrahan C, New JP: Antidepressant Medications: The FDA-Approval Process and the Need for Updates. Ment Health Clin. 2014; 4 (1): 11-16 Publisher Full Text | Reference Source
11. Klerman GL: The psychiatric patient's right to effective treatment: implications of Osheroff v. Chestnut Lodge. Am J Psychiatry. 1990; 147 (4): 409-418 PubMed Abstract | Publisher Full Text | Reference Source
12. Fournier JC, DeRubeis RJ, Shelton RC, Hollon SD: Prediction of response to medication and cognitive therapy in the treatment of moderate to severe depression. J Consult Clin Psychol. 2009; 77 (4): 775-787 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
13. Lynch D, Laws KR, McKenna PJ: Cognitive behavioural therapy for major psychiatric disorder: does it really work?A meta-analytical review of well-controlled trials. Psychol Med. 2010; 40 (1): 9-24 PubMed Abstract | Publisher Full Text | Reference Source
14. Parker G, Fletcher K: Treating depression with the evidence-based psychotherapies: a critique of the evidence. Acta Psychiatr Scand. 2007; 115 (5): 9-24 PubMed Abstract | Publisher Full Text | Reference Source
15. Hollon SD: Cognitive and behavior therapy in the treatment and prevention of depression. Depress Anxiety. 2011; 28 (4): 263-266 PubMed Abstract | Publisher Full Text | Reference Source
16. Barth J, Munder T, Gerger H, Nüesch E, et al.: Comparative efficacy of seven psychotherapeutic interventions for patients with depression: a network meta-analysis. PLoS Med. 2013; 10 (5): e1001454 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
17. Braun SR, Gregor B, Tran US: Comparing bona fide psychotherapies of depression in adults with two meta-analytical approaches. PLoS One. 2013; 8 (6): e68135 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
18. Cuiipers P, Driessen E, Hollon SD: The Efficacy of non-directive supportive therapy for adult depression: a meta-analysis. Clin Psychol Rev. 2012; 32 (4): 280-291 PubMed Abstract | Publisher Full Text | Reference Source

Doubling-blinding is not a requirement for validity claims: A response to Berger. Hollon, S. D., DeRubeis, R. J., & Lorenzo-Luaces, L

We agree with the authors that it is virtually impossible to conduct a double-blind trial of psychotherapy. The patient typically knows what he is getting and the treating clinician knows what is being offered. Like the authors, we have more confidence in inferences drawn from placebo-controlled trials that are conducted in a double-blind fashion, all else equal.
However, the findings from so-called “single-blind” trials, like the one that we conducted, where independent evaluators were kept blind to treatment condition, should not be dismissed when they have potential clinical implications. It strains the imagination to, for example, attempt to come up with an explanation as to why expectancy effects should be limited to the third of the sample that was recurrent and non-chronic, but more severe. In our trial, those patients showed a 30% increment in recovery rates whereas patients who were chronic or less severe showed virtually no additive effect of additional CBT.¹ These findings need to be replicated before they influence clinical practice, though they echo prior findings by Thase et al.² If the findings are replicated, they have the potential to guide improved care for patients with non-chronic depression, consistent with the goals of personalized care.
We share with the authors a preference for double-blind controls. In an earlier trial we found cognitive therapy (CT) roughly comparable to medication treatment and both superior to pill-placebo, CT at the level of a nonsignificant trend.³ We also found evidence that prior CT and continuation medication reduced rates of relapse relative to medication withdrawal, triple-blinded with respect to medications and single-blind with respect to prior CT.⁴ The advantage for prior CT over prior medication is a relatively robust phenomenon but one that depends on single-blind comparisons.⁵ We also found that CT led to greater increments in employment than medication treatment, despite producing comparable reductions in depression.⁶ If the authors are correct that cognitive therapy’s effects are a consequence of nonspecific expectancy effects, it is interesting that those nonspecifics reduce risk for relapse and enhance employability. Moreover, we found evidence that specific subgroups of patients exist who preferentially responded to CT vs. medication above and beyond their average improvement (and the converse), a pattern of findings that is difficult to reconcile with the notion that CT is an inert intervention.⁷
The authors have a valid point, double-blind controls, when they are actually maintained, increase the confidence that we can have in the inferences that can be drawn. We disagree with the authors that inferences cannot be drawn in their absence; it is simply that such a design feature limits the level of certainty about the source of the benefit evidenced by patients in the CT condition. Broad findings from the meta-analytic review comparative literature are quite clear; cognitive therapy is about as efficacious as medications when adequately implemented in the acute phase of treatment and it has an enduring effect that medications cannot match.⁸
In this context, findings that CT is superior to non-directive supportive therapies^9,10 suggest the intervention is more than just a placebo. Other directive psychological treatments are also efficacious in the treatment of depression but the is no debate among researchers that CT is the most widely studied form of psychotherapy and has the most well-established short and long-term evidence base. Confidence in these conclusions would be even greater if they had come from studies conducted using double-blind designs, but this issue echoes across the sciences. Most areas of Astronomy and Evolutionary Biology, for example, have progressed despite similar limitations. Indeed it would be good if one could conduct a double-blind study of a psychotherapy, just as it would be good if one could travel back in time to observe the evolution of the species or back to the moment of the Big Bang to observe the processes that occurred then. Even in these counterfactual fantasies one still would not be able to use some of the other tools of science, such as randomization and experimental manipulation, which would aim to rule out a variety of alternative accounts.


References
1. Hollon SD, DeRubeis RJ, Fawcett J, Amsterdam JD, et al.: Effect of cognitive therapy with antidepressant medications vs antidepressants alone on the rate of recovery in major depressive disorder: A randomized clinical trial. JAMA Psychiatry. 2014; 71 (10): 1157-1164 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
2. Thase ME, Greenhouse JB, Frank E, Reynolds CF 3rd, et al.: Treatment of major depression with psychotherapy or psychotherapy-pharmacotherapy combinations.Arch Gen Psychiatry.1997; 54 (11): 1009-1015 PubMed Abstract | Publisher Full Text | Reference Source
3. DeRubeis RJ, Hollon SD, Amsterdam JD, Shelton RC, et al.: Cognitive therapy vs medications in the treatment of moderate to severe depression.Arch Gen Psychiatry. 2005; 62 (4): 409-416 PubMed Abstract | Publisher Full Text | Reference Source
4. Hollon SD, DeRubeis RJ, Shelton RC, Amsterdam JD, et al.: Prevention of relapse following cognitive therapy versus medications in moderate to severe depression. Arch Gen Psychiatry. 2005; 62 (4): 417-422 PubMed Abstract | Publisher Full Text | Reference Source
5. Cuijpers P, Hollon SD, van Straten A, Bockting C, et al.: Does cognitive behaviour therapy have an enduring effect that is superior to keeping patients on continuation pharmacotherapy? A meta-analysis.BMJ Open. 2013; 3 (4): e002542 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
6. Fournier JC, DeRubeis RJ, Amsterdam JA, Shelton RC, et al.: Gains in employment status following antidepressant medication or cognitive therapy for depression.Br J Psychiatry.2015; 206 (4): 332-338 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
7. Fournier JC, DeRubeis RJ, Shelton RC, Hollon SD, et al.: Prediction of response to medication and cognitive therapy in the treatment of moderate to severe depression.J Consult Clin Psychol.2009; 77 (4): 775-787 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
8. Hollon SD: Cognitive and behavior therapy in the treatment and prevention of depression. Depress Anxiety.2011; 28 (4): 263-266 PubMed Abstract | Publisher Full Text | Reference Source
9. Barth J, Munder T, Gerger H, Nüesch E, et al.: Comparative efficacy of seven psychotherapeutic interventions for patients with depression: a network meta-analysis.PLoS Med.2013; 10 (5): e1001454 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source
10. Braun SR, Gregor B, Tran US: Comparing bona fide psychotherapies of depression in adults with two meta-analytical approaches.PLoS One. 2013; 8 (6): e68135 PubMed Abstract | Free Full Text | Publisher Full Text | Reference Source