The ACCE method: an approach for obtaining quantitative or qualitative estimates of residual confounding

1.1. Differences in control of confounding from included covariates

Although these changes may be expected to be minor, at least in some settings, they deserve thorough consideration as part of an effort to anticipate sources of potential imprecision in the estimates resulting from the ACCE method. Some degree of change between Model 1 and Model 2 is expected in the balance of each of the propensity score covariates present in common between Model 1 and Model 2 (the “included covariates”). These changes are produced as a byproduct of the need to include an additional variable or variables in Model 2 to generate confounding amplification.

(NOTE: The additional “introduced variable” is also an “included” covariate in a limited sense, in that it is included in one of the two propensity scores. For clarity in terminology and because the introduced variable represent a genuinely special circumstance (please see Appendix 2.2), the term “included variable” or “included covariate” is reserved for the variables included in both models. The term “introduced variable(s)” is used for the variable(s) added to Model 2. The term “nonincluded covariate” or “nonincluded variable” will refer to covariates not included in either propensity score.

This change can be best visualized by considering the case of propensity score matching: including an additional variable in the propensity score used for matching would be expected to weaken at least slightly the tightness of the match on the other covariates. In some cases, the differences in covariate balance between models could be quite minimal. However, this balance needs to be explicitly compared between Model 1 and Model 2. This comparison is important because in some cases it may prove difficult to attain a control of confounding from included covariates that is strictly equivalent between Model 1 and Model 2 if sufficient confounding amplification is to be achieved. The impact of confounding amplification is to tend to create a greater number of individuals at the extremes of the propensity score distribution who are less comparable (and thereby, less similar in balance in the covariates included in the model)¹.

If these differences are not trivial, one approach possibly worthy of future exploration would be to examine whether it is feasible to adjust the stringency of the stratification or matching in Model 2 so that the balance in the included covariates in Model 1 and Model 2 are more equivalent. Another would be to use the Bross equation⁸ to attempt to estimate the change in confounding attributable to the observed changes in the balance of these measured covariates.

1.2. Differences in patient sample

The overall patient cohort for the study from which the samples for Models 1 and 2 are derived will obviously not change. Some degree of change, however, can be anticipated to occur (although it may be minimal), in the samples of individuals selected from that overall cohort by Model 1 and Model 2. These differences can arise from differences in the patients that fall under the “Common Support Area” of the propensity scores, and, if matching is employed, differences in the percent of patients matched. The “Common Support Area” refers to the range of propensity score values which include members of both treatment groups; it is often recommended that individuals outside the Common Support Area be “trimmed” (i.e., removed) from the analysis¹⁶. These differences in patient sample, however, will only influence the method’s estimates to the extent that they are extensive enough to produce substantively different compositions of patients between the Models and effect modification exists (whereby the treatments studied have different effects in different patients). In addition, possible strategies exist to minimize some of these potential differences. These are discussed below.

1.2.a. Differences in patient sample from differences in common support area/propensity score trimming. Because confounding amplification tends to make at least patients on the extremes of the propensity score distribution less comparable⁵, it might prove difficult in practice to maintain a highly similar Common Support Area between Model 1 and Model 2. Fortunately, the number and identity of individuals differing between Model 1 and Model 2 is measurable. In addition, different approaches might be compared (such as examining in both models only the subset of patients that fall under the Model 2 Common Support Area, or all the patients that fall under either Model’s Common Support Area). These comparisons might establish the sensitivity of the results to small differences in the Model 1 and Model 2 patient samples arising from different Common Support Areas.

1.2.b. Differences in patient sample in the two models from differences in percent matching. Regarding matching strategies, it may prove difficult for a similar proportion of matching to be preserved between the two models. (The Brooks and Ohsfeldt simulation⁵ showed that as unexplained variance decreased and amplification increased, the number of patients matched for a given caliper decreased). The alternative approach, propensity score stratification, may be determined to be preferable in some cases, since by design stratification retains all individuals from the trimmed sample. (The ACCE Method emphasizes stratification and matching because in the Brooks and Ohsfeldt simulation¹ propensity score weighting produced confounding amplification that was less predictable, at least by R²).

1.3. Differences in specific aspects of the intervention received

While the general nature of the interventions received by the two treatment groups remains identical between Model 1 and Model 2, specific aspects of the intervention received can vary between the treatment groups in ways that are not immediately obvious. It is important to consider these possible differences because they may be another, non-amplification related, contributor to differences in the treatment effect estimates between Model 1 and Model 2.

1.3.a. Differences in dose. Unless the intervention is a single, one-time-only dosed treatment, such as a vaccine, either dose or other “quasi-dose” aspects of how the intervention is administered may vary at least slightly between the individuals receiving the intervention in Model 1 and those receiving the intervention in Model 2. Even for nonmedication-based interventions, such as a psychotherapy or educational intervention, the timing of visits or the number of visits may vary slightly among the individuals included in the intervention arm in Model 1 versus Model 2. Therefore, when implementing the ACCE method, it may prove important to examine whether the overall mean dosage, number, or timing of treatments is similar between Model 1 and Model 2, and potentially within strata for stratified analyses. If sufficient sample size exists in particularly large patient samples, an additional approach might be to restrict the analysis to patients only receiving one particular dosage of the medication.

1.3.b. Differences in discontinuation rates. Treatment effect estimates are likely to be sensitive to discontinuation rates, whether an intent-to-treat or as-initially-treated analysis (i.e., with follow-up censored upon alteration of the initial treatment) is conducted. Because of this, investigators should examine the rates of discontinuation observed in Model 1 and Model 2 to determine their similarity. Ideally, a determination that the reasons for discontinuation within the patient sample for Model 1 compared to Model 2 were also similar is the ideal, but such information is often not available¹⁷.

In many cases the difference in discontinuation rates for the treatment of interest between the two models may be quite small, and the practical impact of this difference unclear. Differences in discontinuation rates appear to be at least slightly more significant in this approach, however, than in a propensity score or regression analysis involving a single model. If any effect modification exists, then changes in the group remaining in treatment produced by differences in discontinuation between the two models could produce some degree of difference in the underlying treatment effect estimate between the models, even if the factors governing discontinuation in both cases were not related to outcome.

Addressing specific differences in the intervention received by patients between the models, if they exist, may be difficult. One modest strategy might involve simply evaluating the differences in intervention specifics when different strategies are explored to minimize differences in patient sample and/or in the control of confounding from included covariates. Then the approach that also minimizes differences in intervention specifics might be favored, or at least examined as a sensitivity analysis.

As indicated in the main manuscript (Endnote A), this manuscript generally does not consider confounding arising after treatment initiation from differences in patient characteristics of patients who remain receiving in the two treatment groups. However, three points should be made. First, what this means is that the “unconfounded treatment effect estimate” that is estimated from this method is not necessarily estimating a completely unconfounded effect estimate, in the strictest sense of the term. Rather, the method seeks to provide a treatment effect estimate unconfounded from baseline confounding. Second, if confounding between the treatment groups exists that arises after treatment initiation but is similar between the two models (which may be somewhat plausible, for instance, if discontinuation rates vary little between the two models), then a treatment effect estimate largely unconfounded from baseline confounding, at least, can still be estimated. (This estimate can occur since the confounding post-initiation would be expected to stay relatively constant and not be a major source of differences in the treatment effect estimates between the two models). Third, it is conceivable that the same approach used here – deliberately introducing confounding amplification to estimate the original confounding present – could be used, at least in theory, to also estimate confounding post-initiation, occurring from differential discontinuation during treatment. This is likely to be a substantially harder and more complex endeavor, however, since the most commonly used approach to addressing post-initiation confounding, generating a “pseudopopulation”, usually occurs by weighting, which has not been shown in simulation to produce very predictable confounding amplification⁵. Matching instead of weighting could be considered, but the results would become applicable to only a smaller and smaller subset of patients. One relatively simple pragmatic approach might be to conduct this matching at only a single additional time point – study completion. Clearly, much additional work is needed to fully understand if the ACCE method could also help address confounding occurring after treatment initiation.

Appendix 2.1. Approaches to estimating confounding amplification

In the lower ranges of exposure prediction (at least as measured by explained variance in terms of R²), a simulation has shown that a predictable relationship exists between amount of remaining unexplained variance in the prediction of exposure and confounding amplification⁵. Differences in prevalence between treatment arms in covariates that are not included in the propensity score increase linearly with increases in R². This increase in the imbalance of uncontrolled (i.e., nonincluded) factors is the phenomenon underlying the amplification of residual confounding. However, in this simulation in the upper portion of the range of R² the relationship becomes increasingly nonlinear, with changes in R² underestimating the increased imbalance in nonincluded covariates⁵. It is unclear whether this nonlinearity relates to particulars of how the simulation was designed; therefore it is difficult to judge whether similar nonlinearities will continue to be observed, even for the R² metric of exposure prediction. However, since the Brooks and Ohsfeldt simulation is the only data currently available, it is worth considering the implications of possible nonlinearity in the upper portions of the range of exposure prediction in detail.

If this nonlinearity in the upper portions of the range continues to be observed when using R², but is reduced or not apparent for other metrics of prediction of exposure, then these metrics should be preferred. If this nonlinearity in the upper ranges of exposure prediction continues to hold for other metrics, then three strategies suggest themselves. The first approach, the “low amplification strategy”, would be to deliberately limit Models 1 and 2 so that the prediction of exposure these models achieve are in the lower end of the possible range, where the relationship is most linear. In some cases, propensity score models may already fall into this range. In other cases, this approach may involve reducing or minimizing the variables included in the propensity scores. Such reduction might entail including only variables with a significant a priori expectation, based on evidence, of being confounders¹⁸. An even more restrictive strategy would including those variables estimated (by using the Bross equation⁸) to be the most substantial confounders, or suspected a priori as being the most certain confounders (e.g., age, Charlson Comorbidity index, etc.). Reductions in the number of included covariates could increase residual confounding relative to some other models that could be constructed. To the degree genuine confounders were removed from the propensity score, the analysis would become increasingly dependent on the ACCE method to capture this larger amount of residual confounding and accurately remove it in order to obtain an unconfounded treatment effect estimate.

However, at least two other strategies suggest themselves that may prove feasible. One alternative would be to develop a formula that captures any nonlinearity in the chosen metric of exposure prediction. This could permit the amount of expected amplification to be relatively accurately predicted over larger portions of the range. The second alternative strategy would be to develop an “internal marker” covariate that would reflect how much increased imbalance in the nonincluded confounders is occurring. The internal marker would be a measured covariate deliberately left out of the propensity scores. The increase in its imbalance in Model 2 could be measured and serve as an indicator of confounding amplification.

Intuitively, an internal marker strategy has some attractive qualities, since it might sidestep any uncertainties about the relationship between confounding amplification and metrics of prediction of exposure. There is also already is some evidence to support this approach. In the Brooks and Ohsfeldt simulation study¹, it was shown that covariates not included in the propensity score and that are uncorrelated with the included covariates all amplify to a remarkably similar extent, at least in that simulation. Thus, in principle, it appears feasible to use an “internal marker”, produced by withholding one (or a few) measured covariates from the Model 1 and Model 2 propensity scores, to track and estimate the general amount of confounding amplification. A key practical consideration in real-world datasets, however, is the need for these internal markers to have a minimal correlation with any of the covariates included in the propensity scores. Any such correlations might “constrain” the ability of the internal marker to reflect the degree of confounding amplification that is influencing the nonincluded confounders. (As an aside, these included-nonincluded covariate correlations would also similarly influence the confounding amplification of the nonincluded covariates that are not being used as internal markers. This interference, however, does not appear (based on present information) to be problematic for the method, for reasons discussed in Appendix 2.2).

Since some degree of correlation is unavoidable, then in the strictest sense the internal marker strategy may intrinsically underestimate true confounding amplification, although potentially only minimally.

Fortunately, this correlation is readily measurable. Therefore it should be possible to deliberately select the nonincluded measured covariate with the least correlation with the included covariates and the introduced variable to serve as an internal marker. Alternatively, simulation research may suggest quantitative approaches to correct for this correlation.

In summary, multiple aspects of confounding amplification are worthy of investigation. These include the presence and predictability of nonlinear relationships between the metric estimating the prediction of exposure and confounding amplification, and the potential strategies to address these nonlinearities.

Appendix 2.2. An initial exploration of the impact of correlation on confounding amplification

The data that exists to date from the Brooks and Ohsfeldt simulation⁵ indicates that confounding amplification is uniform between simulated covariates that were not included in the propensity score. However, this similarity may be a byproduct of their simulation. In theory, aspects of real-world data might create heterogeneities in amplification between covariates. One aspect, the effect upon confounding amplification of correlations between covariates (expected to be a common feature of real-world data), is considered below. This conceptual exploration of the impact of correlations, which also draws upon aspects of the Brooks and Ohsfeldt simulation⁵, tentatively concludes that most correlations do not appear to interfere with the ACCE Method. A special case is posed by correlations between nonincluded covariates and the introduced variable. In that case, at least a partial remedy is intended to be provided by the adjustments performed in Steps 3 and 4 of the method.

Correlations can be categorized into five types, based on whether the correlated covariates are or are not included in the propensity score model. Correlations can exist between two nonincluded covariates, between a nonincluded and an included covariate, between nonincluded covariates and the introduced variable(s), between included covariates and the introduced variable(s), and between two included covariates. For the latter two categories, substantial amplification involving either of the correlated variables is not expected, at least in the same sense as the term applies to nonincluded covariates. Correlations between two nonincluded covariates also do not appear to be problematic. Both of the correlated variables would constitute part of the residual confounding being amplified, and thus be expected to be amplified to a similar extent, based on the Brooks and Ohsfeldt simulation⁵.

Correlations between included covariates and nonincluded covariates, however, seemingly could pose the possibility of creating constraints to amplification for certain nonincluded covariates. The measured covariates that are included in the propensity score cannot amplify substantially, and thereby might seem to constrain amplification, to a degree, of the correlated nonincluded variable. Upon further consideration, however, it appears that while the correlated nonincluded variable would be expected to have an overall change in imbalance in Model 2 that is less than the estimated confounding amplification, this might not interfere with the method’s performance. The implications of the Brooks and Ohsfeldt simulation suggests that the change in a correlated variable can be alternatively modeled as a fraction that is largely unchanging between the model (to the degree that it is correlated with included covariates whose balance is not appreciably changing), and a fraction (alternatively, a “residual”) that amplifies as much as any uncorrelated nonincluded covariate (Reference 2, Supplemental Appendices). (These elements will be termed the “included” fraction and “nonincluded” fraction, of the nonincluded covariate, respectively).

It is important to recognize that part of the goal for the nested models in the ACCE Method is to minimize change in the balance of included covariates, to the extent feasible (Appendix 1.1). If the nested models do successfully exhibit little change in the balance of included covariates, then the Brooks and Ohsfeldt simulation⁵ would suggest that the amount of imbalance observed in the “included fraction” of the correlated, non-included variable also would not change substantively between models. This would leave the amplification of the nonincluded fraction (which Brooks and Ohsfeldt have observed amplifies as completely as for the uncorrelated, nonincluded covariates)⁵ as the only contribution to the change in treatment effect estimate attributable to this covariate. Thus, in principle, the method would still provide accurate final effect estimates. Further research regarding this conclusion, however, would clearly be beneficial.

This consideration of the effects of included covariate-nonincluded covariate correlations reinforces the need to keep the balance of the included covariates as similar as possible between the two models. It also brings to attention the fact that a small fraction of confounding would be expected to exist, attributable to the residual imbalance in the included covariates (and the correlated fraction of their nonincluded covariates), that is not detected because it does not amplify. Such confounding would be somewhat addressable by achieving as close a balance as feasible between treatment groups for any variable included in the propensity score models. The need for tight control of any included variables could also conceivably be yet another practical reason for limiting the included variables in the propensity scores (as discussed in Appendix 1.2.c and Appendix 2.1). Another alternative might be to include all of the included propensity score covariates, as is done in Step 3, in the outcome model estimating the Model 1 and Model 2 treatment effects (Step 1 of the method). This might largely address residual confounding arising from the remaining imbalance in these included covariates.

Correlations between nonincluded covariates and the introduced variable are a particularly distinct circumstance that can be considered as a special case. In this case, the introduced variable is an included variable in only one Model (Model 2). As a consequence, its balance is being deliberately changed from Model 1 to Model 2 to produce the needed confounding amplification. The “included fraction” of any nonincluded covariate relating to its correlation with the introduced variable(s) is now being controlled in Model 2 and thus more closely balanced in Model 2 than in Model 1. This is because in Model 1 the nonincluded covariate’s correlate, the yet-to-be introduced variable, is not controlled at all. However, it is partly for this circumstance that the Step 3 and 4 procedure was designed involving deriving regression coefficients and applying the Bross equation⁸. The intent of Steps 3 and 4 is that the change in confounding of the introduced variable is estimated, along with the change in confounding resulting from the change in imbalance in the fraction of the correlated variable(s) that is balanced as fully as the introduced variable. Whether the effect of correlation upon regression coefficients, however, is sufficiently similar to the effects of correlation on the balancing of covariates using a propensity score, to permit a generally effective adjustment in typical practice has not been determined. Conceptually, this regression-based approach appears to be a reasonable strategy to provide at least some first approximation of the changes in confounding associated with the correlated variables, but the strict accuracy of this approximation is unclear.

This is an area of the ACCE method in which further research would be particularly beneficial. The comparability of the quantitative effects of correlation on regression coefficients versus on covariate balance in propensity score analyses could be examined further through simulation, and perhaps theoretically through frameworks based on the general location model¹⁹ or other methods. Such simulations would helpfully allow the strength of the correlation between correlated variables and the amount of confounding amplification existing between the two models to be varied. Finally, a valuable role in validation may exist for real-world studies as well. These studies could investigate how frequently the ACCE Method provides what appears to be improved treatment effect estimates (i.e., closer to the result that is expected based on randomized trials)¹⁵ than typical propensity score or regression methods. Any imprecision in the ability of the adjustment in Steps 3 and 4 to adequately reflect the change between the models in confounding attributable to the covariates correlated with the introduced variable would depend on the number and strength of such correlations. While this is impossible to quantify the correlations present for truly unmeasured covariates in real world data, it may turn out, based on the comparisons to randomized data discussed above, that in practice these correlations often do not appear to be numerous or strong enough to substantially affect the method’s estimates. Also, even if the regression coefficient-based adjustment was ultimately shown to only poorly capture the effects of correlation, it is possible this may not interfere markedly with the overall accuracy of the method if most of the residual confounding is not correlated with the introduced variable. Further research is clearly warranted.

Even in the “worst case” scenario in which the adjustments in Step 3 and 4 of the method do not perform well and comparisons with randomized data suggested that this limitation typically impairs the method’s estimates substantially, three special circumstances exist in which the ACCE Method’s performance would not be generally expected to be adversely impacted by this limitation. These special circumstances would include the use of a true instrumental variable as the introduced variable (although it is uncertain whether any significant advantages would exist for the ACCE Method compared to conventional, 2-stage instrumental variable analysis). These special circumstances would also include introduced variables that are either near-instrumental variables or an additional category of variable: variables with an independent association with outcome but little correlation with other confounders. As long as the Bross equation⁸ adequately captured the effects upon confounding of increased control of this type of introduced variable upon confounding, then imprecisions in how the regression-coefficient based adjustment in Steps 3 and 4 captured the effect of correlated covariates would be relatively immaterial (since little correlation would be present). The frequency of such variables, however, is unclear. In addition, as pointed out above, it is impossible to determine whether a variable is correlated with unmeasured confounders in real-world data. However, a first step towards applying this variant would be confirming its lack of significant association with any the measured covariates available (although such a lack of correlation could not be taken as conclusive evidence of a lack of association with unmeasured covariates).

In addition, even if the ACCE Method was ultimately determined to typically provide estimates of total residual confounding that routinely have substantial imprecision, at least four beneficial applications to the method suggest themselves. The first is simply determining the direction of the remaining residual confounding in an association after efforts to control for confounding. Second, even imprecise estimates from the method may be able to provide indication of whether residual confounding appears to be a small, moderate, or large contributor to the observed treatment effect estimate. Along similar lines, associations between treatments and multiple outcomes could be able to be investigated, with the method providing information concerning which treatment-outcome associations appear to be more confounded, and which less confounded. The published data example examined in the manuscript, with its suggestion that the statin-mortality analysis is less confounded than the statin-hip fracture analysis, could be seen as an example of this application. For these reasons, this method may be a particular benefit to database surveillance research that seeks to identify promising associations for further, detailed investigation. Finally, it is conceivable that by concentrating at least part of the uncertainty concerning residual confounding to a particular focus upon the introduced variable and its potential correlates, the method may suggest, in some instances, beneficial future investigations. This may be helpful, for instance, if additional information about the introduced variable and its likely correlates is expected to be more easily gathered (e.g., through chart review) for a particular dataset than efforts to obtain information about new potential confounders.

Appendix 2.3. Other considerations, such as the impact of the form (exponential versus linear) of the treatment effect estimate

Other important considerations in the application of the ACCE Method can be envisioned. It may, for instance, be determined that the method works better for linear, rather than nonlinear, outcome models (such as either continuous outcomes, or probabilities of dichotomous outcomes rather than the outcome itself, despite the well-known shortcomings of such an outcome measure). Some other innovations in nonrandomized treatment research have been introduced as approaches applicable to linear regression models²⁰.

Perhaps one valid measure of the value of the ACCE Method, however, should not be whether it derives an estimate that is 100% accurate in theory, but rather whether it, in practice, simply helps to substantially further the goal of achieving nonrandomized study results that more closely approximate randomized trials. (To be specific, more closely approximating the findings which would be obtained if a randomized trial could be done on precisely that particular population). It is hoped that publication of this proposed method will spur research on this question as well as the other questions highlighted in these Appendices and the main manuscript.