The ACCE method: an approach for obtaining quantitative or qualitative estimates of residual confounding that includes unmeasured confounding

1.1a. A simple example based on growth of “harmful” bacteria

To start at the simplest level, one point of confusion about the method might involve the question of why an investigator would deliberately introduce additional confounding bias into their analytic models. Why amplify confounding? One extremely simple metaphor is provided by the practice of culturing blood to identify pathogens. Bacteria that are obtained from a patient blood sample are then grown for a period of time in media, thus “amplifying them”. As the bacteria grow in number they become easier to study. Thus, we amplify something we don’t want (the bacteria) to learn more about them and enable us to better eliminate them. The same is true, in a general sense, of confounding in the ACCE Method.

This metaphor can be taken further without adding too much complexity. Suppose one wanted a sense of which and/or how many bacteria were present in a sample that contained both harmful bacteria and beneficial white blood cells. By placing the sample into nutrients where only the bacteria would grow and increase in number, rather than the white blood cells, it should be possibly to get a sense of whether, and how many harmful bacteria are present. In this example, the harmful bacteria would represent the harmful confounding. We want to get a sense of how numerous these bacteria are within the sample. The white blood cells would represent the treatment effect estimate. When Model 2 is constructed, the intent is to amplify the “harmful” bias from confounding while keeping the “helpful” treatment effect estimate as constant as possible.

Finally, to go even further with this metaphor, if we knew the ultimate number of bacteria present, the typical dividing time for the bacteria growing in this set of nutrients (that is, the time it takes them on average to double their number), and we knew how long we grew the sample in the nutrients, we should be able to extrapolate backwards to the number of bacteria originally present. In this refinement, knowledge of how rapidly the bacteria divide in that particular set of nutrients would represent the knowledge of the proportional amount of confounding amplification expected to occur between Model 1 and Model 2.

1.1b. A simple metaphor based on microphones (sound amplification systems)

To perhaps allow some additional numerical “sense” of the workings of the method, consider the case of turning up the volume on a television, radio, or cell phone. If one knew that turning up the volume a certain amount doubled the volume (if, for instance, the volume settings were genuinely proportional, so that turning up the volume from “20” to “40” doubled the sound produced), and one knows the actual volume obtained after this doubling occurred, it should be both possible and simple to extrapolate back to determine what the original volume was. That is, it would not be necessary to know the original volume if you knew these other two quantities (the final volume, and the proportion by which the volume changed). In nonrandomized studies, one never knows exactly the “volume”, or amount, of total confounding, so an additional wrinkle that the ACCE Method uses is to measure the change in the overall effect estimate that occurs between two models. (Furthermore, this change is measured when circumstances have been deliberately constructed so that the models differ to the minimum extent feasible, other than the differences resulting from the amplification of confounding).

To make our example even more comparable, consider the scenario in which one knew that a particular type of microphone would double (i.e., “amplify”) the static, or white noise, in whatever sound is being broadcast. If one knew the original total, or overall, volume (on a linear scale) was “90 units”, and when we changed to that type of microphone the total volume increased to 93 units, then we would know that the static made up 3 units of the original sound (since doubling it added 3 units). This would mean that the volume devoid of any static would be 87 units. (I deliberately do not use the highly-recognized units of “decibels”, rather referring to an imaginary linear unit, since decibels use a logarithmic scale, which would make the example less easily appreciated). “Static” in this example can be seen as analogous to residual baseline confounding, and the sound volume without the static as analogous to the unconfounded treatment effect estimate.

1.2. A more complex, but more comprehensive, metaphor based on the recording of sound

The ACCE Method, as pointed out in the text, has a basic simplicity that can be represented in four steps or as few as two overarching components. As also pointed out in the text, however, these components (especially the first component) have a number of elements that still may be difficult for some readers to follow. To create a metaphor to more fully illustrate the important aspects of the proposed method, I provide a somewhat more complex example below. Hopefully, the payoff for this added complexity will be an intuitively understood, relatively nonmathematical picture of all of the important elements of the method.

Rather than basing an example on the amplification of sound, let us instead base it on the recording of sound. Assume now we are in the pre-digital recording era, when much recording of music occurs on cassette tapes. A taping companying has introduced a new tape recorder with a “noise reduction” feature. This noise reduction feature deliberately dampens down, or reduces, the white noise “hiss” that results from the operation of the internal machinery of the tape player. Prior taping equipment would introduce 11 units of this hiss into every 100 units of total sound they recorded. The new tape system only introduce 1 unit of this hiss for every 100 units of total sound recorded. The company’s engineers are very excited, and they make simple prediction. When we play an instrument at the exact same “volume” into the new equipment, we will get a recording with 90 units of sound, rather than 100 units. Not only will this lower volume be acceptable, but actually it will be highly desirable, because the recording will sound so much clearer and better. In fact, they expect people will rush to buy the new system.

Unfortunately, when the engineers test the first prototype of this new taping machine, they are thoroughly discouraged to notice that the instrument now registers as producing 105 units of sound, rather than 100 units, and the “hiss” is more audible than it had ever been with the old recording equipment. They are initially perplexed how this could happen when they were certain that the new system would reduce hiss from the working of the internal machinery by 10-fold. However, when they investigate their new system, they discover a new source of hiss from some degree of nonspecific deposition of the recording medium (e.g., iron filings) onto the tape surface due to electromagnetic fields generated during the taping process. This electromagnetic field causes some of the recording material to arrange itself incorrectly, creating this new source of “electromagnetic hiss”.

Their investigations subsequently show that somehow their new design, by seeking to minimize the noise introduced by the mechanical working of the tape player, unintentionally doubled the amount of hiss resulting from this 2^nd source of hiss. This discovery is so new that no one is even sure how much of this source of hiss is present in tapes. But the engineers are confident from their experiments that however much of this hiss exists, the new taping machine will exactly double this amount of hiss compared to the old taping system. (Let’s assume that in order to get the new machine to run the tape through especially quietly, the engineers had designed it with 4 recording heads, instead of 2, and it is known that each head will unavoidably introduce an undefined but highly consistent amount of electromagnetic hiss to the tape). The engineers realize that if they know the total volume of sound recorded by the 2 devices, they are now in a position to quantify the amount of the newly-discovered hiss originally present based solely on the information they have on hand. Here is how they would proceed.

The overall sound with the new recording system was unexpectedly amplified from 100 units to 105 units. They had expected the sound to go down to 90 units, because in their overly simple initial model they expected the only source of hiss was the 11 units of mechanical hiss introduced by the old system. Thus, they expected the original sound they were hearing was made up of 89 units of genuine instrument sound and 11 units of hiss. They had totally overlooked the second source of hiss -- because they had not known about it – and therefore had not factored it into their equations. They therefore expected the new system, with its only 1 unit of mechanical hiss, would have transmitted 89 units of genuine instrument sound and 1 unit of hiss, for a total of 90 units of sound. Armed with the knowledge that the new system, because it used 4 recording heads instead of 2, would be expected to double the 2^nd source of hiss (electromagnetic hiss), the engineers realized that the fact that the volume of the sound increased in the new system, rather than decreased, meant that quite a substantial amount of this electromagnetic hiss had to be present. A doubling of this electromagnetic hiss was more than enough to overcome the 10 unit reduction in the sound from the reduced mechanical hiss that they had expected. They determined that since they observed a 5 unit increase in volume rather than the expected 10 unit decrease, this meant that the doubling the electromagnetic hiss must have added 15 units of hiss to the sound: the 5 units increase in total sound they observed added to the 10 units of sound from mechanical hiss that their system had removed. (This equals a total of 15 units of sound). This calculation in turn would imply that there were 15 units of electromagnetic hiss in the original taping system that they had never noticed before (and certainly had never been able to quantify). For a doubling of this new source of hiss to increase the volume 15 units, it must have been originally present at an amount of 15 units.

In this example the genuine instrument sound the engineers are seeking to optimize represents the unconfounded treatment effect estimate. The mechanical hiss represents the Introduced Variable and its known effect on confounding (i.e., known “noise”). Its presence was not controlled in Recording System 1 (hence the contribution of 11 units of mechanical hiss sound), but its presence was highly controlled (and reduced) in Recording System 2 (only contributing 1 unit of sound). In a sense the engineer’s development of a noise reduction system largely removing the impact of mechanical hiss, can be seen as analogous to reducing the imbalance in the Introduced Variable by inserting it into the Model 2 regression. The ability of the engineers to predict the magnitude of decrease that occurs is analogous deriving a regression coefficient for the Introduced Variable(s) and predicting the change in the Model 1 treatment effect estimate through use of the Bross equation.

The electromagnetic hiss represents unmeasured confounders (i.e., unrecognized, unmeasured “noise”). The two systems represent Model 1 (the old system) and Model 2 (the new system). The new systems’ four recording heads relative to the two recording heads of the old system, and the predictable change in electromagnetic hiss that this change brings, represents possessing knowledge of the proportional amount of unmeasured confounding amplification expected to occur between the two models. Thus, being able to count that four heads are present instead of two is an analogue to the function of determining 1 - R² for the two exposure prediction models (or by using an “internal marker”).

What this example shows nicely is that if one wants to derive the true, genuine instrument volume of 74 units, one needs to take into account the amplification (i.e., doubling) of electromagnetic hiss in Recording System 2, but also to subtract the change in mechanical hiss from both the observed change in total sound from Recording System 1 and Recording System 2, and from the original total sound value for Recording System 1. (The genuine instrument volume would be equal to 100 units of sound minus the 11 units of mechanical hiss minus the 15 units of electromagnetic hiss detected by back-extrapolation from the increased hiss detected when this bias was doubled. That is, 100 - 11 - 15 = 74 units of sound).

This example helps illustrate key points. First, knowledge of the Introduced Variable effect (amount of mechanical hiss) is crucial, since without it, a very erroneous estimate of the quantitative effect of 2-fold amplification would result. If the reduction in mechanical hiss occurring in System 2 was somehow completely unaccounted for, then one would assume that a doubling of electromagnetic hiss that occurred led to merely a 5 unit increase in sound, and the value of the genuine instrument sound was 95 units (100 - 5) instead of 74 units. So correcting for the effect of the new, changed system (Recording System 2) on mechanical hiss is extremely important to obtaining an accurate estimate of electromagnetic hiss, and, by extension, and accurate estimate of the genuine instrument sound. Second, it is crucial to have a good estimate of the proportional amount of amplification of electromagnetic hiss. For example, if the two new heads were made of a different material than the other two, and this material had unknown effects on electromagnetic hiss, then the calculations given here could not be carried out.

There is a slight “difference in intent” between this metaphor and the actions being taken in the ACCE Method. The engineers only knew about one source of noise (the mechanical hiss), and sought to reduce this through their noise reduction system (our analogue for the Introduced Variable). Thus, they accidently, rather than intentionally, amplified the second source of noise, of which they were previously unaware. In biomedical or other intervention research, we know a second source of noise typically exists (unmeasured confounding), and we know the conditions that will amplify that second source of noise (through confounding amplification). Thus, in the ACCE Method we are using our two systems (models) to intentionally produce amplification (specifically, to the degree possible, a known amount of amplification). In biomedical or other intervention research, the fact that we are likely reducing the impact of the first source of “noise” (mechanical hiss) is almost incidental, in that we are influencing the first source of noise only as the means to achieving a predictable change in the second set of noise. The effects of our Introduced Variable(s) on the first set of noise needs consideration solely so that an adjustment can be made for its effect, so as not to misestimate the change brought about through amplification of the second source of “noise” (and also so we do not misestimate of the total original noise present in our final calculations).

Despite this metaphor’s inherent greater complexity than the bacteria and sound amplification metaphors described earlier, this sound recording metaphor has at least two notable advantages. First, it fairly completely (and hopefully intuitively) represents each of the most important steps and concepts in the full ACCE Method. Second, if desired, it can be easily expanded to illustrate some current sources of uncertainty in the method (as illustrated below).

1.3. Expanding the recording metaphor to represent uncertainties in the operation of the method

To take the metaphor further, it is possible to use it to also illustrate some of the key sources of potential uncertainty in the ACCE Method estimate mentioned in the text (and elaborated on further in the Appendices that follow). If the increase in sound is unable to be accurately recorded beyond a certain volume, this might be seen as representing nonlinearities in the 1 - R² prediction of confounding amplification. If the inaccuracy at higher values further increased the volume of sound greater than expected, this could be analogous to what is observed in the Brooks and Ohsfeldt simulations above R² = 0.56⁵. If the inaccuracy resulted in lesser increases in the volume of sound than expected, this could represent real-world “constraints” to amplification, if they exist. If the engineers were able to identify some component of the sound (say, a certain wavelength of sound) whose change could be estimated accurately even in the presence of these nonlinearities (perhaps since its overall volume, representing only a small fraction of the total sound, was low) this might be seen as analogous to the “internal marker” strategy of estimating confounding amplification.

Also, consider if the old recording system and the new recording system also had other components which could contribute “noise”. For instance, what if the new recording system used jacks (plug-in connectors) on the end of its wires that were twice as large in diameter as the jacks used in the old system, and the larger jacks also reduced noise to an unrecognized extent? This would represent a source of altered noise separate from mechanical hiss or electromagnetic hiss. This can be seen as analogous to changes between the models that are not due to either confounding amplification or the effect of the Introduced Variable(s) which lead to treatment effect estimate differences. Obviously, such a “third source” of differences in sound volume would make accurate determination of the original sound present more complex. The easiest solution would probably be simply to design the two recording systems to use the same wires and jacks, so that this source of change in the noise recorded is unchanging. This is the basis of our recommendation to attempt to minimize other differences between the models, and/or to at least check to determine the extensiveness of these differences.

1.4. Summary

The sound recording metaphor shows that there may be reasons to suspect that confounding amplification in nonrandomized studies may bear useful similarities to the amplification of bias in other systems. Most importantly, our ability to quantify the original “genuine instrument sound” through the steps taken in this metaphor supports the possibility that the same basic approach (as laid out in the ACCE Method) should enable us to obtain an estimate of a genuine treatment effect.

The degree to which nonrandomized datasets actually permit straightforward quantification of unconfounded treatment effects through a procedure very similar to what was done in this very simple metaphor is not yet clear. Can we accurately estimate the proportional amplification we generate between two propensity score models as closely as knowing that four recording heads will double noise compared to two? Can we create our two models to minimize the chance that we distort our estimate of the unconfounded treatment effect by involving other changes that affect treatment effect estimates (similar to ensuring that the jacks at the ends of the wires were not changed between the Recording Systems)? The precise extent to which these or other aims can be achieved in the ACCE Method is uncertain, but research is clearly needed given the potential of the method to improve inferences from nonrandomized studies.

2.1 Differences in the balance observed for included covariates

Although these changes may be expected to be minor, at least in some settings, they deserve thorough consideration as part of an effort to anticipate sources of potential imprecision in the estimates resulting from the ACCE Method. Some degree of change between Model 1 and Model 2 is expected in the balance of each of the propensity score covariates present in common between Model 1 and Model 2 (the “included covariates”). (NOTE: The “Introduced Variable” is also an “included” covariate in a limited sense, in that it is included in one of the two propensity scores. For clarity in terminology and because the Introduced Variable represent a genuinely special circumstance [please see Appendix 3.2b], the term “included variable” or “included covariate” is reserved for the variables included in both models. The term “Introduced Variable[s]” is used for the variable[s] added to Model 2. The term “nonincluded covariate” will refer to covariates not included in either propensity score. Nonincluded covariates may either be unmeasured covariates, which inherently cannot be included, or measured covariates not selected for inclusion into the propensity score).

These changes are produced as a byproduct of the need to include an additional variable or variables in Model 2 to generate confounding amplification. For instance, including an additional variable in the propensity score used for matching would be expected to weaken at least slightly the tightness of the match on the other covariates. In some cases, the differences in covariate balance between models could be quite minimal. However, this balance needs to be explicitly compared between Model 1 and Model 2. This comparison is important since in other cases it may prove difficult to attain a degree of balance in the included covariates that is highly equivalent between Model 1 and Model 2 if sufficient confounding amplification is to be achieved. The impact of confounding amplification is to tend to create a greater number of individuals at the extremes of the propensity score distribution who are less comparable (and thereby, less similar in balance in the covariates included in the model)⁵.

One approach worth consideration would be to examine whether it is feasible to adjust the stringency of the stratification or matching in Model 2 so that the balance in the included covariates in Model 1 and Model 2 are more equivalent. An alternative, and particularly rigorous, approach would be to use the Bross equation⁸ to attempt to estimate the change in confounding attributable to the observed changes in the balance of each included covariates (even though this may represent a large number of covariates). (Going to the effort of applying the Bross equation to each of the included propensity score covariates potentially has an additional advantage, as discussed in Appendix 3.2a, in permitting the contribution to confounding from the residual imbalances in included covariates that are present in Model 1 to be estimated and removed from the Model 1 treatment effect estimate at the final stages of the process (please see Appendix Figure 1c).

2.2. Differences in patient sample

The overall patient cohort for the study from which the samples for Models 1 and 2 are derived will obviously not change. Some degree of change, minimal or otherwise, can be anticipated to occur, however, in the samples of individuals selected from that overall cohort by each model (Model 1 and Model 2). These differences can arise from differences in the patients that fall under the “Common Support Area” of the propensity scores, and, if matching is employed, differences in the percent of patients matched. The “Common Support Area” refers to the range of propensity score values which include members of both treatment groups; it is often recommended that individuals outside the Commons Support Area be “trimmed” (i.e., removed) from the analysis¹⁸. These differences in patient sample, however, will only influence the method’s estimates to the extent that they are extensive enough to produce substantively different compositions of patients between the Models and effect modification exists (whereby the treatments studied have different effects in different patients). In addition, possible strategies exist to minimize some of these potential differences, as discussed below.

2.2a. Differences in patient sample from differences in Common Support Area/propensity score trimming. Because confounding amplification tends to make at least patients on the extremes of the propensity score distribution less comparable⁵, it might prove difficult in practice to maintain a highly similar Common Support Area between Model 1 and Model 2. Fortunately, the number and identity of individuals differing between Model 1 and Model 2 is measurable. In addition, different approaches might be compared (such as examining only the subset of patients that fall under both model’s Common Support Areas). These comparisons might establish whether the results are sensitive to small differences in the Model 1 and Model 2 patient samples arising from different Common Support Areas.

2.2b. Differences in patient sample from differences in percent matching

Regarding matching strategies, it may prove difficult for a similar proportion of matching to be preserved between the two models. (The Brooks and Ohsfeldt simulation⁵ showed that as unexplained variance of exposure decreased and amplification increased, the number of patients matched for a given caliper decreased). The alternative approach, propensity score stratification, may be determined to be the preferable choice for routine use in the ACCE Method, since by design stratification retains all individuals from the trimmed sample. (The ACCE Method emphasizes stratification and matching rather than weighting, because in the Brooks and Ohsfeldt simulation⁵ propensity score weighting produced confounding amplification that was less predictable, at least by R²).

As mentioned at the start of Section 2.2, these differences would only be expected to have relevance to the degree that effect modification existed (i.e., to the degree that the differences in patient sample would be expected to result in a change in the genuine treatment effect estimate between the two models). Because of the apparent dependence on effect modification for differences in Common Support Area or percent matching to impact the method’s performance, a separate approach to take to this issue would be to test for whether effect modification is detectable related to measured covariates. (However, this would obviously not address possible effect modification related to unmeasured covariates, although a consistent lack of observed effect modification might suggest a lower likelihood of effect modification than could be surmised before these observations). Clearly additional research (especially simulation research) is needed to assess the sensitivity of the method’s performance with and without the presence of treatment effect modification.

2.3. Differences in the specifics of the intervention received

While the general nature of the interventions received by the two treatment groups remains identical between Model 1 and Model 2, specific aspects of the intervention received can vary between the treatment groups in ways that are not immediately obvious. It is important to consider these possible differences because they may be another contributor to differences in the treatment effect estimates between Model 1 and Model 2 unrelated to confounding amplification.

2.3a. Differences in dose

Unless the intervention is a single, one-time-only dosed treatment, such as a vaccine, either dose or other “quasi-dose” aspects of how the intervention is administered may vary at least slightly between the individuals receiving the intervention in Model 1 and those receiving the intervention in Model 2. Even for nonmedication-based interventions, such as a psychotherapy or educational intervention, the timing of visits or the number of visits may vary slightly among the individuals included in the intervention arm in Model 1 versus Model 2. Therefore, when implementing the ACCE Method, it is important to examine whether the overall mean dosage, number, or timing of treatments is similar between Model 1 and Model 2, and potentially within strata for stratified analyses. If sufficient sample size exists in particularly large patient samples, an additional approach might be to restrict the analysis to patients only receiving one particular dosage of the treatment.

2.3b. Differences in discontinuation rates

Treatment effect estimates are likely to be sensitive to discontinuation rates, whether an intent-to-treat or an as-initially-treated analysis (i.e., with follow-up censored upon termination or alteration of the initial treatment) is conducted. Because of this, investigators should examine the rates of discontinuation observed in Model 1 and Model 2 to determine their similarity. Ideally, a determination that the reasons for discontinuation within the patient sample for Model 1 compared to Model 2 were also similar would be the ideal, but such information is often not available¹⁹.

In many cases the difference in discontinuation rates for each treatment between the two models may be quite small, and the practical impact of this difference unclear. Differences in discontinuation rates appear to be at least slightly more significant in this approach, however, than in a propensity score or regression analysis involving a single model, if effect modification is present. In this case, differences in discontinuation rates between the two models could produce some degree of difference in the underlying treatment effect estimate between the models, even if no selection directly relevant to outcome was occurring.

Addressing specific differences (e.g., dose) in the intervention received by patients between the models, if they exist, may be at least slightly more challenging than minimizing differences involving confounding control from included covariates. One modest strategy might involve simply evaluating the differences in intervention specifics when different strategies are explored to minimize differences in Common Support Area or in the control of confounding from included covariates. Then the strategy that also minimizes differences in the intervention could be examined as the main analysis or as a sensitivity analysis.

As indicated in the main manuscript, this manuscript generally does not consider confounding arising after treatment initiation from differences in patient characteristics of patients who remain receiving in the two treatment groups. However, three points should be made. First, the “unconfounded treatment effect estimate” that is intended to be provided by the ACCE Method refers to an estimate unconfounded from baseline differences, but not necessarily completely unconfounded (i.e., it could still be confounded by differences arising after treatment initiation). Second, confounding after treatment initiation can exist, but if it is similar between the two Models, then it need not pose a barrier to obtaining a treatment effect estimate largely unconfounded from baseline factors. While the method is particularly dependent on the difference between Model 1 and Model 2 treatment effect estimates being attributable as much as possible to confounding amplification, in this case, confounding post-initiation would not be expected to be a major source of differences in the treatment effect estimates between the two models. In contrast, if substantially different amounts of confounding post-initiation exist in Model 1 and Model 2, this circumstance both produces a source of confounding not addressed by the ACCE Method and interferes with the ACCE Method’s ability to generate an accurate estimate of baseline confounding. At a minimum, rates of discontinuation should be checked to ensure they are similar between Model 1 and Model 2. Others have made the point, however, that similar discontinuation rates between treatment groups that are being compared can still conceal confounding after treatment initiation if the reasons for discontinuation differ¹⁹. It remains to be seen, however, how often similar discontinuation rates between Model 1 and Model 2, which are likely to share a considerable number of individuals in common, in fact conceals differing reasons for discontinuation between the two models. (That is, the concerns raised in general about the possibility of different reasons for discontinuation existing despite similar discontinuation rates between treatment groups may not be as pertinent to the specific circumstance here of similar discontinuation rates between Model 1 and Model 2, since these models may share a substantial number of individuals in common). As mentioned above, the presence of similar discontinuation rates should be expressly confirmed. Further research is clearly needed.

Third, it is conceivable that the same approach used here – deliberately introducing confounding amplification to estimate the original confounding present – could be used, at least in theory, to also sequentially estimate residual confounding attributable to differential discontinuation during treatment. This is likely to be a substantially more difficult and complex endeavor than the use of the ACCE Method to estimate residual baseline confounding. For instance, the most commonly used approach for addressing measured confounding from differential discontinuation, generating a “pseudopopulation” by weighting, has been shown in simulation to produce confounding amplification that is considerably less predictable than matching or stratification⁵. Re-matching patients after initiation could be considered, but the results would be expected to become applicable to only a smaller and smaller subset of patients. One relatively simple approach might be to conduct rematching at only a single additional time point: study completion. One challenge to using the ACCE Method to estimate confounding after treatment initiation may be that presumably the Introduced Variable(s) used to amplify confounding after treatment initiation will need to explain differential exposure to continued treatment (i.e., factors that lead to the discontinuation of one treatment but not the other). It may prove difficult to find variables that explain a substantial amount of continued exposure to one treatment but not the other (e.g., specific adverse events particular to one treatment might only affect a small percentage of individuals discontinuing treatment). This problem may be more easily addressable, however, if a set of variables can be introduced simultaneously (Appendix 6).

Thus, while conceivably it could be ultimately determined that an assumption may need to be made for this method of “similar reasons for discontinuation between models”, it is too premature to make that conclusion. The possibility at least exists that this method, designed initially to address baseline confounding, may also allow residual/unmeasured confounding post-initiation to be addressed in those instances in which a suitable Introduced Variable or Variables for amplifying confounding post-initiation can be identified. Clearly, much additional work is needed in order to assess the feasibility of applying this approach to address confounding occurring after treatment initiation.

2.4. Summary

The need to evaluate, and potentially address, the diverse factors that may contribute to a change in treatment effect estimates between Model 1 to Model 2 initially may seem daunting. However, it may be ultimately determined that in practice little difference between Models in these aspects is typically observed. In theory, there may even be circumstances in which sizeable differences in some of these aspects do not prevent the method from providing accurate estimates (e.g., a difference in timing of an intervention whose effects have been shown to not be very sensitive to the timing of its administration). In most cases, however, substantive differences of the types described between the models are a concern. If these differences cannot be minimized by the strategies suggested, or approaches to quantifying the likely effects of these differences cannot be identified, then caution in interpretation is clearly warranted. Validation studies using simulated or real-world datasets would provide useful information concerning both the frequency with which these differences occur and their impact on the ACCE Method’s estimates.

Appendix 3.1. Approaches to estimating proportional confounding amplification

In the lower ranges of exposure prediction (at least as measured by explained variance in terms of R²), a simulation has shown that a predictable relationship exists between amount of remaining unexplained variance in the prediction of exposure and confounding amplification⁵. Differences in prevalence between treatment arms in covariates that are not included in the propensity score increase linearly with increases in R². This increase in the imbalance of uncontrolled (i.e., nonincluded) factors is the phenomenon that underlies the amplification of residual confounding. However, in the upper portion of the range of R² the relationship becomes increasingly nonlinear, with changes in R² underestimating the increased imbalance in nonincluded covariates⁵. If this nonlinearity in the upper range of R² is replicated, but is reduced or not apparent for other metrics of prediction of exposure, then these metrics should be preferred. If this nonlinearity in the upper ranges of exposure prediction continues to hold for other metrics, then three strategies suggest themselves. The first approach, the “low amplification strategy”, would be to deliberately limit Models 1 and 2 so that the prediction of exposure these models achieve are in the lower end of the possible range, where the relationship is most linear. In some cases, propensity score models may already fall into this range. In other cases, this approach may involve reducing the variables included in the propensity scores. Such reduction might entail including only variables with a significant a priori expectation, based on evidence, of being confounders²⁰. An additionally restrictive strategy would including those variables estimated (by using the Bross equation⁸) to be the most substantial confounders, or suspected a priori of being particularly certain or strong confounders (e.g., age, Charlson Comorbidity index, etc.). As discussed in Appendix 4, a particular high priority likely needs to be given to including variables for confounders that are correlated with the Introduced Variable(s), especially if they are strong confounders. Reductions in the number of included covariates could increase residual confounding relative to some other models that could be constructed, however, thus increasing reliance on the accuracy of the ACCE Method to address that increased confounding.

However, at least two other strategies suggest themselves that may prove feasible. One alternative would be to develop a formula that captures any nonlinearity in the chosen metric of exposure prediction. This could permit the amount of expected amplification to be relatively accurately predicted over larger portions of the range. The second alternative strategy would be to develop an “internal marker” covariate that would reflect how much increased imbalance in the nonincluded confounders is occurring. The internal marker would be a measured covariate deliberately left out of the propensity scores. The increase in its imbalance in Model 2 could be measured and serve as an indicator of confounding amplification.

Intuitively, an internal marker strategy has some attractive qualities, since it might sidestep any uncertainties about the relationship between confounding amplification and metrics of prediction of exposure. Furthermore, use of internal markers might prove the easiest way currently to apply this approach when logistic models are used to estimate exposure (as is commonly done for propensity scores).

There is already some evidence to support the “internal marker” approach. In the Brooks and Ohsfeldt simulation study⁵ it was shown that covariates not included in the propensity score (and that are uncorrelated with the included covariates) all amplify to a remarkably similar extent, at least in that simulation. Thus, in principle, it appears feasible to use an “internal marker or markers” (produced by withholding measured covariates from the Model 1 and Model 2 propensity scores) to track and estimate the general amount of confounding amplification. A key practical consideration in real-world datasets, however, is the need for these internal markers to have a minimal correlation with any of the covariates included in the propensity scores. Any such correlations might “constrain” the ability of the internal marker to reflect the degree of confounding amplification that is influencing the nonincluded confounders. (These correlations, at least if positive correlations, would largely not be expected to interfere with confounding amplification for nonincluded covariates that are not being used as internal markers, for reasons discussed in Appendix 3.2). Since, in the strictest sense, some degree of correlation is virtually unavoidable, then the internal marker strategy may intrinsically underestimate true confounding amplification, although in some cases only minimally.

Fortunately, this correlation is readily measurable. Therefore, it should be possible to deliberately select the nonincluded measured covariate with the least correlation with both the included covariates and the Introduced Variable(s) to serve as an internal marker. Alternatively, simulation research may suggest quantitative approaches to correct for this correlation. If the internal marker strategy is used, this may constitute another practical reason to limit the covariates included in Model 1. In addition, should all candidate internal markers have some significant degree of correlation with covariates included in the propensity score, then any internal marker would be expected to underestimate the proportional increase in confounding amplification. Given the mathematics of the method, dividing a particular change in treatment effect estimates by an underestimate of proportional confounding amplification will overestimate confounding and lead to a conservative estimate of treatment effect.

In summary, multiple aspects of confounding amplification estimation are worthy of investigation. These include the presence and predictability of nonlinear relationships between the prediction of exposure metric and confounding amplification, and the potential strategies to address these nonlinearities.

Appendix 3.2. An initial exploration of the impact of correlation on confounding amplification

Once the proportional confounding amplification has been estimated in general (through R² or other prediction of exposure metrics, or an internal marker), additional aspects of confounding amplification deserve consideration. These aspects center on the question of whether the predicted confounding amplification will in fact occur for all confounders.

The data that exists to date from the Brooks and Ohsfeldt simulation⁵ indicates that confounding amplification is uniform between simulated covariates that were not included in the propensity score. However, this similarity may be a byproduct of their simulation. In theory, aspects of real-world data might create heterogeneities in amplification between covariates. The effect upon confounding amplification of correlations between covariates, expected to be a common feature of real-world data, is considered below. This conceptual exploration of the impact of correlations, which also draws upon the Brooks and Ohsfeldt simulation⁵, tentatively concludes that many correlations do not appear to substantially interfere with the ACCE Method. Correlations can be categorized into five types, based on whether the correlated covariates are included or not included in the propensity score model: correlations between two nonincluded covariates, between a nonincluded and an included covariate, between nonincluded covariates and the Introduced Variable(s), between included covariates and the Introduced Variable(s), and between two included covariates. For the latter two categories, substantial amplification involving either of the correlated variables is not expected, at least in the same sense as the term applies to nonincluded covariates. (However, as pointed out in Appendix 2.1, the balance in included covariates can change at least somewhat between the two models and needs to be evaluated). Positive correlations between two nonincluded covariates also appear to often be nonproblematic. Both correlated variables would constitute part of the residual confounding being amplified, and thus be expected to be amplified to a similar extent, based on the Brooks and Ohsfeldt simulation⁵.

A problem may exist, however, if some of the nonincluded variables are negatively correlated with some of the other nonincluded variables, since this might pose “constraints” to each amplifying to the extent predicted (Appendix 3.3). Also potentially problematic, are correlations between nonincluded covariates and included covariates, and between nonincluded covariates and the Introduced Variable, although in the case of the former the impact is typically expected to be minor, and in the case of the latter, potential remedies may exist although these need to be evaluated. To explore the distinct issues raised by each of these classes of methods, we deal with each of these types of correlations as “special cases” below so as to explore the distinct issues raised by each.

Appendix 3.2a. The special case of correlations between included and nonincluded covariates

Correlations between included covariates and nonincluded covariates, however, could initially seem to pose the possibility of creating constraints to amplification for certain covariates. The measured covariates are included in the propensity score, cannot amplify substantially, and thereby might seem to constrain amplification, to a degree, of the correlated nonincluded variable. In fact, it is true that in this case the correlated nonincluded variable would be expected to have an overall change in imbalance in Model 2 that is less than the estimated confounding amplification. However, the implications of the Brooks and Ohsfeldt simulation⁵ suggests that the change in a correlated variable can be alternatively modeled as a fraction that is largely unchanging (to the degree that it is correlated with included covariates that do not appreciably change), and a fraction (alternatively, a “residual”) that amplifies as much as any uncorrelated nonincluded covariate (Reference 5, Appendices). (These elements will be termed the “included fraction” and “nonincluded fraction” of the nonincluded covariate, respectively).

It is important to recognize that part of the goal for the nested models in the ACCE Method is to minimize change in the balance of included covariates, to the extent feasible (Appendix 2.1). If the nested models do successfully exhibit little change in the balance of included covariates, then the Brooks and Ohsfeldt simulation⁵ would suggest that the amount of imbalance observed in the “included fraction” of correlated unmeasured or nonincluded variables also would not change substantively between models. (The term “unmeasured or nonincluded” confounders or covariates will refer throughout these appendices to either unmeasured factors, which inherently cannot be included, or measured covariates that could potentially be included but are not included in a particular propensity score model). This lack of change in the included fraction would leave the amplification of the nonincluded fraction (which Brooks and Ohsfeldt have observed amplifies as completely as for the uncorrelated, nonincluded covariates)⁵ as the only contribution to the change in treatment effect estimate attributable to this covariate. Thus, in principle, the method would still provide accurate final effect estimates of the confounding attributable to the amplifiable fraction of Model 1 confounding. However, a separate issue is raised by these included-nonincluded covariate correlations. A problematic “reservoir” of residual confounding would be built up that is neither part of the amplifiable fraction nor part of contribution to Model 1 confounding from the Introduced Variable(s) and its correlates. This “reservoir” is problematic, in theory, because even if an additional step to the method involving applying the Bross equation to each included covariate is added on, as described in the next paragraph, the bias from these correlates might prove to be only partially addressable.

As mentioned immediately above, consideration of the impact of included-nonincluded covariate correlations brings to the fore the fact that, even in the absence of any included covariate nonincluded covariate correlations, a small fraction of confounding would typically be expected to exist after application of the method attributable to the residual imbalance of the included covariates (since perfect balance in all included covariates is implausible). This residual confounding could be addressed in at least 3 ways. First, such confounding may be able to be minimized by achieving as close a balance as feasible between treatment groups for any variable included in the propensity score models. (This is yet another reason, in addition to those discussed in Appendices 3.1, to possibly favor smaller propensity score models, although again, this “adds pressure” on the method to deliver an accurate estimate of the now-larger residual confounding). Second, regression coefficients for each of the covariates in the model could be derived from multivariate models (including all of the other covariates) for each of the covariates in the model. The Bross equation could then be used to estimate the contribution of the remaining imbalance in each of these variables to residual confounding in Model 1 (in addition to accounting for any small change in balance in these covariates observed in Model 2 compared to Model 1, as discussed in Appendix 2.1). For investigators particularly interested in obtaining the most rigorous unconfounded treatment effect estimate available through this method, such an approach may be preferable to performing no adjustment, even if these efforts change the unconfounded treatment effect estimate only modestly or minimally (Please see Appendix Figure 1c). Third, all of the included propensity score covariates could also be inserted into the Step 1 treatment-outcome regression equation estimating the Model 1 and Model 2 treatment effects (i.e., in addition to including these covariates in the propensity score). This should largely address residual confounding arising from the remaining imbalance in these included covariates (again, this might pose some upper limit on the number of covariates that could be included in propensity score). However, it should be noted that the degree to which either the Bross equation approach or the insertion of the correlated included covariate into the treatment-outcome regression fully corrects for the impact of residual confounding from correlated nonincluded confounders is uncertain. Confounding amplification relating to the estimation of regression coefficients for the included covariates could also conceivably be a problem. Therefore, it is uncertain whether attempting to minimize covariate imbalances between the treatment groups or address those differences through one of the two regression-based approaches should be clearly preferred. On the other hand, it must be pointed out that we are considering the impact of a fraction of what already is a fraction of the confounding from the nonincluded covariate. That is, we are considering the residual “reservoir” of confounding from the included fraction of the nonincluded covariate that is not addressed through the Bross equation or insertion in the main regression. This residual is itself just a fraction of the “included fraction”, which already is a fraction of the overall confounding from the nonincluded covariate. It therefore may prove that, in a practical sense, this concern is typically not a major problem. It could be argued, however, that the sum of a larger number of these minor residuals might have some noticeable impact on the treatment effect estimate. And that the uncertainty around the impact of this element of confounding is concerning because, strictly speaking, its size cannot be easily estimated. This may constitute one aspect by which the method may provide an estimate of residual confounding that is at least slightly incomplete; that is, this method may underestimate residual total confounding to some degree in typical practice. Further research regarding this issue would be clearly beneficial.

Appendix 3.2b. The special case of correlations between nonincluded or unmeasured covariates and the Introduced Variable(s)

Correlations between nonincluded covariates and the Introduced Variable are a particularly distinct circumstance. In this case, the Introduced Variable is an included variable in only one Model (Model 2). As a consequence, its balance is being deliberately changed from Model 1 to Model 2 to produce the needed confounding amplification. The “included fraction” of any nonincluded covariate that relates to its correlation with the Introduced Variable(s) is therefore being more closely balanced in Model 2 than in Model 1. This is because in Model 1 the nonincluded covariate’s correlate, the yet-to-be-inserted Introduced Variable, is not controlled at all, except for any control related to correlations between the Introduced Variable and included covariates, which stay largely constant between Model 1 and Model 2. (To clarify, the set of variables in both models that are the included covariates stay constant, but the exact balance of these variables may vary slightly or somewhat between the models). However, it is partly for this circumstance that the Step 3 and 4 procedure was designed involving deriving regression coefficients and applying the Bross equation⁸. The intent of Steps 3 and 4 is that the change in confounding of the Introduced Variable is estimated, along with, to some degree, the change in confounding resulting from the change in imbalance in the fraction of the correlated nonincluded variable(s). Whether the effect of correlation upon regression coefficients, however, is sufficiently similar to the effects of correlation on the balancing of covariates using a propensity score to permit a generally effective adjustment is uncertain.

This is an area of the ACCE Method in which further research would be particularly beneficial. The comparability of the quantitative effects of correlation on regression coefficients versus on covariate balance in propensity score analyses could be examined further through simulation, and perhaps theoretically through frameworks based on the general location model²¹ or other methods. Such simulations would helpfully allow the strength of the correlation between correlated variables and the amount of confounding amplification existing between the two models to be varied. Real-world studies could contribute by investigating how frequently the ACCE Method provides what appears to be improved treatment effect estimates (i.e., closer to the result that is expected based on randomized trials)^16,17 than typical propensity score or regression methods.

Finally, any imprecision in the ability of the adjustment in Steps 3 and 4 to adequately reflect the change between the models in confounding attributable to the nonincluded covariates correlated with the Introduced Variable would depend on the number and strength of such correlations. While this is impossible to quantify the correlations present for truly unmeasured covariates in real world data, it may turn out, based on the comparisons to randomized data discussed above, that in practice these correlations often do not appear to be numerous or strong enough to substantially affect the method’s estimates. Even if the regression coefficient/Bross equation-based adjustment was ultimately shown to only poorly capture the effects of correlation, it is possible this may not interfere markedly with the overall accuracy of the method if most of the residual confounding is not correlated with the Introduced Variable. As discussed in Appendix 4.1, it may be possible to test for this correlation, even though the correlation involves unmeasured confounders, by a particular re-application of the ACCE Method.

Even in the “worst case” scenario in which the adjustments in Step 3 and 4 of the method do not perform well and comparisons with randomized data suggested that this limitation typically impairs the method’s estimates substantially, three special circumstances exist in which the ACCE Method’s performance would not be generally expected to be adversely impacted by this limitation. These special circumstances would include using as Introduced Variables either 1) true instrumental variables (although it is uncertain whether any significant advantages would exist for the ACCE Method compared to conventional, 2-stage instrumental variable analysis); 2) near-instrumental variables, or 3) a related, but less restrictive, category of variable: variables with an independent association with outcome but little correlation with other confounders. As long as the Bross equation⁸ adequately captured the effects upon confounding of increased control of this outcome-associated but uncorrelated-with-other-confounders type of Introduced Variable upon confounding, then imprecisions in how the regression-coefficient based adjustment in Steps 3 and 4 captured the effect of correlated covariates would be relatively immaterial (since little correlation would be present). The frequency of such variables, however, is unclear. In addition, as pointed out above, it is impossible to determine conclusively whether a variable is correlated with unmeasured confounders in real-world data. As we describe in Appendix 4.1, however, one approach may exist to attempt to test for a lack of correlation between the Introduced Variable and unmeasured or nonincluded confounders of the treatment effect estimate. Another, likely less powerful, approach towards applying this variant would be confirming its lack of significant association with any the measured covariates available (although a lack of correlation with measured covariates certainly could not be taken as conclusive evidence of a lack of correlation with unmeasured covariates).

In addition, even if the ACCE Method was ultimately determined to typically provide only substantially imprecise estimates of total residual confounding, several beneficial applications suggest themselves. The first is simply determining the direction of the remaining residual confounding in an association after efforts to control for confounding, which sometimes can differ from the direction of initial confounding. Second, even imprecise estimates from the method may be able to provide indication of whether residual confounding appears to be a small, moderate, or large contributor to the observed treatment effect estimate. Along similar lines, associations between treatments and multiple outcomes could be able to be investigated, with the method providing useful information concerning which associations between a particular treatment and a variety of outcomes appear to be the least confounded, even if a precise estimate of this confounding cannot be obtained. For these reasons (as pointed out in the text) this method may be a particular benefit to database surveillance research that seeks to identify promising associations for further detailed investigation. Lastly, by at least partially concentrating uncertainty concerning residual confounding to a particular focus upon the Introduced Variable(s) and its potential correlates, the method may permit, in some instances, a beneficial focusing of future investigations. This may be helpful, for instance, if additional information about the Introduced Variable’s anticipated correlates can be easily gathered (e.g., through chart review), thus moving this correlate, to some degree, from “unmeasured” to “measured.”

While many possibilities can be anticipated theoretically, a clearer answer concerning the significance of issues around potential correlations involving unmeasured or nonincluded covariates will likely await both simulation studies and empirical testing of the method on real-world data in which the presumed genuine treatment effect is known. For example, a fraction of the 234 “negative controls”¹⁷ identified by The Observational Medical Outcomes Partnership for which there is particular confidence about the lack of association might provide a very useful “substrate” to test the method’s ability to remove residual unmeasured confounding and provide unconfounded treatment effect estimates.

Appendix 3.3. The possibility of additional “constraints” upon confounding amplification

Another important issue to consider is the possibility that, in real-world datasets, inherent limits or “constraints” may exist to how much a covariate can conceivably amplify regardless of changes in exposure prediction. Two such possible constraints have been already mentioned in Appendix 3.2: the “constraints” possibly imposed by a negative correlation between two nonincluded confounders, and by nonincluded covariate-included covariate correlations.

Initially, the possibility of a different set of constraints might also seem plausible, based on limits to how much an imbalance in a particular covariate could inherently change. Such a concern might seem plausible, for instance, if the sample matched on the propensity score already included almost all individuals possessing this characteristic. However, further consideration suggests that such seeming “practical constraints” may not interfere greatly with the method in practice. While such a constraint presumably would limit how much the prevalence of that specific covariate in a treatment group could increase, it would not limit how much the prevalence of that covariate could decrease in the opposite treatment group. Since the imbalance between treatment groups relates to the relative difference in prevalence between the groups, such an imbalance could occur as easily through a loss of representation of the covariate in one group as from a gain in representation in the other. One obvious, although potentially modest worry, however, would be whether such a decrease in the prevalence of a covariate would come at the cost of creating a substantial difference in the Common Support Area or percent matching between the propensity score models. (If so, this would represent one of the potential differences in the patient sample between models discussed in Appendix 2.2).

Among the most substantial concerns regarding possible “constraints” is the issue that was noted in passing in Appendix 3.2: negative correlations between two nonincluded or unmeasured covariates. Covariates are not free to change in isolation. For instance, let us assume that, for a medication study, Model 2 confounding amplification is supposed to double the imbalance in two unmeasured confounders: recent adverse effects from surgery and health care access. Assume as well that those individuals at greatest risk for experiencing adverse events from surgery in the study period are, on average, also those individuals with superior health care access. Now assume that in the treatment group there is a greater prevalence of adverse events from surgery but, for unclear reasons, a lower degree of health care access than in the comparison group. For confounding to double for both covariates, a greater difference in the number of individuals with adverse events from surgery needs to be present (with more of these individuals in the treatment group), along with a greater difference in the number of individuals with health care access (with fewer of these individuals in the treatment group). Thus, it appears difficult, solely through adjusting the prevalence of individuals with adverse events and superior health care access to achieve a worsening of both of these imbalances. Conceivably, this means that the difference between the treatment group in the specific fraction of individuals with just adverse events, or just superior health care access, might be particularly amplified, but this remains purely speculation until simulation can address this issue.

It is also difficult to predict the routine plausibility of this scenario (in which many of the individuals in a treatment group who exhibit one confounding characteristic also possess a second confounding characteristic, yet a negative correlation between these variables exists overall). Once again, since the degree of imbalance in the truly “unmeasured” confounders can clearly be specified only in simulation (although mock “unmeasured confounders” can be created in real-world studies by deliberately not including some measured covariates), the first step to assessing the implications of these scenarios would almost certainly be through simulation research.

In the meantime, I have sought to capture these potential uncertainties in the manuscript through the use of the general term “the predictability of confounding amplification”. Uncertainties about the predictability of confounding amplification could fall into at least four general categories. One potential category would be if the change in prediction of exposure (or change in an internal marker) failed to correlate well with the overall confounding amplification observed (i.e., imprecision in the estimation of proportional confounding amplification). A second potential category would be if different unmeasured confounders simply inherently amplify to different extents for a given change in the prediction of exposure (i.e., there is an inherently wide variability in how much individual confounders amplify, even if they completely lack correlations). The third potential category relates to the possibility that unmeasured confounders may amplify essentially as predicted until a certain point of exposure prediction when “constraints” appear to manifest themselves and progressively impede amplification. The final category would relate to the impact of correlations on the method's estimates.

It should be pointed out that if real-world “constraints” to amplification exists, they would be generally expected to produce opposite effects with increasing prediction of exposure to what was observed in the Brooks and Ohsfeldt simulation⁵. That is, real-world constraints in the amount that a confounder could amplify would become progressively evident at higher ranges of exposure prediction, when confounding amplification is expected to be greater. As a result, presumably result in less change in the Model 2 estimate would be observed than predicted by a 1 - R² amplification of confounding, rather than the greater than expected change observed by Brooks and Ohsfeldt above an R² of 0.56⁵. As an obvious corollary, if constraints to amplification exist then their effects are likely to be less evident at lower levels of amplification, for which less change in the balance of unmeasured covariates is expected in Appendix 3.1 and Appendix 3.2. Thus, this consideration may have similar implications to others previously mentioned in support of working in the lower ranges of exposure prediction. Nevertheless, the adoption of such a radical strategy would need to be supported by considerable research supporting this method’s quantitative accuracy, especially since it may prove more difficult to the lower range of exposure estimates to accurately discern differences between treatment estimates.

Appendix 3.4. Other considerations, such as the impact of the form (exponential versus linear) of the treatment effect estimate

Other important considerations in the application of the ACCE Method can be envisioned. To provide one example, since the method involves the straightforward subtraction of one treatment effect estimate from another, it is possible that the noncollapsibility of odds ratios and hazard ratios will be problematic. That is, noncollapsibility may provide another source of difference between the Model 1 and Model 2 treatment effect estimates (i.e., a difference not related to confounding amplification). (As mentioned previously, a key requirement of this method is that the difference between the Model 1 and Model 2 treatment effect estimates represent, to the extent feasible, only the effects of amplified confounding). Therefore, it may be determined that this method works better for linear outcome models (such as those involving either continuous outcomes, or probabilities of the outcome rather than the presence or absence of the outcome itself), or log-linear (e.g., log-binomial, Poisson) outcome models (which provide risk ratios) than logistic outcome models. This manuscript already focuses upon using linear, rather than logistic, propensity score models to allow for estimation of the change in the prediction of exposure. This choice was made simply because the linear model is the model for which the work on the quantitative relationship between changes in prediction of exposure and confounding amplification has been the most developed⁵. Obviously, there are well-known shortcomings of using probabilities for dichotomous healthcare endpoints such as treatment, mortality, or disease onset. Nevertheless, some other innovations in nonrandomized treatment research have been introduced in the form of their application to linear regression models²². The best modelling approaches to pursue for both treatment exposure and outcome to optimize the performance of the ACCE Method are not yet clear.

Appendix 4.1. Advantages and disadvantages of amplification in the Introduced Variable-outcome regression coefficient

One potential challenge to the capability of the ACCE Method to provide quantitative estimates of unmeasured confounding involves the degree to which confounding amplification also biases the Introduced Variable-outcome regression coefficient. Theoretical work has raised the possibility that confounding amplification may result whenever a variable is conditioned upon⁴ (although, from a practical perspective, the problem may end up typically being most pronounced for propensity scores, given the large number of covariates often able to be included in these models). If true, this means that confounding amplification might also occur with the Introduced Variable-outcome regression, affecting the Introduced Variable-outcome regression coefficient. Such confounding amplification, to the extent it exists, would be a concern for two reasons: 1) this Introduced Variable-outcome regression coefficient is used twice within the method; and 2) since the bias relates to unmeasured factors, it is difficult to estimate its extent (however, please see discussion below of two closely-related strategies that may address this concern).

Amplification of confounding in the Introduced Variable’s regression coefficient would bias estimates of the Introduced Variable’s true impact on two terms from the ACCE Method Summary Equation provided in the manuscript, Appendix 5, and the Appendix Table. Bias would be present in the terms Conf_{IntV∆(M2-M1)} (the contribution of the Introduced Variable(s) to the change in the treatment effect estimates) and Conf_IntVM1 (the contribution of the Introduced Variable(s) to the original, Model 1 confounding). (NOTE: Although it appears feasible to insert more than one Introduced Variable simultaneously, as discussed in Appendix 6, and thus more than one Introduced Variable-outcome relationship will exist, I will use only the singular here to improve readability. When multiple Introduced Variables are inserted, the approaches discussed here would need to be considered for each Introduced Variable).

However, several important points regarding this possible bias to the Introduced Variable-outcome regression equation are worth considering. First, there are reasons to suspect that confounding amplification of the Introduced Variable-outcome association usually will not be as extensive as that for the treatment effect estimate, both due to lessened prediction of exposure and a lesser amount of confounding initially to be amplified. The overall focus of the ACCE Method, like other methods, is to address potential confounding related to the treatment, rather than confounding related to the Introduced Variable. Therefore, it is not clear that inclusion of the same covariates in a regression examining the Introduced Variable’s association with outcome will lead to the same proportional confounding amplification. This is especially probable if an Introduced Variable(s) is available to be chosen that is relatively uncorrelated with the measured covariates being included in the model. Furthermore, the degree of predicted amplification (e.g., whether “small” or “large”) should be readily detectable by measuring the R² for the prediction of the “Introduced Variable” exposure in the within treatment group regression equations.

In addition, the Introduced Variable may not have as many correlations with unmeasured potential confounders of its relationship with outcome as the treatment itself does, thus leading to less confounding to be amplified. For instance, consider a cancer treatment in which three major unmeasured or incompletely measured factors influence treatment choice: 1) presence of metastases; 2) specific primary tumor symptoms reported; and 3) degree of access to health care. An Introduced Variable(s) might be associated with the presence of metastases but may not be as associated with specific symptoms of the primary tumor (e.g., bleeding, swelling, etc.), or degree of access to health care. Thus, while unmeasured confounders relating to each of these areas would be expected to be correlated with (and thus confound) the treatment-outcome relationship, only unmeasured confounders related to the first area would be expected to substantially confound the Introduced Variable-outcome relationship. Therefore, in some cases, the Introduced Variable-outcome relationship may be less confounded than the treatment-outcome relationship, and thus less quantitatively sensitive to the same degree of proportional confounding amplification (i.e., show less change in estimate even at the same particular R² value). (In actuality, as discussed above, the R² value for the Introduced Variable-outcome model is also likely to be less than that of the treatment-outcome model). Thus, both the proportional amount of confounding amplification (1 / (1 - R²)) and the underlying amount of unmeasured confounding is likely to be less in the Introduced Variable-Outcome relationship, especially in circumstances in which an Introduced Variable can be chosen that is relatively uncorrelated with measured, included covariates.

Importantly, there also are two approaches that suggest themselves for addressing uncertainty about what degree of Introduced Variable-outcome confounding amplification that may exist. The first approach hinges on the possibility that it may prove feasible to iteratively reduce the magnitude of this bias by also estimating quantitatively confounding amplification in the Introduced Variable(s)-outcome regression through essentially serially repeating the steps of the ACCE Method. This would involve determining the Introduced Variable-outcome relationship from a regression comparing two groups, one with the Introduced Variable, and one lacking the Introduced Variable, with likelihood of being in one of the two groups predicted by a propensity score involving all of the Model 1 propensity score covariates and treatment. (In essence, exposure to, or possession of, the Introduced Variable would be taking the place of exposure to treatment in the main analysis). Then a single or set of variables predictive of Introduced Variable exposure could be inserted for generating confounding amplification in this “2^nd order” application of the method. The same steps that were applied to produce an unconfounded treatment effect estimate could then be applied to determine an unconfounded Introduced Variable-outcome coefficient.

This would obviously not fully resolve the uncertainty, since confounding amplification of this “second order” Introduced Variable(s) might still contribute some uncertainty to the estimate of the unconfounded Introduced Variable(s)-outcome relationship, which would in turn result in some lesser amount of uncertainty in the unconfounded Treatment Effect Estimate. This approach, however, would serve to transfer much of the concern about confounding amplification in the regression coefficient onto yet another variable or variables (those “second order” Introduced Variables) further removed from the estimate of greatest interest (the unconfounded treatment effect estimate). Presumably, in some cases this process could be repeated once again if desired, if a variable or set of variables sufficiently associated with exposure to the second-order Introduced Variables existed.

Finally, while the potential for bias in the Introduced Variable(s)-outcome relationship is unfortunate, there is one sense in which this potential for bias may actually prove to be an advantage. The only way for the Introduced Variable(s)-outcome regression coefficient to suffer confounding amplification is for nonincluded factors (typically, unmeasured factors) to exist that are correlated both with the Introduced Variable and outcome. Without a correlation between the Introduced Variable(s) and one or more unmeasured factors, the Introduced Variable-outcome association is not confounded by unmeasured factors. This offers the potential, in some cases, to address an important source of uncertainty in the ACCE Method, as discussed in the text and Appendix 3.2. This uncertainty concerns whether the effect of inserting the Introduced Variable is adequately captured by insertion of the Introduced Variable-outcome regression coefficient into the Bross equation if correlated unmeasured confounding exists.

Thus, if the Introduced Variable does not show confounding amplification (e.g., shows a stable coefficient in several models which have differing R² values, and for which the “2^nd order” Introduced Variable also has a stable coefficient), then this constitutes a line of evidence suggesting that unmeasured confounders correlated to the Introduced Variable are limited or do not exist. Thus, this finding helps diminish concern about this potential uncertainty. To the extent that this approach can be used to identify Introduced Variable(s) that do not appear to be correlated with other unmeasured confounders through this approach, the Introduced Variable(s) can then be inserted into the propensity score with some confidence that uncertainty from both the change in correlated unmeasured confounders, and from confounding amplification of the Introduced Variable-outcome regression coefficient, is likely to be relatively minor.

How commonly Introduced Variables can be identified that are uncorrelated with unmeasured confounders remains to be determined. If typically they are difficult to identify, however, the first method described above of sequentially repeating the ACCE Method calculations to estimate the unconfounded Introduced Variable-outcome coefficient could then be considered.

Appendix 4.2. Final observations, including an initial consideration of whether the method systematically underestimates or overestimates residual confounding

It may seem that the number of potential uncertainties about the method (such as those mentioned in Appendix 2, Appendix 3, and Appendix 4) are daunting, but it should be recognized this is likely partly attributable to the fact that the method has been just formulated. As data is gathered about the method’s performance in practice, on either simulated or real-world datasets, it likely will become apparent which concerns may have a discernible impact on the treatment effect estimates, and which concerns are much more minor. It is hoped that the detailed anticipation of potential sources of uncertainty provided here helps facilitate the rapid conduct of such research.

It is important to note that these uncertainties all relate to the methods used to estimate and correct for the different components of confounding, when it exists. Some may produce imprecision that is difficult to predict in advance, such as the extent to which similar patient samples and “dose” of intervention is maintained between the models, and the possibility that correlations may only be able to be addressed incompletely). Some uncertainties might tend to lead to overestimates of confounding, such as whether the Introduced Variable regression coefficient is subject to confounding amplification, as well as, if adjustment for the included covariates is performed (Appendix Figure 1c), the regression coefficients for the included covariates. Some uncertainties might lead to underestimates of confounding, such as the possibility of constraints to amplification. Of these possible sources of imprecision in the estimate, those that lead to underestimates of confounding would be the most concerning, because they would result in an overestimate of the treatment effect. This is precisely the reason why simulation and real-world research is clearly needed. Such research might indicate the general accuracy of the estimates which are generated by the method and if the method tends to primarily overestimate or underestimate residual confounding. If a systematic overestimation or underestimation is indicated consistently, then this finding would still allow the results from this method to be used as an upper or lower bound on residual confounding.

Given that many of the uncertainties surround the approaches that the method uses to quantify residual confounding when it exists, it may prove that this method has its greatest value when it demonstrates a lack of confounding amplification, and it may be prudent to give those results the most weight for the time being. Part of the value of this method may be to provide a widely applicable method to making the determination of whether an estimate appears to be largely unconfounded. The wide applicability would result from the permitting use of an Introduced Variable with an association with outcome to probe for this unconfoundedness (although if the Introduced Variable has a substantial association with outcome, this then necessitates at least one of the quantitative adjustments for which there may be imprecision), or the use of a set of variables as the Introduced Variable. I also anticipate that the method, even when it does not provide a strictly accurate unconfounded treatment estimate, will provide an estimate that is beneficially closer to the unconfounded treatment estimate than conventional analysis, as well as indicate the general size and direction of residual confounding. However, these suppositions need to be demonstrated through research.

7.1. Tentative suggestions for implementing the ACCE Method based on current knowledge

Much simulation and real-world research needs to be performed to evaluate the degree to which the ACCE Method provides valuable estimates of unconfounded treatment effects, or at least estimates closer to the likely unconfounded treatment effect than the conventional estimates obtained prior to application of the method. Such research may include potential “tuning” of the method to simulated or real-world datasets through determining the optimum choices for the method’s implementation, either in general or in specific circumstances (see below, and preceding Appendices). Research of this type must occur before strong recommendations can be advanced concerning preferred approaches to use in implementing this method. Such research would also clarify the weight that should be given from estimates from this method compared to other methods for addressing unmeasured confounding. Hopefully, the method has been communicated in the manuscript and these Appendices clearly enough that researchers can start to implement the method in a variety of datasets to determine, and determine how to optimize, its performance.

Nevertheless, some researchers may understandably have a current interest in preliminarily exploring what this method suggests about the likely presence and size of unmeasured confounding in their analyses. A few key considerations suggest themselves:

First, there appear to be three key qualities for the Introduced Variable or Variables:

The ideal Introduced Variable would presumably possess both attributes #1 and #2 (a strong association with exposure and little to no apparent correlation with other unmeasured confounders). The association with outcome (#3) is expected to be less crucial, especially if a lack of correlation with other unmeasured confounders is suspected. In this circumstance, the effect upon confounding of adding this variable into the propensity score, regardless of its association with outcome, will hopefully be adequately captured by the regression coefficient/Bross equation-based adjustments outlined in the manuscript. However, as discussed in Appendix 4.2, at this point the method’s estimates probably should be seen as at least slightly more uncertain whenever a substantial quantitative adjustment to the Model 1 treatment effect estimate has to be performed. In other words, the method may allow for the most confident conclusions to be drawn when the method suggests little residual confounding exists.

Among the first two attributes listed, it is difficult to offer definitive advice about which to prioritize until the method’s performance in the context of substantial random variation is determined. If random variation represents a major threat to the accuracy of the estimates resulting from the method, then high priority will likely need to be given to adding an Introduced Variable or set of Introduced Variables that is strongly associated with exposure. This presumably would produce the greatest expected change in the treatment effect estimate, likely increasing the chance of detecting a relatively accurate change in the setting of substantial random variation. Similarly, research is also needed to determine the extent to which the proposed adjustments to quantify the effect of inserting the Introduced Variable into the second model also successfully captures the change that results in other, correlated, unmeasured confounders. Until that point, it would appear that choosing an Introduced Variable with little or no apparent correlation with other unmeasured confounders, if one or more can be identified, would be advisable. How frequent such variables are routinely available is uncertain. It may prove possible, however, to screen for such variables through selection procedures based on detecting an apparent lack of change in the Introduced Variable(s)-outcome association when unmeasured confounding amplification is expected (this procedure is broadly outlined in Appendix 4).

The lack of association with outcome appears to be a less important consideration than the preceding two. However, if it is difficult to demonstrate with satisfaction that an Introduced Variable appears uncorrelated with unmeasured confounders, than there may be an advantage to choosing an Introduced Variable with a minimal association with outcome. At a minimum, such a strategy would appear to have a reasonable chance of minimizing confounding amplification of the Introduced Variable-outcome association, especially if there were no a priori reasons for believing the variable would have an association with outcome. A variable thought to have no association with outcome that has a regression coefficient close to the null likely has a better chance of having minimal correlation with unmeasured confounders than one with an observed association with outcome, although it is always possible that the apparent null association results from an Introduced Variable with a substantial association with outcome closely counterbalanced by correlated unmeasured or nonincluded confounders biasing in the opposite direction. Again, however, the test of the sensitivity of the Introduced Variable coefficient to the number of variables in the model proposed in Appendix 4.1 might permit the evaluation of that possibility (by determining if substantial correlations with unmeasured or nonincluded confounders are suggested.

Whether using Introduce Variables with a null or near-null association with outcome is any advantage over conventional instrumental variable regression, however, is uncertain. It is conceivable that using the ACCE Method with such variables may provide additional statistical power over a conventional instrumental variable regression, and/or may not as prone to the biases that weak instruments can engender (especially if the use of a set of Introduced Variables produces stronger changes in the prediction of exposure). These are untested possibilities, however, and await both systematic investigation of the method’s performance in general and development of methodology to generate confidence intervals via bootstrapping or other approaches.

In addition to the qualities of the Introduced Variable, at least one additional major strategic consideration will likely influence ACCE Method model-building decisions: the judgment of how extensive Model 1 should be (i.e., the number of covariates it contains). In general, there are obvious advantages of having the model be as extensive as possible (i.e., to adjust/balance for as much measured confounding as possible). Methods for minimizing measured confounding in such a fashion have been developed for decades, while the ACCE Method has just been proposed. A second clear-cut reason for attempting to control as much measurable confounding as possible is that such a strategy reduces the risks that the nonincluded covariates will represent “unmeasured confounding” that is correlated with the Introduced Variable(s). Since the method’s performance in the setting of such correlations is more uncertain, it makes sense to attempt to construct the models to avoid this condition as much as possible. Certainly, at a minimum, it would seem like any known confounders that clearly are correlated with the Introduced Variable should be included as propensity score covariates. A third reason to maximize the number of included covariates is that in general, the greater degree to which exposure is predicted, the larger the predicted amount of confounding amplification between Model 1 and Model 2. This larger proportional confounding amplification means that the quantitative estimate of confounding amplification, with its associated uncertainty, is multiplied by a lesser factor than if little difference in proportional confounding amplification is observed. (For an extreme example, if two models are compared, one with an R² for exposure of 0.1 and one with an R² of 0.2, this means the quantitative change in treatment effect estimates would be divided by (1 / (1 - 0.2) / 1 / (1 - 0.1)) - 1 = ((1 / 0.8) / (1 / 0.9)) - 1 = 1.125 - 1 = 0.125. Dividing the change in treatment effect estimates by 0.125 is the equivalent to multiplying this change by a factor of 8! In general, it seems more prudent to, if the opportunity exists, work in a higher range of exposure prediction where the treatment effect estimate difference is multiplied by a much lesser factor).

However, despite the attractiveness in general for making Model 1 as complete as possible, there are a couple considerations which may, in certain circumstances, support creating a Model 1 that is deliberately less extensive than might be possible. The first consideration is the simple reality that the Introduced Variable association with outcome needs to be determined via regression. The number of variables that can be inserted into a regression equation depends upon the number of outcomes, while the number of covariates that can be introduced into a propensity score is often far greater, since this quantity relates to the total number of individuals exposed. Thus, in settings with relatively few outcomes, the number of covariates that can be entered into a propensity score may easily exceed those that can be included in a regression without risking bias. Second, if it is determined that the apparent nonlinearity that starts to affect the confounding amplification relationship above an R² of 0.56 in simulation is genuine, and this nonlinearity cannot be modeled mathematically and/or reflected in the change in “internal markers,” then a strong reason might exist to keep the number of propensity score covariates in Model 1 smaller than otherwise might be possible. A need would exist to keep the Model 1 propensity score covariates at a number for which addition of the Introduced Variable or variables in Model 2 does not produce an R² greater than 0.56. However, it is not clear whether this observed nonlinearity relates to some specific characteristics of this simulation. Third, the possibility of “constraints” upon confounding amplification is another possible reason for limiting the number of covariates representing measured confounders in the Model 1 propensity score. To the extent that some level of “constraints” to confounding amplification occur within real-world data (which is currently unclear), the general expectation would be that these constraints would be less pronounced at lower R² values of the two models (since less overall amplification would likely be resulting) than higher R² values. Finally, it is worth considering the desire to keep Model 1 and Model 2 as comparable as possible except for the Introduced Variable. Differences between the models in balance of measured confounders, patient sample, or dose or duration of the intervention could produce different underlying “treatment effects” in the two models, although how large these differences would typically be is uncertain. In general one would expect that as the amplification becomes greater between the two models, the potential for a difference in these factors becomes greater. These differences can be measured, however, and it may prove that this consideration is relatively minor compared to, for example, the advantage of substantially amplifying confounding in the setting of substantial random variation.

As mentioned in Appendix 2.4 and Appendix 4.2, the number of considerations that potentially may need to be juggled when implementing the ACCE Method may seem daunting. However, this may partially reflect, at least in part, the lack of testing of the method that has thus far occurred, since the method has been just proposed. I would anticipate that as the method is implemented in simulated and real-world datasets, the relative importance of some of these components relative to others will become much clearer.

7.2. Potential advantages of multiple applications of the method using different Introduced Variables/Introduced Variable sets

One possibility that may emerge, especially as a better sense is developed of the impact of random variation and the strength of association with treatment exposure typically needed for reliable estimates, is that this method may perform best with multiple applications of the method featuring different Introduced Variables or Introduced Variable sets. These multiple applications of the method would obviously be in addition to any bootstrapping to obtain confidence intervals using the same Introduced Variable(s) (if this is the method settled upon for confidence interval generation). These multiple applications could either involve simply introducing different Introduced Variables to the same Model 1 or, probably a superior but more labor-intensive approach, to different Model 1s. This more labor-intensive approach would address the possibility that the correlated included and unmeasured confounders would likely vary from Introduced Variable to Introduced Variable, and because it may be desirable to include the other candidate Introduced Variables in the Model 1 when these variables are not being used as Introduced Variables. In the ideal scenario, even though each of these comparisons would involve at least slightly different measured covariates and, thus, different amounts of residual/unmeasured confounding in Model 1, the ACCE Method would yield the same, or close to the same, ultimate value for the unconfounded treatment effect.

If a similar estimate of the unconfounded treatment effect is not obtained, then one approach would be to choose the estimate that is most “conservative” (i.e., yields the lowest unconfounded treatment effect estimate). Whether in practice this might end up being too conservative (depending on the potential impacts of random variation) remains to be determined.

While the approach of applying this method multiple times is clearly more labor-intensive, in the era of Big Data, ample computer power, and automation, such an approach likely would be justified if it substantially improved the method’s estimate of unmeasured confounding. (The same consideration may apply to other potentially labor-intensive steps, such as estimating the residual confounding contributed by each of the included propensity score covariates [Appendix 2.1 and Appendix Figure 1c]).

7.3. Final thoughts on the ACCE Method and the need for additional research

What has been outlined here is a method that, in theory, can derive a treatment effect estimate from nonrandomized studies less biased from residual baseline confounding, even when some of the residual confounding is unmeasured. It is even possible that in some cases this approach may also permit derivation of a treatment effect estimate that is less biased from confounding arising after treatment initiation as well (Appendix 2.3b). The logic of the method is relatively straightforward. In addition, its core principle that introducing a predictable amount amplification of residual confounding should permit estimation (through extrapolation) of the amount of confounding prior to amplification may have enduring value, even if the specifics of the method outlined here are altered by future research. What remains to be determined is how well the method actually works in practice, when applied to simulated and real-world data.

It is difficult to predict the method’s performance a priori. While many challenges can be anticipated, further study of this method appears clearly warranted. Importantly, some of the many potential challenges anticipated in this manuscript would be expected to reduce the accuracy of the proposed method but not invalidate the basic approach. It also should be noted that the potential exists for this method to provide several distinct benefits: 1) the potential to use a greater variety of variables than strict instrumental variables as the basis to address unmeasured confounding (specifically, being able to use variables to that have associations with outcome, as well as those that do not); 2) the apparent potential to use a set of variables to produce confounding amplification when no single variable is sufficient, and 3) possibly the potential to address confounding after treatment initiation as well as at baseline. Thus, this proposed method seems to have enough potential advantages to justify its further investigation as one approach that might contribute to the broad aim of making nonrandomized treatment effect estimates less subject to confounding. Such research would also help establish whether this method also might contribute to the more challenging objective of deriving nonrandomized treatment effect estimates that truly approximate those that would be observed in randomized trials in that population.

For all these reasons it is hoped that the extent and the detail of the information provided here will allow research on this proposed method to proceed rapidly. Such research would allow for the important determination to be made regarding where this method stands in value compared to other methods proposed for obtaining unconfounded treatment effects.

Summary Equation for the ACCE Method^h,i

^a”Internal Marker” = a measured covariate deliberately not included in the propensity score that is generally uncorrelated with other propensity score covariates. The internal marker serves to index the amount of confounding amplification between treatment groups that occurs between Model 1 and Model 2. If an internal marker is used, then Confounding Amplification (Camp) = Final Internal Marker Covariate Imbalance / Initial Internal Marker Covariate Imbalance.

^bFor all associations involving the Introduced Variable, the association would include the association of the Introduced Variable plus the associations of its correlates, to the extent that these associations influence the observed association between the Introduced Variable and outcome. When balance in the Introduced Variable is referenced, this also refers to balance in both the Introduced Variable and, to a lesser extent, its correlates.

^cEither within-treatment arm or overall regressions can be performed. Examining the association within treatment arms prevents the association with the intervention, which may be substantial, from influencing the estimation of the IntV-Outcome association. The association between Introduced Variable and Outcome is an aggregate of direct and indirect associations. This aggregate association is then used in Step 3b to estimate the quantitative effect on the treatment effect estimate of adding the Introduced Variable into the propensity score (and increasing its balance through stratification or matching) that is independent of confounding amplification.

^dBased on averaging the coefficients (i.e., ln(RR)). The most straightforward circumstance for the within-treatment arm approach is if the observed association is highly comparable in both treatment arms and both models. If so, as an approximation these values can be averaged. Determining these Introduced Variable-outcome regression coefficients separately for each model has an essential function. In Model 2, the Introduced Variable-outcome association would also be expected to suffer some confounding amplification relative to Model 1. In this hypothetical example, the observed association is varied slightly to illustrate that (Appendix 4).

^eSince the Introduced Variable(s) is a measured covariate or covariates, it is possible to determine its initial imbalance in Model 1, and the degree to which this imbalance changes in Model 2. This information can then be combined (through use of the Bross equation⁸) with the coefficients derived in Step 3a to estimate the component of the change in the Treatment Effect Estimate between Model 1 and Model 2 that is attributable to increased balance in the Introduced Variable(s) and its correlates.

^fThe congruence between the scenario’s genuine treatment effect and total confounding and these values should not be seen as validating the method. The scenario’s effects estimates were selected based upon what would be expected from confounding amplification operating consistent with the system described here. However, this step-by-step example does illustrate how, absent the effects of random variability, the mechanics of how this series of calculations would function to produce the desired values (i.e., Total Residual Confounding and an Unconfounded Treatment Effect Estimate) from the initial values (i.e., the confounded Model 1 and 2 treatment effect estimates, an estimate of confounding amplification, and an estimate of the contribution to confounding of the Introduced Variable and, to a more uncertain degree, its correlates).

^gTo obtain the most rigorous estimate of the unconfounded treatment effect, in theory, two additional terms are necessary (Appendix 2.1, Appendix 3.2a and Appendix Figure 1c). The first term estimates the contribution of any change in imbalance in the measured, included propensity score covariates from Model 1 to Model 2 to Model 2 residual confounding (as estimated through multivariate regression coefficients and the Bross equation) needs to be subtracted from the change in the treatment effect estimates (in addition to Confint delta). This subtraction is needed since this change in imbalance in the included propensity score covariates represents another process separate from full confounding amplification (i.e., to the degree predicted by R²) that can contribute to the change in treatment effect estimate from Model 1 to Model 2. In general, I would expect the insertion of the Introduced Variable into Model 2 to typically worsen, at least slightly, any imbalance in the propensity score covariates that existed in Model 1. The second term represents the residual confounding in Model 1 attributable to the fact that the propensity score variables are not brought into perfect, 50/50 balance. The practical significance of these components in most instances is uncertain and these steps add considerable labor. Nevertheless, if these calculations are able to be performed routinely (perhaps by automating the process), this will ensure those instances where this adjustment is important will not be overlooked. Please see Appendix 2.1 and Appendix 3.2a for further discussion and Appendix Figure 1c for the more complex, more strictly rigorous, equation.

^hIf preferred, the equation can be written with a slightly simpler denominator:

However, I emphasize the slightly more elaborate form in this manuscript for two reasons. First, I think it makes the less reduced form of the equation makes clearer why the reduced form of the equation has the form it does (i.e., why the fraction in the denominator simplifies to: $\frac{(1-R_{\text{M1}}^{\text{2}})}{(1-R_{\text{M2}}^{\text{2}})}$). Second, I also think the less reduced form of the question, which relates to the understandable concept of “the confounding amplification of Model 2 divided by the proportional confounding of Model 1” in straightforward fashion, makes the overall equation seem more straightforward and easily understood, and hopefully, easier to remember.

In addition, the equation could also be alternatively written as follows:

a form which emphasizes the fact that both terms subtracted from TEE_M1 together constitute the sum of residual confounding.

ⁱFor the scenario posed at the beginning of the table, the method’s Summary Equation would arrive at the following estimate of the unconfounded treatment effect:

beta coefficient = 0.2351 - ((0.2612 - 0.2351) - (-0.0273)) / (((1 / (1 - 0.5)) / (1 / (1 - 0.25))) - 1) - 0.0293 = 0.0990 (unconfounded TEE) (exponentiated, the unconfounded TEE equals RR = 1.10).