Analyses within risk strata overestimate gain in discrimination: the example of coronary artery calcium scores

Introduction

Several new biomarkers, including coronary artery calcium scores (CACS), have been proposed to improve cardiovascular (CVD) risk prediction models, such as the Framingham Risk Score (FRS) or Pooled Cohort Equations (PCE). Their incremental value is usually judged by any improvement discrimination, using measures such as the C-statistic - the area under the receiver operating characteristic curve (AUC) - despite some limitations.^1,2

Several recent assessments of CACS added value to CVD risk models were done within CVD risk-strata specific gain but restricting the study population in this way may inflate the apparent gain.

Methods

In the process of a systematic review to assess the incremental value of CACS beyond traditional CVD risk assessment, we identified two studies that report the change in C-statistic from adding CACS within CVD risk strata as well as for the overall cohort.³ We used these two studies to illustrate observed paradoxes, and then explore possible reasons using a simple simulation. All analyses were performed with R Project for Statistical Computing (version3.6.3, RRID: SCR_001905).

Results

Two studies provided sufficient data - the Heinz Nixdorf Recall (HNR) and Multi-Ethnic Study of Atherosclerosis (MESA) studies.^4,5 Both compared the C-statistics of base models to C-statistics of extended models (including CACS) in sub-groups defined by CVD risk scores. The apparent increase in C-statistic from adding CACS was greater within every risk sub-group that for the overall cohort (Figure 1) – so all strata gains were above average. There are two paradoxes, the first explaining the second.

Figure 1. Overall and stratum-specific C-statistics for base and CACS extended model in two studies.^3,4

Data of Panel A extracted from Geisel 2017, Table 3; Data of Panel B extracted from Blaha 2021, Figure 2. Both studies compared the C-statistics of base models (FRS in HNR, PCE in MESA) to C-statistics of extended models (including CACS) in sub-groups defined by CVD risk scores. CACS: coronary artery calcium scores.

The first paradox is that the discriminative ability of the CVD risk score within individual CVD risk strata is worse than for the overall population. This surprising “finding” is a statistical artefact: the discriminative ability of a variable will always appear to be less if its range is limited (or within a more homogeneous population), than within the full (more heterogeneous) population.⁶

The second paradox is the apparent gain in C-statistic for CACS added to the base model is greater within each individual risk strata than for the whole study population. This is not a true “gain”: within each CVD risk stratum the “discrimination” is artificially reduced, and hence the “gain” from CACS artefactually increased. This results in overestimation of the improved discrimination provided by CACS.

Discussion

These two paradoxes related to stratification may seem somewhat surprising but may be more readily understood with other examples. Intelligence quotient (IQ) might be predictive of a young person’s future income level, but any discrimination is weakened by assessment within 10-unit IQ strata. Similarly, blood pressure predicts future stroke, but this prediction is weakened if examined within 10 mmHg bands. This apparent weaker predictive ability is due to the artificial constriction of the predictor and the nature of the discrimination measure.

Figure 2 provides a hypothetical example to help explain these paradoxes. Figure 2A shows 42 people - 21 who have an event and 21 do not - grouped into low, moderate, and high risk according to a risk score. The C-statistic is good for the overall cohort (0.78), but lower in the narrower risk subgroups of 14 people (low risk: 0.61, moderate risk: 0.57, high risk: 0.61), because some of the “discrimination” is already used in separating into these groups. Figure 2B adds a second prognostic factor which “improves” the C-statistic more within each of the risk subgroups (low risk: 0.02, moderate risk: 0.03, high risk: 0.03) than in the overall cohort (0.01).

Figure 2. A simple illustration of the paradoxes.

Figure 2A shows 21 who had event (red dots) 21 do not (blue dots) - The C-statistic for the overall cohort (0.78) is higher than in any of three risk subgroups (low risk: 0.61, moderate risk: 0.57, high risk: 0.61). Figure 2B – For a second indicator (crosses; Odds Ratio of ~2.0) added to model the C-statistic “improves” more in each of the risk subgroups (low risk: 0.02, moderate risk: 0.03, high risk: 0.03) than in the overall cohort (0.01).

Given the increasing use of risk stratified analyses of prognostic gain, we recommend the incremental discrimination provided by a new biomarker should not be analysed within risk stratified subgroups based on the CVD risk score. Authors, reviewers, and editors should be aware of this flawed analysis and avoid it. More generally, the limitations of discrimination measures¹ mean we should consider alternative measures to assess the incremental value of new biomarkers⁷ and be wary of stratified analyses, particularly when the stratification and the base CVD risk score are the same.

Data availability

All data underlying the results are available as part of the article and no additional source data are required.

Author contributions

L.Z.: Methodology, Software, Writing - Original Draft. K.B.: Methodology, Supervision, Writing - Reviewing and Editing. A.S.: Supervision, Writing - Reviewing and Editing. P.G.: Methodology, Conceptualization, Supervision, Writing - Reviewing and Editing.

Competing interests

No competing interests were disclosed.

Grant information

KB is supported by NHMRC Investigator grant 1174523. PG is supported by NHMRC Australian Fellowship grant 1080042. This study was funded by NHMRC Centre of Research Excellence grant 2006545.