Derivation and validation of a mortality risk prediction model in older adults needing home care: Updating the RESPECT (Risk Evaluation for Support: Predictions for Elder-Life in their Communities Tool) algorithm for use with data from the interRAI Home Care Assessment System

Source of data

Since 2004, the care needs of home care clients expected to require at least 60 days of service (i.e. long-stay clients) have been determined using the RAI-HC, a standardized, multi-dimensional assessment instrument that contains information across a variety of elements including sociodemographic information, cognitive and functional capacities, chronic diseases and comorbidities, and signs of health instability, as well as recent use of health care.^13–15 In April 2018, the RAI-HC was replaced by interRAI HC, an updated version of the assessment instrument that contains similar data elements.¹⁶ RESPECT was initially developed and validated using RAI-HC data between 2007 and 2013. To estimate the 6-month mortality risk in a contemporary cohort, we validated and assessed RESPECT in the interRAI HC data collected between April 1, 2018 and September 30, 2019.

Record-level interRAI HC data was linked to other provincial health administrative databases housed and analyzed at ICES. ICES is an independent, non-profit research institute whose legal status under Ontario’s health information privacy law allows it to collect and analyze health care and demographic data, without consent, for health system evaluation and improvement.

Participants and sample size

The model derivation cohort consisted of adults aged 50 years or older who were eligible for publicly funded home care in Ontario and received at least interRAI HC assessment between April 1, 2018 and September 30, 2019. Assessments with missing information on sex and dependence across activities of daily living (ADLs) were excluded. The exclusion criteria applied to create our study cohort are presented in Figure 1. The internal validation cohort comprised 500 bootstrapped samples, each containing a random selection, with replacement, of 5% of the total cohort.

Figure 1. Cohort creation.

Outcome

The primary outcome was death within 6 months of an interRAI HC assessment. The Registered Persons Database, a registry of health card numbers that have been issued under the Ontario Health Insurance Plan to all eligible residents of Ontario, was used to ascertain death date.

Predictors

All predictors in the original RESPECT model¹² were included unless the variable was not available in the interRAI HC instrument. Predictors include measures of physical function (dependence in ADLs or instrumental ADLs [IADLs], and worsening ADLs); cognitive impairment (worsening decision-making capacity); sociodemographic factors (age, sex at birth); diseases (stroke, congestive heart failure [CHF], coronary heart disease [CHD], Alzheimer’s disease and other dementias, multiple sclerosis, Parkinson’s disease, cancer, chronic obstructive pulmonary disease [COPD]); healthcare use (number of hospital admissions or emergency department visits in the last 90-days); symptoms of reduced health (vomiting, edema, dyspnea, low fluid intake, weight loss, decrease in food or fluid consumption); prescription and receipt of life-sustaining therapies/treatments (chemotherapy, dialysis, oxygen therapy, ventilator or respirator); and clinician diagnosis of an end-stage disease. Missing data on a predictor was considered as not present with the exception of worsening ADLs and worsening decision-making capacity. Other cohort characteristics (i.e., the year that the interRAI HC assessment was performed and the reason for assessment) were also included to account for remaining heterogeneity and temporal trends. Predictors in the original RESPECT model that were not available in the interRAI HC data and, therefore, not included in the re-estimated model are as follows: education, hypertension, asthma, emphysema, renal failure, and any interactions with at least one of the aforementioned variables. In total, there were 27 predictors with 51 degrees of freedom in the final model. All predictors were specified as categorical variables except for age, which was modelled using a restricted cubic spline with five knots, placed at the 5th, 27.5th, 50th, 72.5th and 95th percentiles of the age distribution.

Statistical analysis methods

We estimated a Cox proportional hazards regression predicting death within 6 months of an interRAI HC assessment. All assessments performed between April 1, 2018 and September 30, 2019, were included and follow-up was censored by death or end of follow-up (i.e., 6 months after the assessment date). A robust sandwich variance estimator was used to account for within-subject correlations given the possibility of multiple assessment records for individual home care users within the study period.

For model validation, bootstrap sampling with replacement was performed. We generated 500 bootstrapped samples, each containing a random selection of 5% of the total cohort. Then, coefficients from the derivation model were applied to the validation cohort to estimate the 6-month predicted mortality risk. Predictive performance was assessed in the validation cohort using two predictive accuracy measures, discrimination and calibration. Discrimination was assessed using the c statistic. Calibration was assessed for the overall model as well as across all predictors by comparing predicted risk of death at 6 months and observed risks that were derived from Kaplan-Meier estimates of the survival function.

Risk groups

To create risk bins, we first examined the median survival across percentiles. To estimate the median survival of individuals who survived past 6 months, we extended the follow-up period for each assessment to the most recent data available at the time of this study (i.e., June 29, 2022). Then, bins were constructed based on recommendations of clinical experts who identified meaningful differences in life expectancies that would be helpful in decision-making. This resulted in the creation of 61 risk bins. As the bin number increases, the 6-month mortality risk decreases and median survival increases (Table 1). Between bins, the incremental increase in median survival varies. For example, the incremental increase in median survival for bins 1-5 (i.e., individuals with high mortality risks) is approximately 3 weeks, while increases in median survival for bins 56-61 (i.e. individuals with a lower mortality risk) is greater than 2 months. Sample sizes for each risk group, their predicted 6-month mortality risk and median survival are presented in Table 1. See Table 2 for derivation of RESPECT Risk Score and Bin.

Ethics approval

The data used in this project is authorized under section 45 of Ontario’s Personal Health Information Protection Act and does not require approval by a Research Ethics Board.

Participants

The derivation cohort comprised of 247,377 community-dwelling older adults who used home care during our study period (Table 3). They contributed to a total of 405,689 interRAI HC assessments, covering 822,051.47 person-years (PYs) of follow-up. The median number of interRAI HC assessments performed per adult was 1 (interquartile range [IQR] 1-2), with a maximum of 10 assessments. Within 6 months of assessment, 35,497 (14.35%) of home care users died and 42,515 (10.48%) of assessments were associated with a death within 6 months. The validation cohort consisted of 5% of the derivation cohort, resulting in 500 boostrapped samples with an average of 19,255.02 interRAI HC assessments and 41,105.61 PYs of follow-up per sample. The median number of assessments per adult in each sample was 1 (IQR 1-1) with a maximum of 3.91 visits per person. Within 6 months of an interRAI HC assessment, an average of 20,058 (10.69%) of home care users died and 2,128.77 (10.49%) of assessments were associated with a death within 6 months. No assessments were excluded due to missing data on predictors included in the model (see Figure 1 for cohort creation).

A summary of characteristics of home care clients included in this study is provided in Table 4. In the derivation cohort, the mean (SD) age was 80.5 (10.8) years at the time of assessment and the majority of patients were female (61.9%). Alzheimer disease or other dementias was the most prevalent of the included comorbidities (33.4%) followed by coronary heart disease (30.4%) and stroke (17.0%). Other comorbidities included in the model (coronary heart failure, multiple sclerosis, Parkinson’s, cancer and chronic obstructive pulmonary disease [COPD]) had a prevalence of less than 15%. The most common symptoms of health instability reported were dyspnea (43.0%) and edema (34.7%). Other symptoms had a prevalence of less than 11%. Only 1.7% and 1.6% of patients received chemotherapy or dialysis, respectively, while 5.1% received oxygen, ventilator or respirator. A small proportion (2.3%) of home care patients had a prognosis of having fewer than 6 months to live and most did not have an inpatient admission (68.8%) or emergency department visit (75.8%) over the past 90 days. A significant proportion of the home care users (77.9%) required extensive assistance (score of at least 4) in performing IADLs (i.e., preforming ordinary housework, meal preparation or using the phone). A smaller proportion (18.0%) required extensive assistance (score of at least 4) in performing ADLs (i.e., maintaining personal hygiene, using the toilet, locomotion and eating). However, nearly half (49.3%) had reported worsening capacity to perform ADLs and more than a quarter (27.7%) reported worsening decision-making capacity. The majority of interRAI HC assessments were first assessments (58.9%) and were in 2019 (53.8%). The derivation and validation cohorts are comparable across all baseline characteristic (Table 4).

Model specification

Table 5 presents the hazard ratios (HR) for the derivation cohort derived from the Cox proportional hazard regression. Total dependence in ADLs (self-performance score of 6) (HR 3.2, 95% confidence interval [CI] 3.0-3.5) and diagnosis of an end-stage disease (HR 2.8, 95% CI 2.7-3.0) were most predictive of 6-month mortality. These were followed by ADL self-performance score of 5 (HR 2.0, 95% CI 1.9-2.1), a reported history of needing oxygen therapy without COPD (HR 1.8, 95% CI 1.7-1.9), total dependence in IADLs (HR 1.8, 95% CI 1.6-2.0), chemotherapy without cancer (HR 1.6, 95% CI 1.2-2.1), chemotherapy with cancer (HR 1.6, 95% CI 1.5-1.7) and ADL self-performance score of 4 (HR 1.6, 95% CI 1.5-1.6). The remaining predictors had a HR less than 1.5. Of the signs and symptoms of health instability, weight loss was most predictive of 6-month mortality (HR 1.5, 95% CI 1.4-1.5) followed by a noticeable decrease in food or fluid consumption (HR 1.4, 95% CI 1.4-1.5). Edema was least predictive of 6-month mortality (HR 1.0, 95% CI 1.0-1.1). Worsening decision-making capacity and worsening ADL status had hazard ratios of 1.0 (95% CI 1.0-1.1) and 1.4 (95% CI 1.2-1.5), respectively.

Model performance

The calibration plot displays the mean predicted 6-month mortality risk against the Kaplan-Meier survival estimates across all 61 risk bins in the validation cohort (Figure 2). The model is well-calibrated across most risk bins; the calibration-in-the-large was 1.19 percentage points, on average, and the calibration slope was 0.88 (95% CI: 0.86-0.91). The model over-predicts risk (by 5.2% to 15.5%) in the highest mortality risk bins (bins 1 to 6), where the median survival was less than 6 months. To a smaller extent, the model underpredicts risk (by a magnitude of 0.5% to 2.6%) among individuals with moderate mortality risk (i.e., a mortality risk between 10% to 30% or bins 12-26). Overall, model c-statistic was 0.76 (95% CI 0.75-0.77) at 6 months, suggesting good discriminative ability in the validation cohort. Calibration was also evaluated in several subgroups of predictors, including functional status (i.e., ADL and IADL scale scores), select diseases, and the receipt of select life-sustaining therapies/treatments (Figure 3). The model is well-calibrated across all subgroups, with less than a 1.6 percentage point deviation from the observed 6-month mortality risk.

Figure 2. Calibration Plot for 6-Month Mortality Risk in the Validation Cohort Across the 61 Risk Bins from RESPECT.

Figure 3. Calibration Across Key Predictors of 6-Month Mortality.

Survival

The mean predicted 6-month probability of death in our full study cohort was 18.0% and ranged from 1.51% in the lowest risk bin (95% CI 1.0%–1.542%) to 95.95% in the highest bin (95% CI 95.75%–96.15%) (Table 1, Figure 2). Median Kaplan–Meier survival (Figure 4) varied from 36 days (12–130d at the 25th and 75th percentiles) in the highest risk group to over 3.5 years (>1,413d at the 11th percentile) in the lowest risk group.

Figure 4. Kaplan–Meier survival estimates of select risk groups in the derivation cohort.

Limitations

While the RESPECT algorithm demonstrates a good prognostic ability, the findings show current limitations of prognostication, particularly with regards to misclassification. REPSECT moderately overpredicts mortality among the highest risk bins. However, from a clinical application standpoint, these bins capture users with extremely poor prognosis who would likely benefit from the earlier provision of palliative care. The ability to prognosticate those near the end of life can be improved by including additional predictors, such as biomarker data, which will likely become an important area of future work to improve the predictive performance of algorithms like RESPECT.