Data-Driven Decision Support Tool Co-Development with a Primary Health Care Practice Based Learning Network

Background

Increasing amounts of everyday data coupled with advancements in technology and artificial intelligence (AI) and its major subfield of machine learning are transforming healthcare.^1–5 Primary health care settings have received less attention than other sectors, and there is a need for increased engagement of end-users in development of AI-enabled decision support tools.^6,7 This case study describes the process and lessons learned thus far in co-developing a decision support tool with and for a primary health care organization in Ontario, Canada.

Setting

The Alliance for Healthier Communities (Alliance) supports team-based primary health care through 72 Community Health Centres (CHCs) across Ontario, Canada to people who otherwise experience barriers to care. In 2000, all CHCs moved towards a common electronic health record (EHR; Telus Practice Solutions, version code 5.23.100) with standardized data requirements; each client’s EHR has structured fields for sociodemographic characteristics (e.g., sex, gender, education) and dynamic care encounter tables (e.g., ENCODE-FM v7.2024 or ICD -10 codes to indicate diagnoses and procedures) that capture information from all providers in their care team. All CHCs follow a standardized opt-out consent process for the use of de-identified data in research that reports aggregate results, which the analyses in this report fall under. De-identified data with consent for research use were stored on a secure server, with access through encrypted channels granted only as needed to members of the research team. The project was approved by the Western University Health Science Research Ethics Board in March 2018 (ID 111353).

Report structure

Figure 1 summarizes our work completed thus far, with additional details for the six major stages of work presented below in terms of goals and associated activities. The discussion section summarizes overarching lessons learned and reflections.

Figure 1. Case study overview.

First six stages of decision support tool co-development project. Legend: Alliance = Alliance for Healthier Communities, CHC = Community Health Centre, EHR = Electronic Health Record.

Stage 1: Population-Level Descriptive-Exploratory Study

Goals

Activities

Conduct study: This study (published elsewhere⁸) provided an overview of population health and care patterns by applying to EHR data both methods from traditional descriptive epidemiology (e.g., period prevalence of chronic conditions) and from unsupervised machine learning to explore more complex patterns (e.g., non-negative matrix factorization to examine care provider teams). Findings were shared with the Alliance community through a Lunch ‘n’ Learn Webinar in October 2022, and were revisited throughout the decision support tool project to inform problem selection and methodological decisions.

Stage 2: PBLN Team Engagement

Goals

Activities

Population data assessment: The Lunch ‘n’ Learn included an introduction to machine learning and decision support tools in addition to a summary of Stage 1 study findings. Embedded throughout the latter portion were polls and discussion points asking whether findings were consistent with expectations. Areas of inconsistency motivate further exploration to discern if the mismatch is an artifact due to data quality issues or methodological decisions, or if the mismatch is clinically relevant. Areas of potential concern (e.g., accurately high prevalence estimate) could be a good target for a decision support tool.

PBLN team formation: The Lunch ‘n’ Learn was the launching point of the decision support tool project, with additional invitations to participate distributed through the general Alliance community e-mail list and the recently formed PBLN member e-mail list. The resulting core team of people involved in remaining stages of work, described by general roles:

Stage 3: Decision Support Tool Problem Selection

Goal

Activities

PBLN meeting: In October 2022, the first PBLN meeting was held. We briefly reviewed descriptive epidemiology findings and discussed whether any outstanding questions needed to be answered to guide future steps. We then discussed data-driven decision support tools and brainstormed potential directions to pursue in the CHC context.

Discussion synthesis: Ideas were summarized into three “types” of decision support—risk prediction or screening, triaging specialized program entry, and identifying care access needs—with target conditions or application areas for each type ( Table 1).

PBLN meeting: In December 2022, we reviewed the three candidate project directions and while all (and more!) were seen as potentially valuable we decided to focus on one while exploring how to best conduct this type of project: risk prediction for mental health decline after diabetes diagnosis. Rationale included i) expected impact (e.g., high prevalence of diabetes with known mental health comorbidities, coupled with the challenge of needing to choose between many possible care options upon a new diabetes diagnosis); ii) actionable (e.g., all CHCs provide mental health care resources that could help prevent mental health decline for people living with diabetes); and iii) feasibility (e.g., relevant care captured in the EHR, and heavy focus on risk prediction in machine learning methods advancements).

Follow-up action: We connected with Diabetes Action Canada to learn more about related work and potential collaborators. Given the early nature and CHC-specific focus of our project, we proceeded with risk prediction model development using an existing 11-year retrospective extract of CHC EHR data, with the intention to consider expansion or tighter external collaboration after more internal feasibility and impact assessments.

Stage 4: Sandbox Case Study 1 – Individual-level risk predictions

Goal

Activities

Preliminary analysis: Candidate cohort summary characteristics, and an outline of potential predictor and outcome definitions and data sources.

PBLN meeting: In February 2023, we met to review the preliminary analysis and made initial decisions on the cohort and how to operationalize the outcome and predictors. This was done alongside further problem refinement and discussing clinical actions that could accompany a high-risk indication (Examples: brief educational discussion, noting down mental health to discuss in future appointments, referral to a specialized mental health provider, or referral to a CHC group program focused on mental health and/or diabetes).

Model development: The eligible cohort included 1,250 adult ongoing primary care clients receiving care at an East Toronto region CHC, who had at least one diabetes ICD-10 code⁹ in 2011-2018, at least one year of follow-up care, and no mental health care or decline indication in the two years prior to their incident diabetes indication. The outcome prevalence representing mental health decline after new diabetes diagnosis was 8.4%. Five candidate models ranging in complexity from simple linear (sklearn.linear_model.LogisticRegression v0.24.2) to complex machine learning techniques (CatBoost v0.26.1) were trained and compared using a five-fold nested cross validation procedure. Hyperparameters were selected on the inner loop using a grid search for the highest Area Under the Receiver Operating Characteristic Curve (AUROC).

Model performance: Table 2 presents summary performance metrics, with additional methods and results details available upon request.

PBLN meeting: In April 2023, we met to review and revise the initial model development analyses. While we decided the predictive performance of the sandbox model was not clinically useful, the associated discussion surfaced ideas around how to better harness information from the data and around problem refinement. Example insights included broadening eligibility from incident to prevalent cases of diabetes, modifying how to identify active diabetes care, and surfacing the need to engage more people to better understand how different provider types code mental health care in the EHR (e.g., do ENCODE-FM codes for “feeling anxious” vs “anxiety” reliably distinguish symptoms vs. diagnosis or is this more indicative of different provider type scopes). In terms of features, medication and lab value data are expected to increase accuracy of individual-level predictions; these data are planned for integration with the BIRT system (supporting data access), but not yet readily available.

Follow-up actions: We sought further input on the case study and suggested next steps in three ways: a PBLN member lead a focus group at their CHC, a poster presentation was given at a primary care research gathering, and an email call out for further input was circulated through the Alliance email listserv.

Problem refinement: A strong discussion theme was that while all CHCs provide mental health services, these are already at or near capacity and implementation of the decision support tool may increase demand past a point that could be maintained. This highlighted potential value of instead developing a system-level decision support tool to address the upstream problem of how to plan or advocate for adequate capacity within CHCs to address future mental health care service needs.

Stage 5: Sandbox Case Study 2 – Population-Level Planning Predictions

Goal

Activities

Model development: Example methodology changes from the Stage 4 case study were loosening eligibility criteria to include clients already receiving mental health care, broadening the outcome to include additional ENOCDE-FM codes (categories: emotional symptoms, symptoms involving appearance, suicidal ideation, affective disorder, and anxiety), and changing the missingness strategy for sociodemographic variables (collapsed not asked with no answer). We performed a similar five-fold nested CV procedure but restricted to feature-based models due to discussions (supported by case study 1 results) that preprocessed counts of chronic conditions should be more informative than granular codes.

Model performance: Of the 20,329 eligible clients with prevalent diabetes in 2016-2018, 22.2% had a mental health care outcome recorded in 2019. We used a naïve 0.5 probability cut-off for the best performing model (CatBoost v0.26.1) across all outer test folds to demonstrate the type of information that could be made available from this type of model to support capacity-planning decisions or advocacy. Overall model accuracy was 86%; CHC-specific accuracy calculated as the average accuracy across the five outer folds stratified by clients belonging to each CHC ranged from 64% to 97%, plus one CHC where there were no predicted or actual outcome cases (100% accuracy). The CHC-specific proportion of clients with diabetes predicted to have mental health care needs ranged from 46% (vs. actual 48%) to 1% (vs. actual 13%). Calibration performance is in Figure 2, and AUROC and AUPRC metrics are in Table 2.

Figure 2. Outer fold calibration performance for population-level prediction results.

Legend: LR = Logistic Regression, LRP = LR with L1 penalty, CATB = CatBoost, F# = Fold number in cross validation procedure.

Follow-up actions: We presented on all stages of the project at the annual Alliance Conference,¹⁰ and discussed the second case study at a PBLN meeting in July 2023. The sandbox model predictive performance showed promise and discussions further supported the idea that this type of advocacy and capacity planning tool would be a beneficial precursor to the individual-level tool in terms of actionability and associated clinical or system utility.

Stage 6: Project Recap and Next Steps Decision

Goal

Activities

PBLN meeting: In addition to discussing case study 2 results, the July 2023 PBLN meeting included a review of project progress summarized into three points ( Figure 3)—project scoping and problem identification, case study 1, and case study 2—alongside potential next steps with expected resource needs.

Figure 3. Summary of major progress and insights thus far.

Legend: Alliance = Alliance for Healthier Communities, CHC = Community Health Centre.

Next steps: Given the novelty of this project within the Alliance and the broader field of machine learning-enabled technologies for primary health care, the technical and process related progress made thus far towards a fully functional tool to support diabetes and mental health care additionally includes lessons that would benefit future projects of different focus. Therefore, we decided to pause to document our processes before revisiting the population-level planning tool direction.

Summary

Our decision support tool co-development approach with and for the Alliance started with a large-scale epidemiological study to learn about health and care patterns of the population of interest and about how clients are represented in CHC EHR data. We identified a priority problem of supporting proactive mental health care for clients with diabetes, and iterated between sandbox case studies and discussion sessions to further understand the problem, how a data-driven decision support tool could provide support, and the feasibility of achieving a high-quality technical solution given readily available EHR data.

Reflections and lessons learned

This project more generally furthers the Alliance’s learning health system work at three levels: data analysis capacity, stakeholder engagement, and process refinement.^11–15 First, demonstrating the use of machine learning to make data more meaningful through large-scale descriptive and real-time, clinically relevant predictive insights that will ultimately improve care delivery. Second, providing new avenues for clinician and provider engagement in data-driven learning initiatives; future work will additionally be able to engage the newly formed Client and Community Research Partners Program. Third, by providing a baseline process for tool development that future projects can learn from and build upon (i.e., what stages of work to keep, modify, or replace). Each future decision support tool initiative will provide additional opportunities for bidirectional learning whereby data are harnessed through AI and machine learning to tackle a specific clinical problem and improve care delivery, while simultaneously learning how to improve and adapt the processes to achieve that clinical goal for different types of decision support and application areas.

Ethics and consent

Consent for use of de-identified data for research purposes, whereby results are only made available in aggregate, was collected through a standardized opt-out procedure followed by the Community Health Centres where care was provided. Consent is collected via a written form, which is then scanned into the EMR. The need for additional consent specific to the research in this report was waived by the Western University Health Science Research Ethics board (ID 111353).