Combining implementation and data sciences to accelerate evidence integration into healthcare – ImpleMATE

Activity 1. Design data handling process to organize incoming data and a structured implementation ontology

Address research questions: D2K Q1-5

Handle incoming data and ensure data compatibility with large language models (LLMs)

We will focus on curating two types of data sources. The first data source will be published implementation literature: A data pipeline will curate implementation literature from PubMed research database through an application programming interface (API). The initial focus will be on published cancer genetics/genomics studies that used two key frameworks (collectively cited >9000 times for use in implementation studies,^69–71 and recently combined⁷²) to support evaluation and fine-tuning of LLMs: the Theoretical Domains Framework (TDF)⁷⁰ for individual behaviour change, and the Consolidated Framework for Implementation Research (CFIR)⁷³ for organizational and policy factors affecting implementation. Studies using alternative frameworks, other healthcare areas, and/or with unstructured implementation data will be our next focus. To ensure the literature is within the scope, we will use two approaches: (1) keyword searches in titles and abstracts, and (2) topic modelling with both statistical and machine learning (ML) approaches such as Latent Dirichlet Allocation (LDA)⁷⁴ and BERTopic⁷⁵ with embeddings to capture semantic representation of the literature for topic classification and filtering. Regular meetings with clinicians and implementation scientists will guide keyword and topic selection. The second data source will be existing and ongoing project data (e.g., surveys, interviews, meeting records). Initially, this platform will maximise the use of available data and resources across existing cancer genetics/genomics implementation projects in our project team. The data will/has been stored in a secure institutional university data repository.

The collected data will come in various formats, including PDFs, text files, audio recordings, and spreadsheets. To standardise the input for AI processing and ensure compatibility across downstream tasks, all data will be converted into plain text. Prior to any further processing, we will apply and compare de-identification techniques to remove personally identifiable information, in accordance with the State and Federal Australian Privacy Laws. For literature data, segmentation techniques (e.g., using regular expressions) will be employed to identify and separate sections, ensuring the input stays within the token limits (i.e., amount of content LLMs can process) required by LLMs and adheres to their specific formatting requirements. For project data, we will assess and address missing data and duplication. This preprocessing step is critical to ensure data sanitisation, privacy compliance, and security, while also preserving the integrity and accuracy of the data representation when deploying LLM-based solutions.

An implementation schema to guide the development of ontology

To support structured knowledge extraction and synthesis, we will define a comprehensive implementation schema in collaboration with stakeholders. This schema will capture key entities, their interrelationships, and the overall data structure. It will include categories such as implementation study characteristics, intervention characteristics, implementation determinants (i.e., factors influencing implementation success), strategies, and outcomes, including costs. For instance, study characteristics may include the study setting, design, target population, and geographic location. Interventions may involve, for example, whole body MRI scanning to more accurately detect cancers in patients with LiFraumini syndrome.⁷⁶ Implementation determinants may include system level factors such as workflow integration, changes to existing approaches for cancer prevention in these patients, and access challenges for patients in remote locations, and individual clinician factors including remembering to use this new approach, skills to interpret results, and trust in the technology. These factors can be aligned to established implementation frameworks (e.g., TDF and CFIR), allowing for a structured identification of barriers and facilitators. Strategies may include specific methods (e.g., system mapping and workflow redesign, prompts, audit and feedback) used to promote the adoption and integration of interventions into practice. Outcomes (which can be aligned to frameworks such as Proctor’s Outcomes Framework⁷⁷) may include measures such as fidelity, acceptability, and sustainability, as well as service level, clinical and economic outcomes, including associated implementation costs.^78,79 The schema will be developed through an iterative and collaborative process that involves gathering stakeholder feedback, refining entity definitions, and structuring the schema to reflect real-world implementation scenarios. This is to provide a structured framework for describing and managing an ontology to represent the implementation knowledge.

Throughout this process, stakeholders will be actively involved in reviewing and validating the framework, including assessments of inter-rater reliability to ensure consistency and clarity in the defined categories. Consumers will be engaged early to ensure that the ontology reflects patient priorities and is grounded in real-world healthcare experiences. This co-design approach ensures that the ontology is not only technically rigorous and accurate, but also practical, contextually meaningful, and sustainable over time.

The schema will be developed in alignment with the Open Biomedical Ontologies (OBO) Foundry principles,⁸⁰ ensuring the framework is interoperable, transparent, and adheres to accepted standards for ontology development. The implementation schema will support the identification and organisation of information that addresses critical implementation questions about what works, why, in which settings, for whom, and at what cost. An example of the implementation science ontology is illustrated in Figure 3.

Figure 3. An example of implementation science ontology.

Ethical sharing of project data and responsible use of AI

To develop a system that relies on both the sharing of research and live project data, and the synthesis of that data using AI, it is essential to establish a solid and sustained foundation for data privacy, security, and the responsible use of AI. To achieve this, we will establish a Principle Setting End User Committee, which will both inform the AI governance framework and act as the conduit to the wider group of relevant consumers, researchers, and clinicians. The End User Committee will coordinate and lead a series of workshop-style focus groups to present proposed data journey pathways and collaboratively explore approaches to optimise data access while maintaining privacy standards. The goal is to meet both legal requirements and public expectations, establishing and maintaining a social license for data use, and promoting an approach to AI that is fair, inclusive and trustworthy.

The focus groups will also explore and refine principles for AI transparency and accountability, ensuring that the AI technologies employed are understandable, explainable, and ethically guided. The End User Committee will remain engaged throughout the project to support the iterative design, development, and deployment of the platform, ensuring its ethical integrity is maintained during the development.

In addition, we will conduct targeted interviews to gather diverse stakeholder perspectives on how best to ethically manage and share implementation project data, and how to ensure the responsible application of AI in this domain. Insights from these qualitative investigations will be analysed to develop a set of ImpleMATE Ethics Principles. These principles will be reviewed with the committee to establish consensus and will inform the development of comprehensive data and AI governance frameworks. This ensures transparency in data processing, clarifies the full data journey, and supports accountability for how AI and analytics are used in the project, ultimately reinforcing public trust and supporting ethical innovation in implementation science.

Activity 2. Develop and validate an AI system to automate ontology creation and deliver real-time solutions to implementation challenges

Address research questions: D2K Q6-8, IP2D Q1-4

AI model selection and implementation concept extraction

To enable automated, real-time identification and mapping of implementation science concepts, we will develop and validate an AI-driven system that aligns with the ontology defined in Activity 1. Implementation science experts will first analyze preprocessed data to create a gold-standard dataset by manually annotating concepts including study characteristics, barriers, facilitators, strategies, and outcomes defined in the ontology framework. We will start with 1,000 datasets, each representing an individual document, including published literatures and project data. These datasets will be used to evaluate transformer-based models for their capacity to extract and structure relevant information. Given the diverse and heterogeneous nature of implementation science concepts, ranging from study characteristics and outcomes to more nuanced elements such as barriers, facilitators, and strategies, these datasets will support the exploratory phase of initial model validation and performance comparisons across concept categories.

Our approach will involve two core development pathways. First, we will test and fine-tune a concept extractor such as BERT model for Named Entity Recognition (NER) and Relation Extraction (RE), enabling the system to accurately identify discrete implementation concepts and their relationships. Second, we will collaboratively design and evaluate prompt engineering techniques tailored to pretrained, locally hosted transformer models such as Mistral⁵⁶ and LLaMA.⁵⁷ These models will be explored using a range of strategies, including zero-shot and few-shot learning^58,59 as well as CoT reasoning,⁶⁰ to determine their effectiveness in extracting relevant information from varied, context-rich data sources. Examples of prompts for extracting implementation concepts are shown in Table 1. The use of locally hosted models is to ensure that sensitive health and implementation project data remain secure and confidential. Hosting models on secure infrastructure reduces the risk of data exposure, supports compliance with privacy legislation and ethical standards, and ensures that all processing occurs within controlled environments.⁸¹

As LLM models learn from increasing volumes of annotated data, their performance in identifying and extracting implementation concepts is expected to improve, enabling them to autonomously construct a comprehensive ontology. This continuous learning process will contribute to the dynamic evolution of the ontology and enable the system to deliver real-time, context-specific implementation insights. These insights will form the backbone of the interactive web application developed in Activity 3, allowing users to query and retrieve relevant strategies and evidence-based recommendations efficiently, and supporting the implementation process.

To ensure accuracy and practical relevance, the outputs of each model will be systematically compared against expert-annotated datasets. Through this evaluation process, we will determine whether a single model or an ensemble approach provides the most reliable and generalisable performance. This effort will also serve to validate the overall feasibility and robustness of a computational curation pipeline, demonstrating its potential to support scalable, automated knowledge synthesis in implementation science.

AI performance evaluation metrics, quality assessment and monitoring

To ensure the reliability, accuracy, and continuous improvement of the AI system, we will implement a structured framework for evaluation and quality monitoring. Standard performance metrics, including precision (the proportion of correctly identified entities among those predicted), recall (the proportion of relevant entities successfully identified), and F1-score (the harmonic mean of precision and recall), will be used to quantitatively assess the accuracy and effectiveness of concept extraction. Beyond this surface-level matching, we will evaluate semantic equivalence between AI-generated outputs and expert annotations to account for cases where the same concept is expressed using different wordings. This semantic comparison ensures that meaning is preserved even when the linguistic form differs, which is especially important for interpreting complex implementation science data. The AI system’s adaptability to new or previously unseen concepts will be closely monitored.

A reserved validation subset from the gold-standard dataset will serve as ground truth for evaluating prompt performance. We will systematically test multiple prompt formulations, comparing their outputs against the ground truth using the defined metrics. In addition to performance, prompt quality will be assessed based on relevance and the occurrence of hallucination, which is defined as content not grounded in the data source. An iterative refinement strategy will be employed, using the LLMs to suggest and optimize its own prompts to generate candidate prompts. The final prompt selection will be based on both effectiveness and generalizability across diverse implementation science contexts.

Regular qualitative assessments will be conducted to identify emerging themes, assess the specificity and generalisability of the model, and highlight areas for improvement. Manual reviews of the evolving ontology, led by implementation scientists, will ensure that AI-generated outputs remain consistent with domain expertise and responsive to real-world practice needs. These evaluations will inform targeted adjustments, such as fine-tuning or prompt control, through a feedback mechanism integrated into the ImpleMATE structure and embedded within the broader governance framework, ensuring that insights from performance monitoring and expert review are systematically incorporated into decision-making processes. This iterative approach will drive continuous system improvement, ensuring the AI remains robust, accurate, and contextually relevant in addressing real-world implementation challenges.

Activity 3. Create and test an interactive web application to support implementation decision-making and streamline implementation processes

Address research questions: K2IP Q1-5, IP2D Q1-4

An AI-generated decision aid, and an implementation coaching and data sharing platform

The interactive web application developed in this project will provide two integrated levels of support for implementation practice. The target users include implementation researchers, clinicians, change managers, and other professionals engaged in evidence translation efforts across healthcare settings. The 1^st level will offer an AI-generated decision aid, while the 2^nd level will deliver a more advanced and comprehensive implementation coaching and data sharing platform.

An AI-generated decision aid – This will allow users to input queries related to their implementation context or challenges they are facing. For example, users might ask, “How can I implement X intervention in my setting?” or “Which strategies could help me address a specific barrier?” The system will analyse each query through a text processing pipeline, matched against the ontology created in earlier stages of the project. The ontology will serve as a knowledge base, enabling the retrieval of relevant concepts and their contextual relationships. The semantic meaning of the query will be interpreted by the local LLM, which synthesises the retrieved information and generates a tailored, evidence-based response that aligns with the user’s context. Basic and comprehensive queries will be prepared and regularly submitted to the decision aid for testing to ensure relevance of the retrieved concepts and relationships, as well as accuracy of the AI-generated responses. This supports iterative improvement in the model’s performance and the quality of outputs.

An implementation coaching and data sharing platform – this provides more comprehensive support that functions as a central hub for implementation professionals and clients. It will enable users to securely log in, communicate with implementation support teams, access coaching and training resources, and share data collected during real-world implementation projects. Users will be able to upload a wide range of data, including surveys, meeting records/documentation, process maps, focus group transcripts, cost analyses, and health system outcome metrics. The platform will support implementation scientists to analyse this data through AI-enabled tools such as automatic qualitative coding and strategy mapping, matched against established implementation frameworks like CFIR and ERIC. This functionality will offer deeper insights into the determinants, strategies, and outcomes relevant to their specific projects.

The platform will include a feedback mechanism that allows users to report perceived inaccuracies in AI-generated responses. These observations will inform further refinement of the system through prompt engineering or fine-tuning of the underlying models. The data and insights generated through this platform will be fed back into the Learning Implementation System, expanding and continuously improving the ontology. The web application itself will be initially developed and validated in a secure local environment before being deployed to a secure web server. The LLM will be accessed through a secure and encrypted channel to ensure data privacy and compliance with relevant legal and ethical standards.

To support broader scalability, the platform will be designed with flexible access levels to accommodate the varying needs of different user groups. This architecture ensures that the system can serve as both a lightweight decision aid for individual users and a full-featured collaborative platform for research teams and institutional partners. Examples of user scenarios and the levels of support available to them are described in Table 2. Examples of the web application is shown in Figure 4.

Figure 4. Examples of the web application.

Establish the acceptability of the interactive web application

To evaluate and ensure the acceptability of the interactive web application, we will employ ethnographic methods grounded in human-centred design principles.^82,83 These include in-depth structured interviews, contextual inquiry, shadowing, and direct observation of end users in their natural work environments. This approach allows us to capture not only what users say they need, but also what they do in practice, revealing tacit knowledge, workflow constraints, informal processes, and contextual nuances often missed by traditional research methods.

Ethnographic insights will inform iterative co-design sessions where users actively contribute to shaping the application’s interface, features, and functionality. Through participatory design workshops and usability testing cycles, we will evaluate interface intuitiveness, interaction patterns, and overall user satisfaction, applying core user experience (UX) design principles such as simplicity, responsiveness, consistency, and accessibility.^84–86

These methods will help assess users’ experiences and expectations of existing implementation training and coaching methods, identifying gaps and unmet needs. Findings will directly inform platform design to ensure alignment with users’ day-to-day practices, digital literacy levels, organisational cultures, and time/resource constraints.

Additionally, we will develop a structured change management and transition strategy in collaboration with users to support platform adoption. This will include tailored onboarding resources and continuous feedback loops to refine the platform post-deployment. The overall goal is to ensure the platform is not only functional and acceptable but also seamlessly embedded into routine workflows, enabling long-term engagement and trust.

Activity 4. Design an evaluation framework and roadmap for national integration

Address research questions: SE Q1-2

An evaluation framework for ImpleMATE

To evaluate the effectiveness of ImpleMATE and its integration into health systems, we will develop a comprehensive evaluation framework that captures both system-level impact and end-user outcomes. A key component of this evaluation will be the identification of relevant data interfaces such as API that can reflect patient-level outcomes influenced by implementation strategies generated or supported through ImpleMATE. These outcomes may include changes in care pathways, access to evidence-based interventions, and improvements in clinical decision-making resulting from the platform’s decision aid or coaching platform.

We will collaborate with our multidisciplinary partners, including implementation scientists, clinicians, data scientists, patients, and healthcare managers, to co-develop a set of key performance indicators (KPIs). These KPIs will be designed to assess ImpleMATE’s success from multiple perspectives: implementation effectiveness, patient outcomes, clinical practice, and system-wide performance.

The metrics selected will be guided by the quintuple aim of healthcare improvement: enhancing patient outcomes (e.g., earlier diagnosis, timely treatment, patient-reported satisfaction), improving provider experience (e.g., reduced cognitive burden, support for decision-making), improving health equity, achieving cost-effectiveness, and optimizing resource use (e.g., increased speed and quality of evidence-based intervention uptake, reduced duplication of implementation effort, improved allocation of resources).

Exploring a national roadmap to integrate ImpleMATE

The successful implementation of ImpleMATE hinges on its seamless integration into existing healthcare systems in a way that enhances, rather than disrupts, current clinical and organisational workflows. Achieving this requires close collaboration with state-wide health entities and organisations to ensure the platform meets the operational, technical, and regulatory needs of healthcare providers.

To support this integration, ImpleMATE will be hosted on a secure cloud-based infrastructure designed to meet healthcare-grade data security and privacy standards. Early in the project, and in alignment with the work of the Principle Setting End User Committee, we will establish a System Integration Advisory Group. This group will bring together technical experts, health service executives, clinicians, IT professionals, and implementation scientists to guide the strategic integration of ImpleMATE across diverse health settings.

Using a process map–guided interview approach,^87,88 we will systematically document and analyse current workflows, information systems, and decision-making structures. This will help identify integration points, organisational readiness, potential barriers, and facilitators. Insights from this process will inform the development of tailored integration strategies that are technically feasible and contextually appropriate.

Study status

We have established the ImpleMATE Network, comprising multidisciplinary experts who will form the core of the project’s governance structure. Partnerships have been initiated with collaborators across IT, research technology, cloud infrastructure, and data and AI security to support the development of a secure platform for hosting the web application. Engagements with hospitals and government agencies are also underway to support the co-design process and plan for integration of ImpleMATE into existing healthcare systems. Feasibility assessments and evaluations of locally hosted LLMs for implementation concept identification and extraction have been completed. In parallel, we have explored and shortlisted tools for developing the web application. The first version of the implementation science ontology has also been developed.