Perspectives and guidance for developing artificial intelligence-based applications for healthcare using medical images

Theme 1- The gaps AI can fill in healthcare

Different classes of AI models support different types of healthcare tasks. Understanding the use case is key and can influence the study designs of development and validation. This ensures that appropriate evidence is generated to enable clinical application.

Theme 2 - Development of AI models for healthcare: practical and important things to consider

At the outset of the research process, there is an element of context - the research question. We strongly recommend thinking about a few questions like “What are we endeavoring to predict?”, “How important is the problem at the level of application?” and “How would clinicians use the tool in their workflow?”. We opine that there should be a level of engagement from the beginning, right at the point of study design, with all stakeholders. This includes regulators, clinicians, patients, and the investigators who will conduct the study.

From the outset, we recommend planning for validation and appropriate metrics for evaluating the model. A few relevant examples would be discussions about ground truth, study design for validation and appropriate metrics (for example sensitivity or specificity) and to optimize for a given clinical context. There are some specific challenges in finding the benchmarks against which these AI models can be compared. The most intuitive way is to compare against a panel of clinical experts. However, it is advisable to keep some caveats in mind, while considering this mode of comparison. When clinicians make a diagnosis, they have more contextual or corroborating information about patients, such as their symptoms, investigations, and clinical history. In contrast, AI models are typically provided with information limited to images and therefore a direct comparison of models with clinicians maybe not appropriate. It is recommended that such contextual information is incorporated while developing the AI models. Quality assurance of the input data, manual annotations (ground truth) and the deidentification process are other pertinent factors. Questions like “How do you measure intra- and inter-annotator agreement?”, “What should be considered the gold standard?”, “How much error is tolerable?” must be discussed while designing the study. In addition, we suggest that both automated data collection pipelines and quality assurance checks should be in place before data collection starts. Automation in both quality assurance and annotation is essential as investigators cannot manually check all the data points, for large-scale studies involving several thousands of images or videos. All these considerations clearly emphasize the importance of an interdisciplinary team and constant engagement from the start with all multidisciplinary stakeholders.

Theme 3 - Good practices for validation of the AI models for healthcare: study designs and sample size calculation

It is well-known that the models perform optimistically when evaluated or tested in the population in which they were developed. The demand for external validation along with development studies has increased from the scientific community. A recent review of studies that developed AI models for diagnostic analysis of medical images found that only 6% of 516 studies had conducted external validation.⁸ Another study reported that there were a relatively small number of prospective studies on medical imaging. Randomized clinical trials conducted in the past have been found to be at high risk of bias and deviating from existing reporting standards.^9,10 Hence, it is essential to carefully consider validation methods and appropriate practices to follow for external validation studies of AI models. In the literature there is sometimes confusion between internal and external validations. During internal validation, the model is evaluated for accuracy and robustness. In contrast, external validation measures the model’s generalizability and clinical effectiveness. We reiterate that testing the model on data collected along with the training data but kept separate from the training process is not external validation. True external validation is when the model is tested on data collected at a site (or setting) completely different from the site (or setting) at which data is collected and used for the development of the model. External validation should have a different setting, population, geographical location, or time and should depend on the context in which the model is meant to be used. Ideally, external validation would be conducted with data from more than one setting.

For models that predict a future outcome or event, the ideal study design for validation of AI models would be a cohort study. A cross-sectional study will be appropriate if the model is intended to diagnose a disease. We suggest that interventional study, like a randomized controlled trial should be considered only when clinical effectiveness of a model is being evaluated. Validation on retrospective data is simpler and quicker but needs to be done keeping in mind that investigators may not have had control over how the data was collected; these validations would have a high risk of bias. The aim of prediction model development studies is to estimate the coefficients robustly, but in external validation studies the focus shifts to estimation of model performance. Since the two steps focus on two different but important aspects, validation studies should not be an afterthought. Critical to good validation study design is sample size estimation. We recommend investigators to follow recently developed guidelines and best practices for sample size estimation for prediction model validation.^11,12

While discrimination is the ability of the model to distinguish between the groups, calibration, on the other hand, measures the alignment between predicted probabilities and the actual frequency of events, ensuring that the model’s confidence scores are reliable. The model performance should be measured in terms of discrimination, calibration, and clinical utility on external validation data sets. Discrimination should be measured area under the curve of receiver operating curve (AUROC) for classification models, Calibration slope and intercept should be reported to assess model bias. Finally, to assess clinical utility, a decision curve analysis is recommended.¹³

Theme 4 - Points to consider when publishing AI models

This theme identifies key considerations while writing a manuscript for peer review and publication. A recent systematic review of the diagnostic accuracy of deep learning and medical imaging found that there is high heterogeneity between studies. This is due to varying methods, terminology, and outcome measures, indicating a need for developing reporting and methodological guidelines for significant issues in this field. Reporting guidelines are not new in the field of medical and epidemiological research (EQUATOR network) The domain of AI in medical imaging is multidisciplinary and mandates that reporting should be in a way that is easily comprehensible by the end users of these models. While a broad consensus towards such reporting is evolving, there are existing publishing guidelines and checklists such as the CLAIM¹⁴ and TRIPOD.¹⁵ The publication of the model is an important intermediary step between the development and validation of the model. Authors should consider sharing all essential information required for reproducibility and transparent implementation of the model by other researchers. Reproducibility is greatly facilitated by releasing the model and implementation code. We recognize that this is not always possible in healthcare AI, especially for stakeholders in industry and where data governance restrictions forbid release of data used to build a model. In such situations, wherever possible, release of a web-based implementation of a model, which would keep intellectual property secure would be important. We also recognize that grand challenges, open competitions with data and code, play an important role in model benchmarking. Although they typically capture only real-world scenarios in a limited way they can encourage “data engineering gaming” to incrementally beat prior work. Overall, researchers need to take the responsibility to report their work in a scientifically rigorous way with clear descriptions of the data set, eligibility criteria, the context in which the model is to be used, appropriate metrics, and an implementation plan to facilitate the next steps in the translation pipeline. Peer review needs to become more rigorous to ensure that AI model reporting standards are elevated.

Theme 5 - Translation towards products

The final step in translating any AI model is to produce a prototype that can be used by healthcare workers. Academic publication plays a crucial role in technology translation and how authors present the information is of paramount importance. Authors should maintain a balance between sharing information and retaining the details that help commercialization of their technology. We emphasize that authors should define the technology performance level clearly and report limitations so that the end-user can utilize it effectively. A major obstacle to adoption of AI is because benefits of AI are not clear to clinicians.¹⁶ It would be advisable that instead of simply handing the stakeholders any new AI-based technology, they should be trained on use of model in real life situations which will also allow them to appreciate the improvement the technology brings to their daily workflow. There is an urgent need for low-cost (or zero extra cost) AI solutions requiring minimal clinical expertise for healthcare workers. In summary we recommend four stages that any AI prototype should undergo before being accepted for clinical use: peer-reviewed publication of the model, external validation, regulatory approval, and recommendation by professional societies. A clear plan of the deployment scenario while developing the product will facilitate translation.

Theme 6 – Challenges and potential solutions from technical perspective

To date, the black-box nature of current AI models has led to slow adoption of this technology in medicine where the cost of a mistake is high. Explainable artificial intelligence (also called interpretable AI) has been suggested to generate trust among the health-care professionals, to bring transparency into AI decision-making, and mitigate biases. Currently, there are several methods to test the explainability of AI models. A heat map (also known as a saliency map) is a popular method that uses activations of convolutional layers to demonstrate the extent to which each region of the image contributed to the model’s decision. They are illustrative and are easy to understand but do not always correspond to human intuition for important in decision-making. Class activation maps (CAM)¹⁷ and its extension Grad-CAM¹⁸ are among the popular methods that are used to generate these explanations. Besides heat maps, locally interpretable model-agnostic explanations (LIME)¹⁹ and Shapley values (SHAP)²⁰ seek to understand decisions at the individual level by altering the input example and identifying the alterations that contributed to the decision. In the case of image analysis, this is done by occluding parts of the image. While the above approaches aid in understanding the clinical perspective, other approaches such as feature visualization are used by machine learning engineers. Feature visualization involves producing synthetic inputs that activate specific parts of a machine learning model strongly. Each model decision can then be described as a combination of a series of features that were detected in the input. Nevertheless, all these methods are just approximations and do not include explanations in terms of medical findings. This in turn makes the interpretation of AI models rather subjective. This area of research is still in its nascent stage and more work is needed to develop truly explainable AI.²¹ In the absence of suitable explainability methods, we advocate for rigorous internal and external validation of AI models as a more direct means of achieving the goals.

The ability of the model to give accurate predictions on an external data set collected separately from the training data set is called generalizability. Despite a large amount of published work on AI applications in medicine, only a relatively small number of these models are implemented in clinical practice primarily due to the lack of generalizability of models, though this number is also growing.^22,23 Site to site customization in image acquisition, different standards of implementation in clinical practice, differences in patient demographics across centers, genotypic and phenotypic characteristics of patients, and tools and methods used to process and develop medical data are just some of the factors that may affect generalizability. Another significant factor that affects generalizability is training the model on biased data. Some of these factors can be adjusted using recent methodological advances such as domain adaptation techniques. In domain adaptation, the goal is to adapt models to target data sets based on labeled samples from the source environment as well as a limited set of unlabeled samples from the target environment.

Probably the best solution is to collect training data from different centers and incorporate all the real-world variations in it. However, collecting a huge medical data set from multiple sites is time-consuming, expensive, and poses data sharing challenges. Federated learning,²⁴ an emerging methodological advancement, may aid researchers in overcoming the data sharing challenge. This framework helps researchers to train AI models while retaining data at individual sites. To summarize, different models of the same architecture are trained separately at each site using data from the site, and then these partial models are combined to create a global model.