Evaluating artificial intelligence for accurate detection of hand and wrist fractures: a systematic review and meta-analysis

2.1 PICO framework

The PICO framework was used to investigate the effectiveness of Artificial Intelligence in accurate detection of hand and wrist fractures compared to traditional diagnostic methods such as clinicians’ manual reading of radiographs. The systematic review involved all age groups with suspected hand and wrist fractures. The intervention aimed to studies that used artificial intelligence, including machine learning and deep learning algorithms, for detecting hand and wrist fractures. These techniques were compared with traditional diagnostic methods, such as manual reading of radiographs by clinicians. The outcome sort were diagnostic accuracy metrics including sensitivity, specificity, positive predictive value, and negative predictive value. The target studies had prospective or retrospective cohort studies, randomized controlled trials, and observational study designs.

2.2 Search strategy

2.2.1 Databases searched

The search was conducted across multiple electronic databases: PubMed, Google Scholar, Wiley Online Library, and Science Direct.

2.2.2 Search terms and keywords used

The search terms and keywords used a combination of both standard terms and general keywords, which were refined into proper MeSH-based queries. The search terms and keywords utilized a combination of Boolean operators (AND, OR, AND) to effectively combine key terms and retrieve the desired literature from the searched databases.

The search criteria for Google Scholar, Wiley Online Library, and Science Direct involved keywords indexing system to capture results across all platforms.

2.2.3 Study selection process

The selection process started with a thorough search of the electronic databases (Google Scholar, Wiley Online Library, and Science Direct, and PubMed), followed by uploading the results to Rayyan software to identify and remove duplicate entries using two distinct phases carried out independently by three researchers. In phase one, the title and the abstract of each study was reviewed to determine their eligibility criteria, while excluding those that did not meet the criteria.

2.3 Inclusion and exclusion process

The inclusion criteria encompassed prospective or retrospective cohort studies, randomized controlled trials, and observational studies published in English, with no time frame limitations, involving patients of all ages with suspected hand and wrist fractures, were included in the systematic review. The reviewed studies focused on relevant outcomes using artificial intelligence (AI), including machine learning and deep learning algorithms, to detect hand and wrist fractures and compare their performance with traditional diagnostic methods, such as the manual interpretation of radiographs by clinicians.

Studies involving animals or cadavers, those not published in English, those not using artificial intelligence for hand and wrist fracture detection, studies lacking a comparator group or comparison with conventional diagnosis methods, and studies lacking sufficient data to build a contingency table were among the excluded criteria. Omitted were reviews, case studies, letters, editorials, and conference abstracts. Studies with a high risk of bias or low quality based on the assessment of study design, sample size, data collection and analysis and lacking relevant factors were excluded.

2.4 Data extraction

A standardized form was created to summarize the data relevant factors to the research questions. The variables in the extraction form included general information about the study, author, year, study design, sample size, population characteristics, type of Al algorithm used, imaging modality, sensitivity, specificity, positive predictive value, negative predictive value, area under the curve (AUC), type of fracture detected, comparison group, data preprocessing methods, handling of imbalanced data, external validation, risk of bias, and funding sources.

2.5 Quality assessment

The selected studies were screened for duplication which were dropped from the systematic review. The risk of bias was assessed using the Methodological Index for Non-Randomized Studies (MINORS) for observational and non-randomized designs, as well as ROBINS-I for non-randomized comparative studies. Studies identified as having a serious risk of bias were excluded from the review.

2.6 Data synthesis

2.6.1 Treatment of missing data

Missing data was handled by checking the completeness of reported outcomes in the included studies. The final included articles were identified based on the methodological index for non-randomized studies – MINORS – a tool used to screen prospective, retrospective or case-control studies to be included in the systematic review. The studies assessed for bias were 23 in total, out of which only one study had used a prospective study design, while the rest used a retrospective study design.

2.6.2 Assessment of Bias

Two reviewers independently used the methodological index for nonrandomized studies (MINORS) to assess the risk of bias in retrospective and prospective non-randomized studies. This is a validated 12-item tool designed to assess the quality of non-randomized surgical studies. Each included article was assessed for risk of bias using the MINORS tool by the two reviewers, with disagreements resolved through discussion or consultation with a third author.

3.1 Study selection

The PRISMA 2020 flow diagram was generated in R¹⁵ to summarize study selection. As shown in Figure 1, we identified 526 records; after removing duplicates, 447 remained. We assessed 22 full-text articles and included 18 studies in the review.

Figure 1. PRISMA flow diagram showing the process of study selection.

Abbreviations: PRISMA = Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

3.2 Characteristics of included studies

A total of twenty studies were included in the study where were nine studies were retrospective cohort studies,^16–24 two were retrospective diagnostic^25,26; two retrospective experimental study^27,28; one retrospective validation study²⁹; four retrospective studies^30–33; and two prospective diagnostic studies,^34,35 Table 1. A retrospective cohort study identifies risk factors and associations, with follow-up at risk groups over time as a key feature. A retrospective diagnostic study evaluates the test accuracy and compares diagnostic results to a gold standard. A retrospective experimental study assesses past interventions using historical intervention data. A retrospective validation study tests models/methods as it validates previous findings on new past data. A prospective diagnostic study assesses the effectiveness of a diagnostic test in detecting conditions.

3.3 Quality assessment of included studies

The MINORS quality appraisal results for each study appear in ( Table 2). Individual item scores (0–2) and total scores are reported for each study; items 9–12 apply only to comparative designs.

3.4 Pooled analysis of sensitivity and specificity

3.4.1 Forest Plot of Sensitivity and Specificity

The forest plots display sensitivity and specificity for individual studies along with the pooled estimates. ( Figure 2) shows sensitivity and ( Figure 3) shows specificity. This showed the variations among studies and how each contributed to the overall results. The studies selected for final inclusion were 18. However, you will notice that 22 entries were included in the forest plot. This is because some studies reported investigations using more than two algorithms, and to avoid overlapping, it was necessary to report each algorithm individually.

Figure 2. Forest plot of sensitivity for AI models in detecting hand and wrist fractures.

Figure 3. Forest plot of specificity for AI models in detecting hand and wrist fractures.

Figure 4 above shows the forest plot for sensitivity estimates. Logit sensitivity was estimated instead of logit specificity because it was the primary target for the systematic review—detection of hand and wrist fractures using AI. The true positive rate (sensitivity) was important because the AI models were designed for detecting hand and wrist fractures, as missing a fracture could have detrimental side effects due to delayed interventions.

Figure 4. Forest plot of specificity for AI models in detecting hand and wrist fractures.

Abbreviations: CI = Confidence Interval; AI = Artificial Intelligence.

Most of the AI models, as shown in Figure 4, cluster around a logit sensitivity of 3–5, which is a positive sign that most models had strong diagnostic performance. The logit sensitivity shows that three AI models—3D ultrasound-Philips, CNN-ResNet34, and Deep CNN Gleamer—had logit sensitivity values closer to 12, suggesting exceptional sensitivity. Models such as VGG-16 and Radius Segmentation U-Net & RetinaNet showed competitive sensitivity with estimates of around 4–5. These variations illustrate the different capabilities of deep learning techniques in detecting hand and wrist fractures. The blue spots in Figure 4 also display lines (confidence intervals), highlighting variations in performance across the different datasets.

High sensitivity scores across most models indicate that the AI models used for detecting hand and wrist fractures were effective at identifying fractures. This is very important in a clinical setup where missing a fracture (false negatives) could have serious consequences, as patients might miss urgent intervention.

3.4.2 Fixed-effects meta-analysis for sensitivity and specificity

The sensitivity and specificity values from multiple studies were combined using a fixed-effects model. The assumption was that all 18 studies were estimating the same true effect, with any differences arising from chance. Consequently, the pooled sensitivity and specificity were estimated as weighted averages, with studies having lower variance receiving more weight.

3.4.3 Logit transformation

The logit transformation function normalizes the data, increasing the reliability of its calculation. The data was normalized and then transformed back to the probability scale for easier interpretation. Since sensitivity and specificity values typically range between 0 and 1, their transformation to the logit scale was necessary.

3.4.4 Heterogeneity analysis and random effect

The Cochran’s Q test and I² statistic were used to measure the variability among the 18 studies. Higher values indicated large variations, suggesting that the 18 studies were not measuring the same thing. To further investigate this high variation, a random-effects model was performed, as it relies on the assumption that each individual study analyzed in the present systematic review had its own true effect rather than relying on a single common effect.

3.5 Assessment of publication bias

Figure 5 above shows the funnel plot that was plotted with studies, showing the relationship between study precisions (standard errors) and effect sizes. The identifiers were annotated with numbers, along with their specific labels and the AI models used. The shape of the plot is asymmetrical, meaning that there was publication bias, specifically a small study effect. In the absence of publication bias, the study points would have been evenly scattered around the red vertical line. The vertical red line represents the overall mean log (DOR)—the log of the Diagnostic Odds Ratios, which measures the effectiveness of a diagnostic test and is calculated as the ratio of the odds of a true positive to the odds of a false positive. A closer observation of Figure 5 shows that 16 AI models are on the left side of the vertical red line, and 7 on the right. The studies on the left side suggest that the AI models reviewed had lower DOR, which could be translated as potentially lower test accuracy or effect size. The seven AI models on the right side suggest a higher DOR, which could be translated as potentially higher test accuracy or effect size.

Figure 5. Funnel plot assessing publication bias in included studies.

Abbreviations: DOR = Diagnostic Odds Ratio; SE = Standard Error.

The imbalance – asymmetrical funnel plot – suggests potential publication bias, which resulted from the studies using different AI models and algorithms with variations in sample sizes. The AI models on the left side were clustered closer together, indicating that their results were more consistent with each other, while the clustering of the AI models on the right side is spread out, indicating more variability in their results and uncertainty. The precision was approximated by the standard error as reflected in the y-axis. The studies with higher precision (smaller error) appeared at the top – closer to the 0 value, while the studies with lower precision (larger errors) appeared at the bottom of the funnel plot. The studies,^{16,25,27,31–33} had standard errors below 0.025, which are closer to 0. This suggests their results are highly precise, probably due to their large sample sizes – an indication that they carried more weight in the overall conclusion. The study by Zhang et al.,³⁵ had a standard error placed at the far bottom of the funnel plot, indicating that its AI model had a higher uncertainty in its estimate, thus less reliable (lower precision). Its placement on the right side of the funnel is indicative of a higher diagnostic odds ratio – better diagnostic performance of the AI model. Nevertheless, the model shows a large standard error, which makes it less trustworthy due to its lower precision value, and thus not as reliable as studies with lower SE, clustered near the top on the right side.

The Egger’s test shown in Table 4 indicates that the precision value was statistically significant (p < 0.001), confirming the presence of publication bias in the published AI models. The R-squared score indicated that 83.4% of the variability in the standardized effect (log_DOR/SE) explained the precision (inverse of SE) in the final model, suggesting that the model accounted for most of the variance in detecting hand and wrist fractures using AI models. Thus, precision was an important predictor of the standardized effect. The adjusted R-squared variance of 82.6% suggested that the variability in the standardized effect was explained by precision.

4.1 Summary of findings

A substantial variability was observed across the studies in terms of sensitivity and specificity. The sensitivity (0.796) and specificity (0.903) in the fixed-effects pooled estimates indicate that the fixed-effects model had higher specificity compared to sensitivity. However, the random-effects model showed higher values for both sensitivity (0.910) and specificity (0.912), suggesting that the random-effects model demonstrated better diagnostic performance for AI in detecting hand and wrist fractures.

The heterogeneity scores for sensitivity (99.09%) and specificity (96.43%) were very high, as shown in Table 3. This indicates substantial inconsistencies across the studies, likely due to variations in the devices and algorithms used across the 18 reviewed studies. As a result, caution is necessary when generalizing these findings across different clinical settings.

In terms of sensitivity of the devices and their respective algorithms, the 3D Ultrasound-Philips IU22 machine,³⁵ CNN-ResNet34 & 3DU,³⁴ and Deep CNN-Gleamer²² reported the highest scores, as shown in Figure 2. The mean sensitivity was approximately 90%.

Studies by Hendrix et al.,²⁷ - (YOLOv5 & Inception-v3), Jacques et al.,¹⁹ - (Deep CNN-Gleamer), Li et al.,²¹- (YOLOv3 & MobileNetV3), Knight et al.,³⁴- (CNN-ResNet34 & 2DU), Cohen et al.,³⁰- (Deep CNN-Gleamer), Min et al.,²⁶- (YOLOv5), Blüthgen et al.,¹⁷- (ViDi Suite Version), Zech et al.,³³- (Region-Based CNN), and Mert et al.,²²- (ChatGPT4) all had sensitivity values below 90%. This indicates that the algorithms used in these studies were less effective in identifying positive cases (true positives) compared to studies with sensitivity values above 90%.

The lower sensitivity in these studies could be attributed to a higher percentage of missed true positive cases (false negatives) when detecting hand and wrist fractures. Therefore, studies with specificity values below 90% signal that the AI models used may not be fully reliable for diagnosing hand and wrist fractures, increasing the risk of missed diagnoses.

The specificity identified algorithms that can distinguish individuals without hand and wrist fractures (true negatives) from those incorrectly identified as having fractures (false positives). The mean specificity was approximately 90%. Therefore, studies with higher specificities (greater than 90%) demonstrated that the algorithms correctly identified individuals without hand or wrist fractures, indicating a minimized risk of false positives, and vice versa.

Studies by Min et al.²⁶ – YOLOv5, Lee et al.²⁹ – CNN-RetinaNet, DeepLabv3 & NasNet, Üreten et al.²⁴ – ResNet-50, Üreten et al.²⁴ – GoogleNet, Raisuddin et al.³² – Deep Wrist Pipeline, Kim and MacKinnon³¹ – Deep CNN, Anttila et al.¹⁶ – Segmentation-Based U-Net Architecture, Zhang et al.³⁵ – 3D Ultrasound (Philips IU22 machine), Zech et al.³³ – Region-Based CNN, and Jacques et al.¹⁹ all reported specificity values below 90%. Nevertheless, studies with higher specificity scores indicated that their AI models performed better in avoiding false alarms.

4.2 Strengths and limitations

Like any other study, the present systematic review had its own strengths and limitations. Its strengths included the fact that most of the studies reported high scores for both sensitivity (14 out of 23 AI models) and specificity (12 out of 23 AI models), with values ≥90%. Higher sensitivity scores indicated fewer missed fractures, suggesting that radiologists could potentially rely on these AI models to detect hand and wrist fractures, and vice versa.

The systematic review indicated that deep learning models, particularly those based on CNNs, dominated the performance of the AI models reviewed. This pattern was reflected in the forest plot of both sensitivity and specificity, where the top quarter was largely occupied by CNN-based AI models. However, this review does not advocate that CNN models are inherently superior to other models; rather, it highlights opportunities for further improvements and modifications to develop better algorithms or models. Enhancements could include training the models on larger sample sizes or fine-tuning hyperparameters to improve predictive performance.

One of the limitations of this review was that some studies had smaller sample sizes compared to others. However, logit transformation was applied to convert the proportions to an unbounded scale in preparation for statistical modelling and meta-analysis, which helped stabilize variance resulting from different sample sizes. The logit function was then back transformed to the inverse logit function for easier interpretation of sensitivity and specificity scores. Additionally, sample weighting was performed to ensure that the final estimates of pooled sensitivity and specificity were reliable. Future AI studies can enhance their models by training on larger datasets and continuously reviewing and improving their performance.

Another limitation arose from the interpretation of the confidence intervals presented in the forest plots. The results indicated that 9 out of the 18 studies had sensitivity values below 90%, which was concerning as it suggested a higher risk of missing hand and wrist fractures.

Lastly, the systematic review aimed to evaluate sensitivity and specificity and ensure that the meta-analysis provided robust evidence for the clinical superiority of one AI model over other comparative AI models. The approach involved assessing bias and robustness in terms of publication bias and sensitivity analysis. Most studies failed to report the AUC, NPV, PPV, and even confidence intervals. As a result, the study relied on sensitivity, specificity, and sample sizes to determine publication bias.