Bridging Artificial and Human Intelligence: Comparative Performance of AI Models and Neurologists in Stroke Recognition- A Cross-Sectional Study

Introduction

Stroke is a serious cerebrovascular disease that results in a major global health burden due to its potential to cause severe brain damage, disability, and even death. It occurs due to an abrupt disruption of blood flow to the brain, and it is classified into two main types: ischemic strokes that result from blockages in blood vessels and hemorrhagic strokes, which indicate rupture of vessels.¹

Stroke is the second leading cause of death worldwide with estimation of 5.5 million deaths annually, and a significant contribution to global disability. Therefore, Prompt recognition and effective treatment are paramount in reducing stroke burden and long-term complications. According to the World Stroke Organization, acute stroke accounts for approximately 10% of all deaths, with an estimated13.7 million newly reported cases each year.¹

The development of large language models (LLMs) has led to a public return on artificial intelligence (AI), with major applications in various fields, including healthcare. Models such as OpenAI’s generative pretrained transformer 4 (GPT-4) can understand and generate human-like text and perform multiple tasks, such as translation, summarization, and coding. Furthermore, considering their practicality, automation advancement appears feasible in various fields when employing LLMs. Previously, chatbots were primarily used for customer service and uncomplicated inquiries. However, with the innovation of LLMs as the underlying technology, they now have the potential to perform challenging tasks because they can comprehend content and memorize previous conversations.^2,3 LLMs have exhibited potential in several aspects of healthcare, such as clinical reasoning and decision support. A previous study focused on physician behavior toward AI in everyday practice, illustrating that 78% of physicians aimed at utilizing AI frequently or always for purposes of diagnosis, exam interpretation, and management if AI solutions were accurate, fast, and available.^4,7 However, although AI has been tested in multiple healthcare tasks such as image interpretation and hospital admission predictions, its role in emergency conditions requires further exploration.^3–6

In this study, we aimed to explore the diagnostic accuracy of AI compared with expert human physicians (neurologists) in recognizing neurological emergencies, specifically stroke.

Methods

Results

Our analysis is based on questions posted online by patients with suspected stroke symptoms collected from ‘AskDocs’ subreddit. The Likert scale is frequently used in the medical education and medical education research fields for response assessment. It was developed by Rensis Likert in 1932 as a psychometric response format used to measure attitudes, perceptions, or opinions through ordered response categories (typically 5–7 points, ranging from strong disagreement to strong agreement). Individual Likert items generate ordinal data. However, when multiple items are summed to form a composite score, the resulting scale is often treated as interval data for statistical analysis.^18,19 Here we used ease of understanding, adequacy, and overall satisfaction as measurement parameters of the Likert scale to assess responses.

Discussion

We conducted this cross-sectional study to evaluate the performance of AI LLMs models comparable with specialist neurologists in recognizing serious neurological emergencies, such as strokes. Furthermore, stroke is responsible for a huge global burden, and between 1990 and 2021, the estimated global cost of stroke was over US$890 billion (0.66% of the global GDP). With the reported increased frequency (70.0% increase in incident strokes, 44.0% deaths from stroke, 86.0% prevalent strokes, and 32% DALYs), it ranks as the second leading cause of death in the world. Countries of lower income and lower middle-income (LMICs) were markedly affected by the global stroke burden (87.0% of deaths and 89.0% of DALYs), according to the statistics of the World Stroke Organization. Additionally, Stroke considerably contributes to metabolic risks, estimated at 69.0% of all strokes, along with 37.0% environmental risks and 35.0% behavioral risks.^8,9

One of the significant findings of our analysis is the notable difference in triage recommendations between neurologists and AI models. Neurologists were noted to be more likely than AI models to advise patients to visit the Emergency Department (ED) for suspected stroke cases. Further, AI responses often labeled cases as indeterminate rather than providing a clear recommendation favoring caution. This tendency resulted in lower overall referral rates to the ED. However, within the context of stroke, prompt diagnosis and immediate intervention are critical; therefore, this difference is not only a matter of performance metrics or stylistic variation but also raises urgent patient safety concerns. Moreover, Stroke is considered a time sensitive condition and missing or delaying assessment carries risks far outweighing those associated with over referral. As a result, AI models with conservative or less decisive triage recommendations could potentially lead to under-referral to the ED in emergent situations where early evaluation is paramount, and delay might compromise patient outcomes.

A notable observation from this study was the disconnect between the readability of AI-generated responses and their clinical adequacy. Although AI answers were persistently rated as easy to understand, pointing toward the fact that they presented information in a clear, effective, and accessible manner for readers, this clarity did not explicitly indicate higher scientific adequacy or overall satisfaction in comparison to neurologist responses.

In presentations concerning stroke, clarity alone is insufficient. Patients require guidance that reflects their clinical urgency, risk stratification, and awareness of acute time sensitivity. This suggests that while AI can communicate in a patient-friendly manner, it does not accurately provide depth or essential clinical framing for emergent neurological situations. Therefore, good readability should not be conflated with safe or adequate clinical guidance. A systematic review illustrated that (AI) and machine learning (ML) systems are advancing healthcare practices in various domains, including error reporting, incident management, medication prescription, and fall prevention,^10,14 however, despite that, AI demonstrates safety concerns on this analysis. A narrative review has reached a conclusion similar to our results that there are biases ingrained in AI algorithms, and limited transparency in decision making may compromise patient safety and data privacy if AI is implemented in clinical settings.^11,12

Furthermore, the assessment of agreement results between AI models and neurologists may appear contradictory. A high percentage of agreement indicates that the AI and neurologists concur with their recommendations. Nevertheless, the kappa values used to evaluate inter rater agreement have demonstrated low or even negative scores, raising concerns. It is critical to understand that these kappa values do not indicate poor agreement. However, they displayed the structure of the data, where most responses were clustered in dominant categories, leading to unbalanced data. In such circumstances, kappa tends to perform insufficiently, often underestimating the true agreement level. Therefore, this might lead to statistical limitations because low kappa scores are a function of the data structure rather than a true discordance between AI and neurologists.

Importantly, the findings of this study emphasize that AI tools should be positioned as adjuncts rather than replacements for clinician decision making in acute neurological emergencies. As LLM models can assist patients in understanding their symptoms and may promote help-seeking behavior in certain situations, they do not consistently exhibit the decisiveness or risk prioritization required when minutes are crucial, as in cases of suspected stroke.

Clinical judgment incorporates experience, pattern recognition, and an understanding of the consequences and capabilities of current AI systems that are inadequate to accurately demonstrate them. As a result, framing AI as supportive informational aid rather than a triage decision authority is paramount. This approach not only aligns with patient safety aspects but also provides accurate expectations regarding the contributions of these technologies in emergency care settings.^13,15–17

Conclusion

To summarize, although AI models such as ChatGPT and Gemini have the potential to recognize stroke, their cautious strategy and failure to provide clear triage guidance raises patient safety issues. Clarity in communication is necessary; however, it should be augmented by clinical adequacy to facilitate effective patient care. Furthermore, integrating AI into clinical workflows necessitates careful consideration of its role as supportive aid, helping to enhance but not replace the critical judgment of healthcare professionals.

Ethics statement

Ethical approval was not required for this study as it utilized publicly available online data from previously published posts and did not involve direct interaction with human participants, access to private records, or identifiable personal information. The study was conducted in accordance with applicable research and ethical standards.

Informed consent

Informed consent was not required for this study because the analysis was conducted using publicly available published online data with no direct participant involvement or identifiable personal information.