[version 1; peer review: 2 approved with reservations, 1 not approved]
No competing interests were disclosed.
Assessment plays a central role in medical education by evaluating learners’ knowledge, skills, and professional competencies. While multiple-choice questions (MCQs) are widely used due to their efficiency and broad content coverage, they primarily assess recall and recognition, limiting their ability to measure higher-order reasoning. Short-answer questions (SAQs), in contrast, promote deeper cognitive processing and provide better discrimination between levels of student performance. However, SAQs are resource-intensive to grade and susceptible to scorer inconsistency and rater bias, highlighting a need for more efficient and reliable assessment solutions.
Artificial Intelligence (AI) has emerged as a transformative tool in medical education, enhancing learning, supporting adaptive instruction, and automating assessment processes. AI-driven systems using machine learning and natural language processing have been increasingly applied to automated scoring of SAQs. These systems offer potential benefits, including reduced grading burden, greater scoring consistency, and timely feedback to learners. Despite promising developments, concerns persist regarding algorithmic transparency, data privacy, and the reliability and validity of automated scoring compared with human graders. Existing studies report mixed results, underscoring the need for a comprehensive examination of current approaches.
This scoping review aims to systematically map the literature on AI-based models used for automated scoring of SAQs in medical education. Specifically, it seeks to identify the types of AI models employed, evaluate their accuracy and reliability relative to human graders, describe reported advantages and challenges, and assess fairness and feasibility within educational settings. Following the Joanna Briggs Institute methodology and the Population–Concept–Context framework, the review will include empirical studies published since 2015 involving medical students and AI-driven SAQ scoring. Findings will provide an evidence-based overview of current practices, highlight gaps in the literature, and inform future research and implementation strategies for AI-assisted assessment in medical education.
Assessment is a central component of medical education, serving to evaluate learners’ knowledge, skills, and attitudes through both formative and summative assessment methods (
Multiple-choice questions (MCQs) and SAQs are among the commonly used written assessment formats in medical education (
SAQs, by contrast, require students to generate concise written responses, thereby encouraging active retrieval, deeper cognitive processing, and higher-order reasoning (
Artificial Intelligence (AI) is emerging as a powerful and transformative tool in medical education. AI technologies are increasingly integrated into educational systems to enhance students’ learning experiences, prepare them for an AI-driven healthcare environment, personalize learning, and improve assessment processes (
The incorporation of AI has enhanced students’ learning through AI-assisted assessment platforms that enable learners to practice applying their knowledge while receiving immediate feedback (
Automated assessment systems use Machine Learning (ML) and Natural Language Processing (NLP) techniques to automated scoring of SAQs and essays (
Automated scoring of short-answer questions (ASAQ) was first introduced in the 1960s and has since undergone substantial development, incorporating increasingly sophisticated statistical, ML, and NLP approaches to improve scoring accuracy and reliability. Recently, the application of ASAQ has gained considerable attention in medical education, where grading large volumes of SAQs is both time-consuming and susceptible to rater bias (
Although the use of AI models for automated scoring of short-answer questions has gained increasing attention in medical education research, the breadth and depth of the existing literature remain unclear. A notable gap exists in understanding the range and types of AI models employed by medical educators to evaluate short-answer responses in student assessments. Consequently, a scoping review is warranted to systematically explore the extent and nature of current research and to map the available evidence on this topic.
Beyond identifying the AI models used, the review will examine the reported validity, reliability, and feasibility of these systems in comparison with traditional human grading methods. Such a review will help clarify the current landscape of AI-driven automated scoring, highlight research trends and gaps, and provide a comprehensive overview of how these technologies are being implemented in medical education assessment.
The aim of the review is to systematically map the existing literature on automated scoring of SAQs in medical education, with a focus on utilized tools, accuracy, reliability, and fairness. The review will address the following research questions:
What AI-based models have been used to automated scoring of SAQs in medical education? How accurately does automated scoring of SAQs reflect the performance of human graders? What advantages and challenges of using AI for automated scoring of SAQs have been reported in medical education? Are these models more effective than human graders in terms of reliability, accuracy, and fairness?
This scoping review will be conducted in accordance with the methodology outlined in the Joanna Briggs Institute (JBI) Manual for Evidence Synthesis, with a focus on AI models used for automated scoring of SAQs in medical school assessments. The Population, Concept, and Context (PCC) framework will guide the search strategy, eligibility criteria, and data extraction processes.
Population: Medical students Concept: Automated scoring of short-answer questions using AI-based models Context: Medical education and medical school settings
Four electronic databases will be searched:
A detailed search strategy will be developed in consultation with a medical librarian. Keywords such as
Covidence systematic review software tool will be used to import references, conduct title and abstract screening, review full texts, and manage data extraction. Two independent reviewers will screen all retrieved studies based on the predefined eligibility criteria. Discrepancies will be resolved by a third reviewer.
Screening will occur in two stages:
Title and abstract screening conducted independently by two reviewers. Full-text review of studies deemed potentially relevant.
A secondary search will involve screening the reference lists of included studies to identify additional relevant literature.
Topic: Studies must focus on automated scoring of short-answer questions. Methodology: Original empirical research, including quantitative, qualitative, and mixed-methods studies. Study Population: Medical students completing assessments containing SAQs. Assessment Type: Any assessment format using SAQs (e.g., low-stakes, high-stakes, summative examinations). Publication Date: Studies published from 2015 onward. Language: English-language publications only. Computational Approach: Studies using AI-based autoscoring models (machine learning, large language models, deep learning). Setting: Medical education or medical school settings.
Studies not focusing on SAQs (e.g., MCQs, essays). Non-English publications. Studies not conducted within a medical education context or not involving medical students. Studies using non-AI-based approaches to automated scoring. Systematic reviews, scoping reviews, and grey literature. Studies published before 2015.
A standardized data extraction form will be used to collect key information from all included studies. Two reviewers will independently extract data, with disagreements resolved by a third reviewer. Extracted data will include:
Study methodology Type of AI-based model used Participant characteristics Assessment type Outcomes relating to the performance of the autoscoring system
Extracted data will be charted and summarized in tables and figures. Tables will outline study characteristics (e.g., author, publication year, AI model used), while figures will illustrate the frequency of AI models applied and the accuracy of their automated scoring.
A thematic analysis approach will be used to identify recurring patterns and themes within the included studies. Key concepts related to the types of AI models used, their scoring accuracy, associated challenges and advantages, and their reported validity and reliability will be systematically coded and synthesized into overarching themes.
This analytical approach ensures alignment with the research questions and provides a comprehensive overview of the literature. The review will evaluate whether automated scoring of SAQs should be integrated more widely into medical education and identify existing research gaps and future directions.
Results will be summarized narratively and supported with tables and figures where appropriate. Both the protocol and final review will follow the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines.
A comprehensive mapping of current literature on the use of AI models for automated scoring of SAQs in medical education. Identification of the benefits and challenges associated with integrating automated scoring into medical assessments. Evaluation of the validity, reliability, and feasibility of automated scoring systems compared with human graders. Recommendations for future research on the validity, reliability, and implementation of AI-based autoscoring systems in medical education.
The findings from this scoping review will be presented at academic conferences and submitted for publication in peer-reviewed journals.
No ethical concerns as this review will include the use pf published peer-reviewed articles and grey literature. Therefore, ethical approval and consent is not required.
No data is associated with this article.
Repository: PRISMA-P checklist for “Protocol for Conducting a Scoping Review on The Use of AI in Automated Scoring of Short-Answer Questions in Medical Education”.
DOI: 10.6084/m9.figshare.30815456 (
Data are available under the terms of the
This protocol addresses a timely and highly relevant topic in health professions education. The increasing adoption of AI-based assessment systems, particularly Large Language Models (LLMs), makes a scoping review of automated scoring of short-answer questions (SAQs) both necessary and potentially impactful. The manuscript demonstrates a clear rationale, appropriate use of the Joanna Briggs Institute (JBI) methodology, and alignment with PRISMA-ScR reporting standards.
However, several methodological and conceptual issues limit the rigor, transparency, and reproducibility of the proposed review.
Major comments
1.The manuscript repeatedly refers to "AI-based models" but does not operationally define the term. Current automated scoring systems include: Rule-based systems, Machine Learning algorithms, Deep Learning models, Transformer-based architectures, Large Language Models (GPT, Claude, Gemini, Llama, etc.), Hybrid NLP approaches. These categories differ substantially in methodology and performance. My suggestion would be to add a conceptual framework that helps readers to distinguish between these terms.
2. Research question number 4 is problematic: "Are these models more effective than human graders in terms of reliability, accuracy, and fairness?" This question implies a comparative effectiveness judgment that may not be appropriate as it is beyond the aims of a scoping review. Scoping reviews aim to map evidence, describe characteristics and identify gaps. They generally do not evaluate superiority. A suggestion would be to replace it with: "How do studies compare the reliability, validity, fairness, and scoring performance of AI-based systems and human graders?"
3. The protocol states that databases will be searched and keywords will be combined using Boolean operators, but no search string is provided. The current reporting of this review does not allow replication of the search methodology. It is suggested to the authors to include a complete search strategy methodology.
4. The review includes studies from 2015 onward. This cutoff appears arbitrary. Authors should consider justifying the decision to exclude publications before 2015.
5. Data Analysis Plan Is Too Generic. The manuscript states that thematic analysis will be used. For a scoping review of AI systems, more detail is needed. A suggestion could be to describe how themes will be developed around: Types of AI systems, Accuracy metrics, Fairness concerns, Implementation barriers, educational outcomes.
6. The proposed extraction variables are too limited. Important variables are missing, such as: AI characteristics, assessment characteristics, performance metrics. It is recommended to
7. The manuscript continuously mentions fairness is assessment and the review aims to examine fairness, yet fairness is a subjective term as no operational definition or framework is provided.
Minor comments
The statement: "No ethical concerns..." should be revised. Perhaps a better wording could be: "Ethics approval was not required because the review utilizes publicly available published literature and does not involve human participants."
The manuscript cites PRISMA-P, which is designed for systematic review protocols. Since this is a scoping review protocol, the authors should clarify why PRISMA-P was selected rather than PRISMA-ScR extensions and JBI scoping review guidance.
Conclusion: The protocol addresses an important gap in medical education assessment research and has strong potential to contribute meaningfully to the field. However, several methodological details require clarification before indexing. In particular, the authors should strengthen the search strategy, provide a detailed data extraction framework, clarify definitions and outcome measures, justify eligibility restrictions, and elaborate on how fairness, validity, and reliability will be synthesized. These revisions would substantially improve the transparency, rigor, and reproducibility of the proposed review.
Is the study design appropriate for the research question?
Partly
Is the rationale for, and objectives of, the study clearly described?
Yes
Are sufficient details of the methods provided to allow replication by others?
Partly
Are the datasets clearly presented in a useable and accessible format?
Not applicable
Reviewer Expertise:
Health Professions Education; Educational Development; Teaching / Learning theory and pedagogies; Technology and Simulation based learning; Authentic Assessment.
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
The authors have presented a clear and well-articulated background and rationale for this scoping review protocol. The proposed study seeks to map the existing evidence on the use of artificial intelligence in the automated scoring of short-answer questions in medical education. The stated aim is to systematically map the literature with particular attention to the AI tools used, as well as their accuracy, reliability, and fairness. This is appropriate and relevant to current developments in medical education assessment.
The review is structured around the following research questions:
What AI-based models have been used for the automated scoring of short-answer questions in medical education? How accurately does automated scoring of short-answer questions reflect the performance of human graders? What advantages and challenges have been reported regarding the use of AI for automated scoring of short-answer questions in medical education? Are these AI-based models more effective than human graders in terms of reliability, accuracy, and fairness?
The use of the Population, Concept, and Context framework to guide the search strategy, eligibility criteria, and data extraction process is appropriate for a scoping review. The involvement of a librarian in the database search is also a strength and supports the methodological rigor of the protocol.
However, the following points require clarification or revision:
Overall, this is a timely and relevant scoping review protocol. Addressing the points above would improve the clarity, transparency, and reproducibility of the proposed methodology.
Is the study design appropriate for the research question?
Partly
Is the rationale for, and objectives of, the study clearly described?
Yes
Are sufficient details of the methods provided to allow replication by others?
Partly
Are the datasets clearly presented in a useable and accessible format?
Not applicable
Reviewer Expertise:
Orthopedic Surgery
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
The authors have attempted to develop a protocol for conducting a scoping review on the use of AI in automated scoring of short answer questions in medical education. Such a protocol without any visible data will probably be of no use to the prospective researcher.
The following is the elements of a protocol for scoping review
1. Review Questions
2. Eligibility criteria
3. Search Strategy
4. Data charting and mock extraction form
5.Template to be used for final report of review
The author should take effort to improve review questions by evidence-based analysis of the literature related to topic. Otherwise, it stands alone without evidence. The eligibility should be more descriptive. Each component P, C and C under search strategy should be described in detail with enough descriptions of each. Present description in general.
Mechanism of data charting, technology used and extraction form should be added
Please also look at JBI manual for evidence synthesis.
Is the study design appropriate for the research question?
Partly
Is the rationale for, and objectives of, the study clearly described?
Yes
Are sufficient details of the methods provided to allow replication by others?
No
Are the datasets clearly presented in a useable and accessible format?
No
Reviewer Expertise:
Technology-enhanced learning, Artificial intelligence, Online assessment
I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.