Effectiveness of a Mobile Application with an AI-Powered Chatbot to Enhance Nursing Students' Awareness of Substance Abuse Prevention: An Interventional Study

Significance of the study

This research adds to the existing body of literature that focuses on the application of digital health technology, specifically AI chatbots, in the fields of nursing education and substance use prevention. The study adds new empirical information regarding the potential effectiveness of AI chatbots as part of nursing education in a university located in the Middle East – an environment underrepresented in digital health research globally. In terms of nursing practice, the study highlights the possibility for utilizing such a technology to complement health promotion practices as the nurse’s involvement in the field of digital health is constantly growing.^9,10 Furthermore, the research is related to SDG 3 (Good Health and Well-Being).

Purpose and hypotheses

This study examined the effectiveness of a mobile application with an embedded AI-powered chatbot in improving nursing students’ substance abuse prevention awareness relative to a matched control group. Three primary hypotheses guided the investigation:

Research design

A quasi-experimental design, with two groups of nursing students independently recruited at the same faculty during the same academic period. The intervention group included students who had completed a three-month AI chatbot educational intervention as they available to be engaged in the intervention for three months, while the control group consisted of students who had not received this intervention and were assessed at the same time. Both groups were drawn from the same student population and were matched on key demographic variables such as gender, age, and academic year to minimize confounding factors.

Setting

Recruitment of participants occurred in the largest nursing faculty in Jordan from October 2025 to February 2026. The selected setting was due to having an accredited undergraduate nursing program, the university’s commitment to adopting technology-driven teaching methods, and a sufficient number of students for conducting the experiment on both the intervention group and control group. Similarity in the context among the two groups of participants was maintained in order to ensure no contextual difference.

Participants

The targeted sample comprised undergraduate nursing students in Jordan throughout all four study years. The accessible population include all students in the selected university, who agreed to participate, and met the following inclusion criteria: have an active enrolment in the Faculty of Nursing, age above 18, provision of informed consent in written form, and available to be engaged in the intervention for three months. Those who did not attend the day of assessment and/or those who left more than 10% blanks in the questionnaire were excluded from participation in the study. The intervention group included 85 students that had used the application of an AI chatbot in the last three months, whereas the control group comprised 85 students who had no contact with the application previously.

Sample size

The sample size for each group was determined based on a priori calculations for an independent-samples t-test. Assuming a medium effect size (d = 0.50), an alpha level of 0.05 (two-tailed), and 80% power, 64 individuals were needed for each condition. The final sample included 85 participants per group, yielding a power level of greater than 92% to detect medium effects, and even greater power levels for the larger effects detected.

Instruments

The data gathering involved the use of an organized survey which consisted of four parts. The first part concerned sociodemographic factors such as gender, age category, year level in school, and self-rating of the relationship quality with parents and friends. The second part of the survey was a measurement tool called Substance Use Risk Profile Scale (SURPS) Woicik et al.,.¹¹ This test includes 23 questions about personality characteristics that have been empirically proven to be differentially risky with respect to substance use. Participants are required to answer the questions on a four-point Likert scale (1 = strongly disagree, 4 = strongly agree). It consists of four subscales that measure different risk factors for substance abuse such as anxiety sensitivity (tendency to utilize substances in order to reduce arousal in relation to fear), hopelessness (a state characterized by depressed cognition that can lead to self-medication of substance use), impulsivity (lowered reflection capacity and risky behavior), and sensation seeking (inclination to search for thrill and novelty). Scores Subscale scores are computed by summing constituent items. The SURPS demonstrates sound psychometric properties across diverse samples, with Cronbach’s alpha coefficients consistently exceeding.70 and confirmatory factor analyses supporting the four-factor structure.

The third component was an adapted Knowledge-Attitude-Practice (KAP) questionnaire.¹² The instrument contained 15 questions – five for each of the three domains. For the Knowledge domain, respondents had to give correct answers that could be scored using the dichotomous format of answers (right/wrong), resulting in a domain score of 0 to 5. The Attitude domain was assessed using the five-point Likert scale (1 – strongly disagree; 5 – strongly agree), which produced domain scores from 5 to 25. The Practice domain entailed dichotomous or frequency measures, with domain scores from 0 to 32. Scores equal to or higher than 70% of the maximum possible domain score were indicative of sufficient knowledge, positive attitude, and proactive actions, respectively. Cronbach’s alpha was measured as.75 to.83 for different domains.

Intervention procedure

Phase I: Application Development and Baseline Assessment

A professional web-based application for prevention was created by a competent software engineer working together with nurse experts specializing in psychiatric nursing and community health nursing. The application consisted of a friendly interface, which included an introductory page explaining the purpose of the program, how to navigate through the interface, and ensuring the confidentiality of any collected data.

The application contained an integrated artificial intelligence (AI) chatbot that offered evidence-informed answers to students’ questions about substance type, effects on the body and mind, personal risks, and prevention techniques. The knowledge database of the AI was compiled based on authoritative sources such as WHO guidelines, UNODC reports, and scientific literature.

The application also encompassed organized educational modules covering the pharmacology of commonly abused substances, biopsychosocial risk pathways, and practical refusal skills. Interactive self-assessment quizzes were incorporated to reinforce learning, along with a curated resources section linking to international organizations (e.g., WHO, UNODC, and the National Institute on Drug Abuse) and relevant local referral services. The content was reviewed and validated by a panel of three experts in nursing education with specialization in substance abuse prior to implementation.

The baseline measurement was done before the intervention process took place. The KAP questionnaire and SURPS were administered among the selected subjects after signing the informed consent form. Data gathered from the baseline assessment would help establish whether or not both groups are equivalent to each other. Independent t-test was performed for comparing the two groups based on their scores while a paired t-test will be done before and after intervention.

Phase II: Intervention Delivery

The chatbot-led educational program remained accessible to the participants over a span of three continuous months. Before implementing the program, participants were instructed in writing and verbally regarding how to access the program, navigate through its different sections, and learn about the abilities of the chatbot. During the process of implementing the program, the participants used the chatbot on their own accord in asking questions from it and seeking clarifications in regard to what they had learned. This approach was adopted purposefully, since it maintained the anonymity and freedom considered significant in the theoretical framework for such sensitive topics. During this period, participants in the control group were not exposed to any material related to substance abuse.

Phase III: Post-Intervention Evaluation

As soon as the program ended after its three months’ duration, the subjects in both groups were asked to fill in the exact same questionnaire as before the intervention, including both KAP and SURPS surveys. The choice to conduct the assessment right after the program and not after a subsequent period of time was made in line with the main purpose of the research, which was to measure any possible immediate impact of the program in terms of education; this procedure is also commonly used in other chatbot-based health education experiments.^13,14 After the intervention, participants in the control group were given the right to access the chatbot for three months to get the benefits of the intervention, equal to the participants in the intervention group.

Pilot study

A pilot study was done before the actual intervention delivery, where 17 nursing students were included as part of the same population, but not in the actual sampling frame. The purpose of the pilot study was to examine whether chatbot usability and content are appropriate, and the questions in the survey were clear and easy to comprehend and estimate the approximate time taken to complete the questionnaire. The results of the piloting assured the need to simplify the instructions to use the chatbot, which was already done in the actual intervention. The results of the participants in the pilot study were not included in the final analysis.

Statistical analysis

Data were examined using IBM SPSS Statistics (version 27). Because the two groups were independently employed and assessed alongside, appropriate parametric tests were applied for both between- and within-group comparisons. Prior to inferential analyses, the distributional properties of all study variables were inspected within each group. Visual inspection of histograms and normal Q–Q plots indicated only minor deviations at the distribution tails. Skewness and kurtosis values for all variables were within ±1.5, supporting the assumption of normality.¹⁵ Levene’s test for equality of variances was conducted for all between-group comparisons and confirmed that the homogeneity of variance assumption was met. Baseline equivalence between the control and intervention groups was measured using chi-square test. To evaluate within-group changes from pretest to posttest, paired-samples t-tests were conducted separately for each group. Between-group differences at posttest were examined using independent-samples t-tests across KAP domains and SURPS subscales. Effect sizes were determined by using Cohen’s d, which is obtained through pooled standard deviation and cut-offs of 0.20 (small), 0.50 (medium), and 0.80 (large). Multiple linear regression was employed to predict SUPRS from KAP and the demographic variables. Standard regression diagnostics confirmed that assumptions of linearity, independence of errors (Durbin–Watson statistic), homoscedasticity, and normality of residuals were adequately met. A two-tailed significance level of p < .05 was adopted throughout the analysis.

Participant characteristics

A total of 170 nursing students participated in this intervention–control study, with 85 students assigned to the control group and 85 students to the intervention group. As presented in Table 1, there was no statistical difference between the two groups regarding all the demographic characteristics, chi-square (p > .05). Female students constituted the majority in both the control (59/85; 69.4%) and intervention (58/85; 68.2%) groups. Most participants in each group were aged 18 to under 24 years (control: 60/85, 70.6%; intervention: 62/85, 72.9%). Second- and third-year students formed the two largest academic-year cohorts in both groups (control: 41.2% and 35.3%; intervention: 42.4% and 34.1%, respectively). Relationships with parents and friends were reported as positive by the majority of participants in both groups (parents: 85.9% and 85.8% rated as good or very good in the control and intervention groups, respectively; friends: 96.5% in both groups). A 100% retention rate was observed in both groups, eliminating differential attrition as a potential threat to internal validity and strengthening confidence in the observed between-group outcome differences.

There was no change in the control group – p-values >.05, small effect sizes (dZ < 0.12). Thus, the current nursing curriculum without using the chatbot could not cause any changes in the substance abuse knowledge, attitude, practices, and risks over the period of three months. However, in contrast to this, the intervention group showed significant changes in all the parameters under investigation (p < .001). Effect sizes in the group are large: knowledge gained dz = 1.35, attitude improved dz = 1.96, and practice gained dz = 2.48. At the same time, there was a reduction in all four SURPS scales – the most considerable one was related to the Sensation Seeking scale (dz = 3.02), followed by the Impulsivity scale (dz = 2.51), Hopelessness scale (dz = 2.44), and Anxiety Sensitivity scale (dz = 2.42). Thus, by confirming that there were no other significant changes besides those observed in the intervention group, we can state that all between-group differences mentioned in the manuscript (Table 2) are related to the chatbot intervention.

The findings indicated significant improvements for those in the intervention group relative to the control group on all measures examined. Prior to the intervention, there were no significant differences between the two groups on knowledge, attitude, practice, or any of the subscales of the SURPS measure (p > .05). After the intervention, the intervention group experienced significant improvements in knowledge, attitude, and practice scores, with large effect sizes (Cohen’s dz = 1.35–2.48). The control group did not experience significant change over time. Posttest comparisons found that the intervention group scored significantly higher than the control group on all measures of the SURPS measure (p < .001), with very large effect sizes (d = 1.58–2.68). In terms of substance use risk profiles, substantial decreases were found for all SURPS dimensions in the intervention group, comprising anxiety sensitivity, hopelessness, impulsivity, sensation seeking, and SURPS scores overall (all p < .001). These decreases were noted to be associated with extremely high effect sizes (dz = 2.42–3.81), while no significant changes were found for the control group. Significant differences between groups at posttest also indicated lower SURPS scores for the intervention group as opposed to the control group (all p < .001), pointing to a highly positive impact of the intervention.

As shown in Tables 3a, b, c, b, and e no statistically significant differences were observed between age groups, male and female, academic levels, type of relationships with parents, and the of relationship with friends across all outcome variables (all p > .05). In Table 4, the multiple linear regression model using the enter method was statistically significant, F (13, 71) = 6.24, p < .001, explaining a substantial proportion of variance in post-test SURP scores (R² = .533). The findings reveal that knowledge, attitude, and practice are significant predictors for reducing the substance use risk personality.

Limitation of the study

There are several limitations that need to be taken into account when analyzing the results obtained during this study. First of all, there was no randomization of participants; instead, the research applied a quasi-experimental nonequivalent control group design, which means that the establishment of causal relationships is difficult and the chance for selection bias exists. Secondly, the study took place in one faculty of nursing in Jordan, which makes it difficult to generalize the results to other groups of nursing students. Thirdly, all the information obtained during the research was based on self-reported data from questionnaires, thus being prone to possible biases. Fourthly, the intervention period was relatively short, and there was no follow-up test carried out in order to see whether changes observed persisted over time. Fifthly, the sample included only nursing students, which makes it difficult to apply the results obtained to the population from other healthcare specialties.

Recommendations

In light of the results obtained from this study, it is highly recommended that AI-assisted chatbots applications be integrated into the undergraduate curriculum in the field of nursing. Such a move will help to supplement the current methods used to impart knowledge to students in psychiatric and community health nursing regarding the prevention of substance abuse. Furthermore, nursing educators are advised to incorporate mobile-based and AI-enhanced learning systems within their curricula so as to maximize student engagement and increase information retention. The implementation of these technological innovations early in a nursing education program can go a long way in boosting their cumulative impact. Additionally, future advancements in these platforms should entail the inclusion of adaptability, personalization, and scenario-based learning to enhance efficacy even more. It would be prudent for subsequent longitudinal studies to determine how effective this method is at promoting students’ performance as clinicians in real-life settings. More studies could also be carried out to assess the suitability of such interventions for other health science courses.