A Case Study on Assessing AI Assistant Competence in Narrative Interviews

The possibility

The use of AI chatbots in various real-life applications is rapidly increasing due to their ability to reduce reliance on humans, lower costs, improve efficiency, and streamline service experiences (Adamopoulou & Moussiades, 2020; Bendig et al., 2019; Chan & Li, 2023; Greer et al., 2019; He et al., 2022; Liu et al., 2022; Omarov et al., 2023; Shah et al., 2017; Tamayo et al., 2020; Tanana et al., 2019; Xu & Zhuang, 2022).

Before we proceed, it is important to clarify some terminologies related to conversational user interfaces. Research on these interfaces, including dialogue systems, embodied conversational agents, and social robotics, is converging towards the development of improved conversational computing systems. These systems are increasingly referred to as “chatbots” in both industry and research, a term that broadly encompasses conversational agents designed for various purposes, such as task completion, information delivery, entertainment, and social interactions, through text, voice, or both (Følstad et al., 2021). Unlike more narrowly defined uses of the term, this understanding of chatbots focuses on the design, development, and implications of these systems rather than specific technologies. Given the rapid evolution of technology and usage patterns, maintaining distinctions between types of conversational agents is challenging, as there often is significant overlap in functionality and modality. In this study, we generally use the term “chatbot,” but we specifically refer to ‘AI Assistant’ when discussing OpenAI’s Assistant API (Application Programming Interface).

Central to this AI chatbot proliferation is GPT (Generative Pretrained Transformer), a deep learning model that uses a self-attention mechanism to weigh different parts of input sentences and determine their importance. By converting sentences into tokens and predicting the next token’s probability, GPTs can generate coherent text that often appears human-written. Various studies have demonstrated this capability well (Brown et al., 2020; Wei et al., 2021; Wu et al., 2021). GPT-based chatbots, for example ChatGPT, can understand natural language inputs and produce responses that are contextually appropriate and coherent, thereby improving interactivity and efficiency (Bendig et al., 2019; Nath et al., 2021; Shah et al., 2017; Xu & Zhuang, 2022).

Since 2023, advancements in customizable features have further transformed the landscape of GPT applications, enabling not just programmers but also non-technical users to become GPT designers rather than merely end-users, thanks to simplified coding and prompt engineering (Stallbaumer, 2023; UC Today, 2023). For example, OpenAI introduced ChatGPT Builder and Assistant API (Application Interface) in November 2023, enabling users to build custom GPTs without technical expertise or coding experience (DeVon, 2023; UC Today, 2023). ChatGPT Builder provides a user-friendly interface for creating custom chatbot versions without coding, featuring built-in tools that OpenAI fully hosts. However, it does not support integration with other application platforms. On the other hand, the Assistant API also offers a Playground interface for building custom chatbots without coding. Unlike ChatGPT Builder, it allows developers to integrate these chatbots into various applications, such as instant messaging platforms like WhatsApp or Facebook Messenger, provided they have some coding skills and knowledge of these third-party platforms.

These backend platforms allow users to use natural language instructions to define the AI’s role and context, set its tone and behavior, and design stepwise procedures, making the chatbot’s responses relevant and coherent. For example, a customer service chatbot can be instructed to greet users politely, answer frequently asked questions, and escalate issues to a human agent if it cannot provide a satisfactory answer. Additionally, a chatbot for educational purposes can be guided to explain concepts in simple terms, provide examples, and quiz users to reinforce learning. By tailoring the AI’s behavior and responses, chatbot designers can create a more personalized and effective experience for various applications. This high level of customization expands and popularizes chatbot applications.

Against this background, chatbot researchers have identified several areas for future development, with democratizing chatbots for social good being one of the core themes (Følstad et al., 2021; Powell et al., 2023; Van Brummelen, 2019). This research theme aims to make chatbots more accessible and useful for people with non-technical backgrounds, including marginalized communities and domain experts.

The challenge

Although there is potential for democratizing AI chatbots, it remains uncertain whether advancements in customizable GPTs will enable social workers to effectively use AI tools. A significant challenge, however, is integrating specialized knowledge from the profession into the development of chatbots and assessing their usability and effectiveness in real-world settings (Følstad et al., 2021; Powell et al., 2023). Integrating domain knowledge into chatbot development requires bridging the gap between AI technology and the expertise of domain professionals. This gap often leads to tensions, as AI developers’ and domain experts’ goals and methodologies can differ significantly. AI developers might focus on technical feasibility and innovation, while domain experts, such as social workers, emphasize chatbots’ practical applicability and ethical implications in their fields. Effective collaboration is needed to reconcile these differences, ensuring that the chatbot is both technologically advanced and aligned with the nuanced needs of its target users.

The challenge goes beyond the simple lack of technical knowledge among domain experts and the need for one-sided training; it requires addressing the deeper, multidimensional barriers to effective collaboration between AI developers and professionals in specialized fields. The concept of affordance may provide a conceptual reference point for understanding how AI technologies may interact with domain experts like social workers. The concept of affordance refers to an object’s potential uses or actions—its latent utility—regardless of whether these possibilities are recognized by a person (Gibson, 1977). In the context of technology, affordances describe the action possibilities of a device that are perceivable by a user, indicating that the use of AI technologies is influenced not just by their design or technical features but also by how users perceive and interact with them based on their social and cultural backgrounds, experiences, and needs (Norman, 1999). Such a concept has been widely discussed in the context of technology in education, inspiring research into the dynamic relationship between technology and human activity (Bower, 2008; Chan & Holosko, 2017; Hammond, 2010; Leonardi, 2011).

These discussions highlight that technologies like AI are not neutral tools; their use and limitations are shaped by their users’ preconceptions and creative adaptations. Therefore, we cannot assume that users will employ AI features exactly as their designers intended. Users often interpret and repurpose technologies based on their unique perspectives and needs, which can extend beyond the original design intentions. For instance, while AI chatbots are typically designed for one-time customer support or information, social workers might repurpose them for ongoing needs assessments in continuous dialogues, leading to unexpected or innovative applications in different contexts.

The extent to which social work can embrace this customizable AI technology remains unclear due to the current research environment in the field. A 2016 review of social work interventions enhanced by information and communication technology (ICT) identified several critical issues (Chan & Holosko, 2016). One primary concern was the “black-box” problem, where unclear processes make developing transferable skillsets difficult. Additionally, the use of ICT has often shifted the focus from skilled practitioners to standardized programs, raising concerns about diminishing the role of social work professionals in such technology-based interventions (Chan & Holosko, 2016). Nearly a decade later, this situation remains essentially unchanged. Discussions around AI in social work do not focus on technical features or processes but primarily on conceptual and ethical issues (Reamer, 2023; Robila & Robila, 2020).

In addition, technical evaluations of chatbots often involve complex algorithms and backend data (Deriu et al., 2021; Finch & Choi, 2020; Lee et al., 2023), which are usually beyond the understanding of lay users and those outside the technical field. This creates a challenge for social workers when it comes to integrating chatbots into their practice. It’s paradoxical that if they lack the knowledge and ability to evaluate a product they have helped create. This disconnect can make it challenging for social workers to effectively assess and contribute to the development process.

Given these rapid and user-friendly technological developments, an important question arises: Can these new customizable AI tools be meaningfully integrated with social work’s domain knowledge and practices? The current literature lacks discussion on how social work professionals can instruct and evaluate customizable AI chatbots in practical settings. While innovative features like creative writing, text-to-image transformation, answering queries, and assisting students with paper writing are becoming more prevalent, they do not directly align with the practical needs of social work. Social work should focus on how well chatbots can meet specific communication needs, support client engagement, and enhance service delivery rather than on novelty or creativity.

This study specifically chose OpenAI’s Assistant API as the platform for developing and testing AI chatbots because it offers high customization and flexibility (DeVon, 2023; Stallbaumer, 2023; Wang et al., 2023), allowing social work professionals to tailor the tool to their unique requirements. OpenAI Assistant is widely recognized for its user-friendly interface and robust capabilities, making it accessible even for those with minimal coding experience. By leveraging these features, social workers can integrate AI more easily into their practice without extensive technical knowledge.

We recognize that OpenAI Assistant is not the only product on the market, and technical innovations continually evolve. However, we chose OpenAI because its readiness for deployment, widespread adoption in various industries, comprehensive support resources, and active user community provide a solid foundation for exploring, testing, and refining the use of AI in social work settings. This socio-technological landscape inspires and enables us to explore how AI chatbots can be practically integrated and evaluated within social work.

Narrative interviews in social work

The focus on interviews in this study is a choice for research purposes because many duties in social work involve conversations, such as training, assessments, counseling, and client interactions. Interviews are fundamental for gathering information, building rapport, and understanding clients’ unique situations and needs (Bogo, 2006; Kadushin & Kadushin, 1997). Furthermore, the essence of generative AI lies in its ability to engage in human-like conversations, making it a potential tool for facilitating these interactions. The need to conduct interviews and the capabilities of generative AI make a perfect match. By focusing on AI-driven interviews, the research aims to explore how generative AI can be utilized to enhance conversational tasks in social work, supporting practitioners in their efforts to connect with clients more effectively and empathetically.

By “interview,” we refer to dynamic, open-ended, and exploratory interactions rather than online forms with a chatbot facade. This study uses the term “narrative interview” to describe a conversational approach facilitating deeper exploration of stories. In narrative interviews, we expect the chatbot to engage in meaningful dialogue, respond to the natural flow of conversation, and adapt to the nuances of participants’ stories. This flexible, client-centered approach contrasts with structured online questionnaires, which follow a predetermined script and do not consider the unique context or needs of the individual.

Existing chatbot products in social and healthcare services do not support such contextualized narrative interviews as they are often therapy-oriented, primarily based on Cognitive Behavioral Therapy (CBT). For example, chatbots like Wysa and Woebot rely on CBT techniques and follow structured treatment plans with preset responses (Omarov et al., 2023; Xu & Zhuang, 2022). Social work interviews often involve asking questions, advocating, building rapport, conducting outreach, and using flexible, narrative-based approaches to explore and address a range of social and emotional issues (Bogo, 2006; Kadushin & Kadushin, 1997). Most existing healthcare chatbot products leave little room for the involvement of social work practitioners, as they are pre-made products based on predefined therapeutic protocols. This creates a gap between current healthcare chatbot products and social work practice’s needs.

For social work, chatbots that support narrative interviews could be highly relevant to a wide range of direct practices (Chan & Li, 2023). Narrative interviews in social work settings may include applications such as counseling intake, initial needs assessments, community studies, and contextualized inquiries. These types of interviews require chatbots that can handle a variety of conversational contexts and adapt to the specific needs of each situation. By configuring chatbots to facilitate narrative interviews, social workers may leverage AI to gather more nuanced information and engage more efficiently with clients.

Some researchers are now at the forefront of developing AI tools specifically for interview purposes (Cordero et al., 2022; Han et al., 2021). Additionally, commercial entities have begun to harness the power of generative AI to conduct qualitative customer interviews, streamlining the process of gathering and analyzing feedback (e.g., Listen Labs https://listenlabs.ai/). These AI tools can conduct interviews, record responses, and use advanced algorithms to interpret data, uncovering insights that might be overlooked. By automating repetitive tasks, AI may ccelerate the interview process and ensures high precision.

Given these advantages, customizable AI-driven narrative interviews may substantially benefit social work. However, this emerging trend is largely under-researched or virtually nonexistent, presenting exciting opportunities and significant challenges. Narrative interviews are inherently complex because they require interaction, continuity, responsiveness, and deep engagement with participants’ stories. In these interviews, the dialogue goes beyond merely collecting information; it involves collaboratively constructing a narrative, which can influence the interview subjects (Chase, 2018; Clandinin, 2007). If AI chatbots are to conduct narrative interviews, what essential competencies must they have to capture the depth and richness of participants’ experiences adequately?

Expected competencies from asynchronous text-based chatbots

This study specifically focuses on asynchronous text-based chatbots because, at the current stage of technology development, these types of chatbots are more accessible, flexible, and easier to implement in diverse settings compared to real-time, synchronous verbal communication systems (Luo et al., 2022). Asynchronous chatbots allow users to engage at their own pace, which can be particularly beneficial in social work and counseling contexts, where participants might need time to reflect and compose their responses. Additionally, asynchronous interactions reduce the pressure on users to respond immediately, fostering a more thoughtful and reflective dialogue that aligns with the principles of narrative inquiry. Furthermore, these chatbots create a retrievable text-based record of interactions, offering a valuable resource for both reflection and documentation, which can be useful for tracking progress and outcomes in therapeutic or consultative sessions.

As an inquiry method, narrative interviewing is closely informed by constructivist and social constructionist worldviews, which suggest that reality is constructed through social interactions and discursive contexts (Abkhezr et al., 2020; Denicolo et al., 2016). Participants engaging in meaningful conversations during interviews can contribute to content construction and reflection (Gergen, 2001; Shotter, 1993). As such, narrative inquiry generally assumes a more active role for interviewers and interviewees in which their “active narrativity” (Gubrium et al., 2012, p. 28) is highlighted.

As AI chatbots continue to evolve, researchers and industry experts have proposed various metrics to assess the performance of AI (Katic et al., 2024; Sharma, 2020; Xu & Jiang, 2024). Existing literature primarily measures the overall technical performance of dialogue systems, such as response time, language proficiency, repetitions, and human likeness (Deriu et al., 2021; Finch & Choi, 2020; Lee et al., 2023), rather than focusing on the user end experience at the conversational level. While evaluating the technical features of language models is fundamental, these assessments can be overly complex and may not directly apply to interview contexts. Some researchers have provided practical measures at the conversation level; for example, Concannon and Tomalin (2023) suggest using an Empathy Scale for Human–Computer Communication, based on raters using a Likert scale to assess the textual content of conversations. This instrument is relevant and valuable, but empathy is just one domain in narrative interviews. Other areas also need to be addressed to ensure the quality of interviews.

Enabling non-technical background developers, such as social workers, to assess AI’s role in narrative interviews is crucial because it ensures that these technologies are used in ways that genuinely enhance social work practices supported by their domain knowledge. By focusing on competencies relevant to social work, such as empathy, adaptability, and the ability to foster meaningful dialogue, social workers can better evaluate whether AI chatbots effectively support their goals. This approach empowers social workers to shape AI chatbots to meet their unique needs actively, ultimately bridging the gap between technological innovation and practical application in human services.

Identifying the full range of competencies expected from asynchronous text-based chatbots is a complex task that extends beyond the scope of this study. Nonetheless, we have selected two core competencies to focus on—fidelity to the interview guide and empathetic engagement—to facilitate the discussion and guide the empirical aspects of our research. The rationales are as follows:

First, text-based online narrative interviewing is much more than just filling out a self-report questionnaire. In narrative interviews, the sequence of questioning is crucial. Unlike self-report questionnaires, where each question is independent and can be arranged randomly, narrative interviews involve a continuous dialogic process that reflects the participant’s evolving story. A narrative inherently implies a sequence and consequence of events, and the order of questions can significantly affect the outcomes (Riessman, 2008; Riessman & Quinney, 2005). Interviewers must be sensitive to this sequence and guide the conversation to remain focused on the specific topic of inquiry while allowing for the natural flow of the participant’s story. Several counselor competence frameworks have highlighted questioning as one of the top skills required for interviewers (Fowler et al., 2021; Rice et al., 2022; Swank et al., 2012). In narrative practice, the structure or progression of questions is often seen as essential to achieving the intended effects (Duvall & Béres, 2011; White, 2007). Furthermore, narrative interviewers need to facilitate participants in exploring and constructing meanings (Abkhezr et al., 2020). This expectation is well-suited for asynchronous text-based chatbots because OpenAI Assistants can be instructed to follow specific sequences and adapt their responses based on interviewee input (DeVon, 2023; UC Today, 2023). Additionally, all interactions are text-based, making the records retrievable and analyzable.

Second, narrative interviewers need to demonstrate empathy. In narrative interviews, participants are not merely providing factual descriptions but actively producing data through their narratives. This requires the interviewer to be able to exercise “active narrativity” (Chase, 2018; Clandinin, 2007). As such, the narrative interviewer’s capacity to be empathic, nonjudgmental, concerned, tolerant, and emotionally responsive is prioritized (Josselson, 2007, p. 539). Demonstrating empathy helps create a safe and supportive environment where participants feel comfortable sharing their stories and actively contributing to knowledge production. Empathy is also frequently highlighted in counselor competence frameworks as a critical skill for effective interviewing (Fowler et al., 2021; Rice et al., 2022; Swank et al., 2012). This expectation is reasonable and suitable for asynchronous text-based chatbots because OpenAI Assistants can be designed to recognize and respond to emotional cues in the participant’s text, providing empathetic responses that align with the client’s conversational flow and demonstrate understanding and support.

The research questions

In summary, among various potential competencies for AI chatbots, we have chosen to focus on two key competencies—fidelity and empathy—because they are particularly relevant to conducting narrative interviews. This focus allows us to explore the minimal requirements for effective performance in this context, serving both heuristic and research purposes. Fidelity involves adhering to the guidelines set by designers and practitioners, ensuring that the chatbot maintains the intended structure and focus of the narrative interview. Empathy requires aligning with the interviewee’s conversational flow and responding to demonstrate genuine understanding and support for the participant’s emotional and narrative journey. By meeting these fundamental competencies, AI chatbots may have the potential to facilitate meaningful interactions during narrative interviews. With this foundational understanding, our study aimed to answer the following questions:

The rationale

In this study, we adopted an autoethnographic case study method to research customizing generative AI for narrative therapy interviews. Autoethnography is a qualitative research method where researchers analyze their own personal experiences to understand broader contexts. It combines autobiography and ethnography, using personal narratives to explore and reflect on cultural phenomena (Ellis & Bochner, 2000). We chose this method for two primary reasons. First, autoethnography is increasingly recognized as a practical approach for researching new technologies and social robotics, as it allows for a critical exploration of personal experiences within broader contexts (Chun, 2019; Mao et al., 2023). This approach provides a unique, firsthand perspective on the subjective aspects of technological interaction. Second, autoethnography allows us to evaluate new tools before introducing them to clients or other human participants. This approach allows us to move cautiously and thoughtfully, considering the technology’s impact and limitations to ensure more ethical decisions in future experiments with service users.

We conducted a collaborative autoethnography, where researchers collectively reflect on and analyze their personal experiences to gain deeper insights into a phenomenon (Chang et al., 2016). We incorporated a jigsaw approach, where each teammate focused on a specific aspect of the study and then shared their insights with the group. This method encouraged a more thorough exploration of the topic by ensuring that different angles were covered and integrated into the overall analysis.

This autoethnographic journey involved several stages. First, we set up an AI chatbot using OpenAI’s customizable GPT system. Next, we explored whether the chatbot could function properly in real-life situations. In this study, the first author designed and deployed the chatbot, carefully crafting the instructions, while the second author participated in blind testing conversations, initially unaware of the specific instructions given to the chatbot by the first author. The third author was responsible for collecting and organizing the data from these conversations, ensuring accurate dialogue transcription, and systematically coding the data for analysis. We also conducted a preliminary review of the transcripts, noting any deviations from the expected conversational flow and highlighting instances of empathetic responses by the AI. This preliminary analysis provided a foundation for the utterance analysis method we later used to evaluate whether the human-chatbot conversations followed the intended progression and displayed the required empathetic attitudes. Finally, all authors came together to discuss their gut feelings, observations, and insights from the experience.

Our background

We are social work researchers exploring the integration of technology into social work practice. The first author has an undergraduate background in social work and over 20 years of frontline experience in Hong Kong. He has also conducted research on the use of technology in social work interventions and is currently an associate professor at a university. The second author, originally from mainland China, is pursuing her PhD under the supervision of the first author. The third author is also a PhD student in the social work department. Our social work background is an asset because it allows us to approach the integration of AI into social work from a practitioner perspective, ensuring that the technology developed is accessible and relevant to social workers. Such an autoethnographic study helps test the boundaries of our current knowledge and skills and raises questions about social work domain knowledge in an era increasingly influenced by AI chatbots. The project was approved by the Hong Kong Baptist University Research Ethics Committee project REC/23-24/0385. This study is an autoethnography, so the authors have indicated their consent to participate.

We instructed the open AI assistant to follow a specific conversation strategy

As social researchers with minimal or almost no programming background, we could configure a chatbot using OpenAI’s Assistant API (Application Interface) (https://platform.openai.com/) using its GPT-4 model. It is worthwhile to note that this Assistant API differs from the highly user-friendly ChatGPT, as it offers more robust features tailored for developers, and this API can enable us to deploy chatbots on websites and various messaging platforms, offering flexibility and user accessibility without relying on OpenAI’s user interface. OpenAI Assistant API allows users to set long, stepwise, and systematic instructions for the chatbot on the back end. All these can be done without coding, using the Playground on OpenAI.

We deployed our chatbot on WhatsApp using coding instructions sourced from developer forums. Using WhatsApp is good for several reasons. First and foremost, the use of WhatsApp in research and interviews is widely discussed as it can potentially enhance communication and collaboration outside traditional lab settings (Gasaymeh, 2017; Suárez-Lantarón et al., 2022) and it allows for convenient, accessible, and flexible communication that fits into users’ everyday lives (Ahmad & Purwaningrum, 2022; Lin et al., 2022). Second, many people are familiar with WhatsApp, so chatting with a chatbot does not require them to pick up new skills. Third, users can get quick replies and support when using an AI chatbot because the interaction is immediate and yet asynchronous, chatting with that chatbot anytime and anywhere.

We adopted a structured conversational framework that prompted participants to engage thoroughly with their personal stories, fostering a deeper self-reflection and integration of insights. The framework used in this study was partially informed by questioning skills used in narrative therapy (Au-yeung, 2023; Chan, 2012, 2023; Chan et al., 2020; Chan et al., 2012; Ramey et al., 2009; Ramey et al., 2010; White, 2007). In this study, the conversation plan involves unfolding details and elaborating connections. It then invited the participant to propose a name based on inductive reflection and explore deeper core values. The stages are as follows: 1. Orientation, 2. Unfolding, 3. Naming, 4. Explaining, 5. Exploring core values, 6. Aspirations, and 7. Closing. Instructions for the AI, specifically the interview guide, are available in the published dataset (Chan, 2024).

In this study, we utilized this framework to balance complexity and manageability in our testing. The conversation plan was designed to be neither too simple nor too complex, allowing us to effectively test the AI’s ability to navigate nuanced dialogues while maintaining continuity and demonstrating interview skills. Unlike more superficial, disjointed question-and-answer dialogues, this structured approach provides a cohesive flow that supports a deeper exploration of the participant’s experiences. This interview plan was tailored to meet the specific needs of our study, but practitioners or researchers can develop their own structured steps or modify these steps to suit various purposes, such as conducting specific assessments, managing service inquiries, or performing different types of tests.

While an in-depth theoretical discussion of interview strategies is beyond the scope of this study, our primary objective is to evaluate the chatbot’s adherence to these instructions and its capability to facilitate meaningful conversations. This interview plan is adequate for us to assess the AI chatbot’s performance in a practical context,

We tested it with our own stories

We engaged Yunmeng, the second author of this article, for an AI-facilitated narrative interview to discuss her cultural adaptation and academic challenges in Hong Kong. The first author designed an AI chatbot to conduct the interview, allowing Yunmeng to respond independently. Notably, Yunmeng was unaware of the specific steps of the interview plan before engaging with the chatbot, although she knew the interviewer was a bot and not a human being. Yunmeng was requested to share the same story across two distinct conversations. She followed a standard process in the first round, earnestly answering the questions as instructed. In the second conversation, she spent more time elaborating on different events related to her issue and deliberately attempted to divert the chatbot from the main interview path. These two versions were developed for research purposes: the first was designed to be serious, while the second was deliberately non-cooperative. We intentionally crafted these differences to test how well the chatbot could capture and reflect these variations. We collected the data for the study via the AI chatbot, which conducted the narrative interviews via WhatsApp on 2024-01-17. Yunmeng interacted with the chatbot asynchronously, which allowed her to respond at her own pace.

We analyzed the conversation transcripts

This study used rating rubrics that social workers typically use to evaluate the performance of trainees. By using these familiar methods, domain experts could effectively assess the competencies of the chatbots. This approach helps ensure that the chatbots are relevant and useful for practical applications in social work.

Assessing fidelity

To assess the fidelity of the chatbot’s performance, we used a three-level rubric (poor, fair, good) to determine the extent to which the assessor (the first author) judged that the interviewer followed the conversation strategy as planned (see Table 1).

Building on the “Expected Competencies” outlined in the introductory section, fidelity is operationally defined as the degree to which the chatbot follows the interview steps—1) Orientation, 2) Unfolding, 3) Naming, 4) Explaining, 5) Exploring core values, 6) Aspirations, and 7) Closing—logically and sequentially (see Table 2 for a detailed breakdown of each interview step and its specific criteria).

Based on this operational definition, we developed a rubric to evaluate the chatbot’s ability to facilitate narrative interviews by asking appropriate questions guiding each step. A “Poor” rating indicates ineffective questioning that disrupts the flow, “Fair” suggests basic but sometimes unclear or incomplete questions, and “Good” reflects effective questioning that ensures a logical, comprehensive interview. This rubric guides evaluators in assessing the chatbot’s adherence to the interview process—covering Orientation, Unfolding, Naming, Explaining, Exploring Core Values, Aspirations, and Closing—rather than serving as a strict scoring tool. Evaluators mark the relevant performance level for each step and form an overall judgment of how well the chatbot follows the interview sequence, allowing for a nuanced assessment of its effectiveness. The first and third authors read the interview texts. They then applied the rubric to evaluate the degree to which the AI chatbot adhered to the interview plan. To support these assessments, specific dialogue examples from the conversations were highlighted to showcase instances of effective adherence, missteps, or deviations in following the structured interview process.

To present the conversation flow more vividly, we also employed a simplified sequential analysis to visualize the development of the conversation utterances. To represent progression in asynchronous conversations, we assigned sequence numbers to each dialogue turn, representing the development of the conversation. Additionally, we used a proportional timeline approach, dividing the conversation into four equal quarters based on the number of utterances rather than actual time. This method allows us to account for varying lengths of conversations. For instance, in a conversation with 100 utterances, each quarter would consist of 25 utterances, with the first 25 as Quarter 1 and the last 25 as Quarter 4.

The utterances from these interviews were organized in a spreadsheet for detailed tagging conducted on 2024-05-14. We manually assigned progression scores to indicate the stage reflected by each conversation utterance, tagging each utterance based on the operational definitions noted earlier. We used ‘1’ for utterances related to orientation, ‘2’ for unfolding story episodes, and so on. A score of ‘0.5’ was assigned to utterances that did not fit into any of these categories, such as misunderstandings, irrelevant comments, or instances where the conversation got sidetracked (see Table 2).

To ensure consistency in the ratings, we conducted inter-rater reliability tests between the manual tagging results and the machine tagging results. The first and third authors independently tagged the two conversations on 2024-05-14. Any differences in the tagging were reviewed and discussed until all disagreements were resolved, ensuring a reliable and accurate assessment. These quantitative ratings are not meant to be absolute measures but serve as heuristic tools to guide our evaluation process.

Assessing empathy

To assess empathy, we used a three-level rubric (poor, fair, good) to determine the extent to which the assessor (the first author) judged the chatbot’s ability to demonstrate empathy in a conversation based on an overall interpretation of that conversation (see Table 3). The empathy level score used in this study is partly informed by the affective component of the Empathy Scale for Human-Computer Communication proposed by Concannon and Tomalin (2023), which advocates using a set of observable indicators to evaluate empathetic behavior. These indicators are assessed using a Likert scale to measure how effectively the chatbot’s responses align with the user’s emotional tone or intensity.

Building on the “Expected Competencies” outlined in the introductory section, empathy is operationally defined as the chatbot’s ability to demonstrate concern, compassion, or empathy that aligns with the user’s emotional tone or intensity. This is assessed by examining the textual content of dialogues to observe how the chatbot uses empathetic language, expressions of understanding, and supportive feedback that acknowledges and validates the user’s emotions.

Based on this operational definition, we created a rubric to assess the chatbot’s ability to show empathy through its language, understanding, and emotional alignment. “Poor” performance reflects a lack of empathy or inappropriate responses, “Fair” suggests some empathy with inconsistencies, and “Good” shows consistent, appropriate empathy, ensuring the user feels understood and supported. The rubric serves as a flexible guide rather than a rigid scoring tool, evaluating the chatbot’s ability to recognize emotions, use empathetic language, offer supportive feedback, and validate the user’s feelings. Assessors select the relevant performance level for each area and form a holistic judgment of the chatbot’s empathy competence, allowing for a nuanced, context-sensitive evaluation.

The first and third authors manually tagged the interview text for meaning units reflecting empathy. They then applied the rubric to evaluate the degree to which the AI chatbot demonstrated empathy. To support these assessments, specific dialogue examples from the conversations were highlighted to showcase the chatbot’s use of empathetic language and responses, illustrating how the chatbot engaged with the user’s emotional state and provided appropriate support throughout the conversation.

We compared observer assessments with interviewee experiences

To ensure a comprehensive evaluation of the AI chatbot’s performance, we compared the observer assessments with the interviewee’s personal experiences. After the interview sessions, Yunmeng was asked to reflect on her experiences with the AI chatbot. She provided feedback on various aspects of the interaction, such as the AI’s responsiveness, its ability to stay on topic, and the emotional resonance of its responses. This subjective feedback was then compared with our rating of the AI’s empathy based on the entire conversations and specific dialogue examples. Yunmeng’s experience provided insight into whether the AI’s empathetic expressions were genuinely felt or seemed formulaic. Similarly, the observers’ tagging of meaning units for empathy was cross-checked against Yunmeng’s own reactions to determine if the AI’s responses matched her emotional needs during the conversation.

Challenges

1. At the outset, translating human skills into generative AI competencies is not entirely straightforward. One of the main challenges for social workers is converting their human-centric skills into competencies that AI Chatbots can follow and replicate. Skills such as questioning or empathy are inherently complex and context-dependent. Developing languages that accurately capture these nuanced competencies requires a deep understanding of both social work practice and AI technology, which can be daunting for many practitioners.

In our study, we have developed some insights into effective, prompt engineering for AI Chatbot, including setting clear contexts by defining the chatbot’s identity, mission, and attitudes; breaking down actions into distinct statements using behavioral terms; crafting step-by-step instructions with conditional (if-then-else) logic; and providing examples to guide AI responses (see Chan, 2024 for the instructions we finally set for the chatbot). This approach is essential for ensuring the AI can mimic the nuanced behavior of a skilled interviewer.

That said, there is also a need to expand language-based communication skills. Our experience designing instructions for the AI chatbot underscored the necessity of enhancing these skills within social work practice. Unlike human assistants, AI requires precise, structured prompts and behavior-based instructions to function effectively in a narrative interview setting. Initially, our approach used a basic framework, but it quickly became apparent that more detailed, step-by-step instructions were needed to capture the subtleties of narrative interviews. This evolution highlights a critical shift from viewing AI as a passive tool to recognizing the active role practitioners must play in configuring AI to shape conversation processes. Developing a comprehensive instruction set for AI differed from traditional social work practices, requiring a deeper understanding of how language influences AI behavior. This journey demonstrated that while tools like the OpenAI Assistant API offer a user-friendly interface, maximizing their potential still requires some foundation in crafting effective prompts.

2. Another critical aspect is considering the extent to which basic coding skills and technical knowledge are necessary or unnecessary for social workers. While platforms like OpenAI and others claim to be user-friendly, there are still significant technical challenges that social workers may encounter. For example, deploying the AI chatbot on WhatsApp revealed the importance of incorporating basic coding skills and technical knowledge into actual practice. Initially, we were cautious about the technical complexities involved in using a familiar messaging platform like WhatsApp. However, ultimately, this approach facilitated a more natural and user-friendly interaction, resembling a casual conversation rather than a lab experiment. Although we believe that as technology advances, platforms will become more genuinely user-friendly, potentially achieving a true “no coding” experience, there will still be a need for some technical understanding in the foreseeable future. Social workers may not need to be coding experts, but a basic understanding of technical concepts can significantly enhance their ability to utilize these tools effectively and facilitate communication with programmers and developers. This foundational knowledge is crucial as we progress towards a time when AI can interact with humans as seamlessly as we communicate. Until we reach that point, the ability to bridge the gap between social work practice and technical expertise will remain necessary for integrating AI into social work in a meaningful and impactful way.

Throughout this process, it became apparent that while detailed programming knowledge is not required, a basic ability to collaborate with technical experts is crucial. As AI technology evolves, social workers may need to remain adaptable and open to acquiring new skills. Including basic coding knowledge in social work education can enhance practitioners’ ability to utilize AI tools effectively, bridging the gap between professional knowledge and technical implementation. This adaptability ensures that social workers can integrate AI into their practice without losing the human touch that is vital for social work.

Opportunities

1. Prompt engineering: To begin with, social workers may have an advantage when it comes to contributing to prompt engineering. Prompt engineering is the process of designing and refining the inputs given to AI systems to elicit specific, desired responses (Wang et al., 2023). Social workers understand the essential skills and qualities needed for effective narrative interviewing, such as questioning skills, empathy, and adaptability. By leveraging this domain knowledge, they can play a crucial role in developing and evaluating AI chatbots, ensuring these tools are tailored to the specific needs of social work practice.

For example, we developed appropriate interview guidelines in this study and instructed the AI to adopt the right attitudes, demonstrating that natural language instructions can effectively guide AI behavior. This approach ensures that the AI responds in a way that is consistent with interview principles, fostering a supportive environment for interviewees. The ability of social workers to craft such prompts means they can help design guidelines for AI that align with best practices in narrative interviews, ultimately enhancing the AI’s ability to facilitate meaningful conversations and engage with clients more effectively. By integrating their expertise into the prompt engineering process, social workers can ensure that AI tools are technically proficient and aligned with their profession's core values.

2. Evaluation tools: Next, social workers may be able to help develop evaluation tools for practical chatbots that are easy to use in everyday social work settings. Traditional evaluation methods often focus on technical performance and rely on complex algorithms or data analytics (Deriu et al., 2021; Finch & Choi, 2020; Lee et al., 2023), which can be overly complicated and not well-suited for use by social workers who may not have advanced technical expertise. For example, traditional metrics might assess an AI’s accuracy in parsing language or its processing speed, but these measures do not capture the qualitative aspects crucial in social work, such as empathy and rapport-building.

Social workers, however, can develop straightforward assessment tools that are easily accessible to lay users, allowing for a more direct and immediate evaluation of AI performance. For instance, simple rubrics used to assess social work trainees could be adapted to include criteria that reflect the qualitative aspects of AI-led narrative interviews, such as the AI’s ability to build rapport, respond empathetically, and maintain a coherent conversation. These customized rubrics would ensure that the evaluation of AI tools aligns with the unique demands of real-world practice, making the integration of AI into social work more effective and relevant. By focusing on these qualitative measures, social workers can ensure that AI technologies are used in ways that genuinely enhance their practice and support their clients.

3. Interview practice through AI collaboration: Customizable generative AI enables social work practitioners to develop interview practices tailored to specific contexts. Instead of simply using preset AI products, social workers can actively participate in designing and utilizing these AI tools, applying their expertise to create solutions more aligned with their clients' unique needs. This approach also opens up the possibility of blended interview practices, such as using AI to facilitate broad intake and engagement and then transitioning to human-led conversations for more nuanced discussions. Our experience with AI-assisted narrative interviews underscores the potential for creating adaptable AI chatbots that can be customized and refined rather than relying on standardized chatbot products that may not fully meet the needs of social work practice.

In our study, we employed a specific framework to guide our research. Our findings suggest that there is transferrable knowledge and skills that practitioners and researchers can use to create or adapt structured steps tailored to various objectives, such as conducting assessments, handling service inquiries, or performing different evaluations. To fully leverage these opportunities, social workers may need to expand their understanding and skills in working with AI, actively participating in designing and implementing intervention processes. By taking a proactive role, social workers can use AI as a tool to enhance their practice rather than simply adopting predefined AI applications. Developing AI chatbots that can accommodate diverse conversation plans ensures that technology complements and supports, rather than replaces, human empathy and expertise.

Concluding remarks: Democratizing AI and the domain experts

Our exploration into the use of AI chatbots for narrative interviews is like navigating untested waters at the dynamic intersection of social work and AI technology. While we found potential in using AI to support narrative conversations, this early version of the AI chatbot also has limitations, such as its inability to handle images, voice messages, and consecutive user inputs. However, our voyage is not about reaching a specific port or proving the tool’s effectiveness; it’s about sailing into the unknown and testing new possibilities.

Rather than concentrating on validating the AI’s current capabilities, our exploration aimed to explore the capabilities of customizable AI and challenge the boundaries of social work knowledge. Introducing AI tools into practice presents a variety of considerations that this paper cannot fully address. Instead, we are steering into uncharted waters, advocating for social workers to broaden their skill sets and reimagine their expertise to actively engage with AI technologies.

The progress in democratizing AI comes with a reciprocal dynamic: users are encouraged to learn new skills, while they, in turn, shape the development of AI by posing new demands and expectations. This evolving relationship enhances our understanding of AI affordance and its role in democratizing technology. As noted early in the introductory section, affordance refers to the potential actions users can take with a tool, shaped by its design and how users perceive its utility (Gibson, 1977; Norman, 1999). In this context, the study shows that non-technical professionals can effectively use advanced AI technologies to meet specific needs, demonstrating that the value of AI is not just in its technical features but also in how users creatively adapt it to their circumstances. This broadens the discussion on affordance, highlighting that the usefulness of AI technologies is co-created through user interaction and adaptation to specific contexts.

The study emphasizes the importance of enhancing user-friendliness, particularly in deploying AI chatbots across platforms and managing complex, step-by-step tasks (as noted in Reflective Observation 1). Simplifying these processes would help make AI more accessible and adaptable, promoting wider use across different fields. This sets a specific demand on the chatbot configuration process. As such, the study illustrates the possibility of co-evolution between AI development and professional practice. As professionals provide feedback and shape the technology, AI tools evolve to create new opportunities for innovation. This reciprocal relationship suggests that both technology and professional practices may grow together, continuously influencing each other’s advancement.

In essence, this study serves as the authors’ nautical adventure and invites all social workers and researchers to explore beyond the familiar boundaries of their expertise while remaining grounded in core principles. By embracing this collective endeavor, we may actively influence how AI is integrated into social work, thereby enriching our practice and enhancing client services in an increasingly digital world. As social work and AI evolve, we encourage ongoing research, discussion, and debate to explore new possibilities and shape the profession's future collaboratively.