Exploring the role of ChatGPT in rapid intervention text development

Introduction

ChatGPT is an artificial intelligence (AI) chatbot, launched by OpenAI in November 2022. It is capable of generating responses to prompts or questions, input by the user, based on OpenAI’s Generative Pre-Trained Transformer (GPT) language model (launched in 2018) (chat.openai.com). Since its launch it has gained significant attention and discussion of the ways in which it might be used across various sectors.

It has been suggested that generative AI, such as ChatGPT, can be used in the generation of content with the purpose of providing information on public health issues, answering questions about health promotion and disease prevention strategies, explaining the role of community health workers and health educators, discussing the impact of social and environmental factors on community health, and providing information about community health programs and services (Biswas, 2023).

The potential of generative AI to support healthcare has been investigated in a range of topic areas. In a review of 60 papers, there was evidence that ChatGPT could be beneficial in supporting scientific writing, research activity (e.g. analysis of large datasets), medical education and healthcare practice (Sallam, 2023). With regards to writing, there were reported benefits in the speed of conducting literature reviews and improved language to communicate ideas, providing information that is easily accessible and understandable. In another systematic review of 31 studies, ChatGPT was effective at generating information on a range of health topics (Muftić et al., 2023). However both reviews highlight potential limitations such as generation of inaccurate information, and limited knowledge on topics or research after 2021. In some cases ChatGPT produced overly detailed content, but this may be addressed by using clear and specific prompts from the user about the desired output (Sallam, 2023).

There is mixed evidence on the quality of health-related information generated by generative AI (with the majority of research using ChatGPT). There is some evidence that AI-generated responses from ChatGPT are outperforming human responses in terms of their quality and empathy. Ayers et al. (2023) asked licensed healthcare professionals to evaluate physician responses and AI responses to health questions from a public social media forum. Responses were rated in terms of the quality of the information and the empathy provided. AI responses were preferred 78.6% of the time and were rated significantly better than physician responses with regards to quality and empathy, suggesting AI may be useful in generating public-facing health information. Grünebaum et al. (2023) reported a high level of accuracy in answering questions about obstetrics and gynaecology, while Barat et al. (2023) report only a 40% accuracy rating for responses to questions about interventional radiology.

Fewer studies have explored the utility in generative AI summarising and optimising text. ChatGPT was effective in summarising text based on palliative critical care scenarios written by human experts (Almazyad et al., 2023). Cascella et al. (2023) argue that while ChatGPT can generate and summarise public health text effectively, this is dependent on the information the researcher inputs, and it did not always adhere to strict word limits.

The Coronavirus disease 2019 (COVID-19) pandemic has highlighted the need to extend existing best practice in intervention development so that interventions can be co-developed and evaluated quickly in response to a rapidly-changing public health concern (Yardley et al., 2023). In an effort to address this issue, the Agile Co-Production and Evaluation (ACE) Framework was proposed (Yardley et al., 2023). The framework is intended to provide a focus for investigating new ways of rapidly developing effective interventions by combining co-production methods with large-scale testing and evaluation. This framework builds upon the Person-Based Approach (PBA) to intervention development (Yardley et al., 2015). This approach to intervention development is grounded in the idea that co-production with an intervention’s intended users is vital in ensuring the techniques and messaging are appropriate and more likely to be effective.

In this paper, we consider and explore the potential of generative AI (using ChatGPT) in supporting the rapid development and optimisation of interventions. We present three case studies in which ChatGPT was used to support intervention development. This includes using ChatGPT to generate information for a health intervention, in order to evaluate its ability to rapidly produce intervention materials. Following this, we explore the ability of ChatGPT to optimise text that was generated by researchers involved in intervention development. In these two latter case studies, we asked patient and public involvement (PPI) contributors for their views on the outputs generated by ChatGPT.

Methods

In a series of case studies, we asked ChatGPT to generate or simplify text relating to health interventions. Text to be generated or simplified for these case studies was selected from studies being undertaken by the Behavioural Interventions Group. These studies were in the process of developing public-facing intervention content using the PBA or ACE (Yardley et al., 2015, 2023). This approach to intervention development ensures the developed intervention is grounded in theory, evidence and views of the intended users. It does this through the use of theoretical mapping (i.e. mapping techniques and ideas against behaviour change and psychological theories), ‘guiding principles’ for how the intervention should be developed, and a logic model to hypothesise how the intervention may work.

In the first case study, ChatGPT was asked to generate intervention content to improve sleep in teenagers (‘Sleep Solved’). We constructed a prompt for ChatGPT: “How would you improve sleep in teenagers? Write your answer in simple language for someone with a reading age of 9 and use bullet points.” We then evaluated the effects of providing a more elaborate request, to encourage ChatGPT to use the ‘programme theory’ (Skivington et al., 2021) for the intervention to guide the text development in the same way as the researchers were doing. The elements of the programme theory that ChatGPT was additionally asked to follow were the ‘guiding principles’ (Pollet et al., 2020): to avoid being patronising or paternalistic; to use question-answer formats to make the advice accessible and engaging; and to make the advice convincing by giving the scientific explanations for advice, but keeping the scientific explanations very brief and simple. In accordance with the behavioural theory informing the content of Sleep Solved, ChatGPT was also asked to base the advice on Bandura’s Social Cognitive Theory (Bandura, 1977), with the aim of building users’ self-efficacy (confidence) that they can follow the advice, and positive outcome expectations regarding the consequences of following the advice.

In the remaining case studies, ChatGPT was asked to optimise text. In case study two, the text was written by a behavioural scientist from the Behavioural Interventions Group who was involved in the development of an intervention to support people experiencing homelessness to stay safe during extreme weather events. In case study three, the text was from an information sheet used in the recruitment of study participants in research conducted by the Behavioural Interventions Group, aiming to co-produce an intervention for post-partum blood pressure management. This text was produced by a study team outside of the Behavioural Interventions Group.

In order to maintain consistency for comparison across the case studies, ChatGPT was provided with the same wording in all three case studies to request accessible text: “make this text suitable for a reading age of 9 and use bullet points”. Each identically-worded request (with different example text to be simplified) was conducted twice, each output compared, and the best output was selected by the researcher.

OpenAI has recently upgraded ChatGPT by using the GPT-4 model. This requires a paid subscription to ChatGPT Plus. Case study two used ChatGPT using GPT-3, whereas case study one and case study three were conducted in ChatGPT using both GPT-3 and GPT-4. In these two case studies, identical prompts were input into the two versions of ChatGPT (i.e. ChatGPT using GPT-3 and ChatGPT using GPT-4) and outputs from each version of ChatGPT were compared.

The outputs from all case studies were evaluated by the researchers developing the intervention text through reflecting on the output compared with the guiding principles, planning tables and text written by the researcher. In case studies two and three feedback was also sought from PPI members of the team and service providers. PPI members were invited to comment on the text produced by ChatGPT as part of their ongoing role within the research teams developing the interventions. Ethical approval is not required for the involvement of public and patient contributors in research.

Results

Discussion

Across three case studies, we have explored how ChatGPT might be used to support rapid intervention development. In our first case study, when prompted to produce advice on sleep for adolescents, ChatGPT’s output undoubtedly had face validity and value; the advice it gave reflected scientific consensus and was succinct and accessible. Indeed, a fourth case study that was planned for this paper was not pursued because the intervention was being developed alongside rather than before this study, and the developer found the text suggested by ChatGPT so helpful that she was unable to create a version that was not influenced by it. However, the ChatGPT text did not closely follow the ‘guiding principles’ for how to make the text engaging that it was asked to apply, and did not include any of the key behaviour change techniques that were developed as part of the co-production process using the PBA. In our second case study, key pieces of advice that were generated as a result of co-production with intended users of the intervention were omitted. In this case study, members of the public (specifically, people experiencing homelessness) provided feedback on ChatGPT’s output and highlighted that these omissions needed to be included in the advice. Our third case study showed that, when prompted to optimise an entire participant information sheet, ChaptGPT (using GPT-3) omitted information that would be necessary to include and the reading level of the text was higher than requested by the prompt. However, when text was entered in smaller chunks without bullet points, the output was improved and the text was considered superior to the original by both PPI and the research team.

We have learned from our case studies that despite the speed with which ChatGPT can provide text, using ChatGPT can be surprisingly time-consuming. Texts requiring optimisation may need preparation prior to being input into ChatGPT (using GPT-4), for example by dividing into short simple sections and simplifying technical terms. ChatGPT using GPT-4 seemed to perform better, and outputs could be improved further by responding to ChatGPT’s outputs with further requests for changes and by giving examples of the required tone and suitable outputs. For instance, generative AI (including ChatGPT) used in health intervention messaging could be prompted to simplify commonly used terms and explain common concepts (e.g. randomisation, anonymisation or informed consent). We faced a steep learning curve in our use of ChatGPT and it is likely that in future researchers will also need to be trained to use generative AI more effectively. Although beyond the scope of this study, it is likely that generative AI can contribute to public health intervention development in other ways. For example, it could be used to rapidly summarise and respond to public reactions to an emergency on social media, and to translate intervention content into multiple languages.

Despite the potential of generative AI to support rapid response during public health emergencies, in its current form, there are significant limitations (see Table 2). Importantly, there is evidence that ChatGPT has problems with accuracy in generating public health information (Biswas, 2023; Jungwirth & Haluza, 2023; Muftić et al., 2023; Sallam, 2023). Furthermore, the use of ChatGPT for arising public health issues is limited due to its language model being trained using a dataset with a cut-off date in 2021 (Jungwirth & Haluza, 2023; Muftić et al., 2023; Sallam, 2023) meaning it is not able to provide up-to-date information about novel public health concerns. In contrast to previous studies (e.g. Muftić et al., 2023; Sallam, 2023), in our three case studies there were no examples of inaccuracies in the text generated by ChatGPT. However the quality of the output (with regards to formatting and language used) was related to the quality of the prompt (Cascella et al., 2023).

Conclusions

ChatGPT (particularly ChatGPT using GPT-4) can produce succinct summaries of text, when this text is prepared by a researcher and entered into ChatGPT in the appropriate format with appropriate context and requests. However, ChatGPT tends to miss key elements that address subtle but important barriers to behaviour change identified through collaboration with intended users and grounded in theory and evidence. While ChatGPT is able to rapidly summarise text to be informative and succinct, it may omit key content grounded in behaviour change theory, which may affect its ability to change behaviour (Yardley et al., 2015). A further problem with AI-generated text is that interventions need to offer users advice and support that seems somewhat novel or unique in order to promote engagement. Because interventions produced using generative AI are based on widely available texts they are unlikely to appear innovative and uniquely informative. Moreover, since the outputs from generative AI are inevitably generalist rather than context-specific there is a risk that they will not be well-adapted to the context of under-represented groups. The outputs of generative AI may thus be able to assist with ‘agile’ content generation but the ‘co-production’ element of the ACE framework will remain crucial to ensure that content is ‘relatable’ and appropriate for the intended users (Yardley et al., 2023).

In summary, generative AI is a novel tool that may be useful in rapid intervention development. There is however a learning curve and as we are learning how to use it; it too is learning how best to respond.