Developing a measure of participant experience of trials: qualitative study and cognitive testing

Why measure participant experience?

In health contexts, measurement of patient experience is increasingly used in quality improvement,⁹ and a similar approach could be used in trials.¹⁰ Most trials already have extensive platforms to measure participant experience of the intervention (e.g. quality of life and quality of care) but few routinely measure experience of trial participation. If the measurement of participant experience of the trial was routine, it would allow measurement of variation in experience over sites and time, and between groups within a trial. Feedback from participant experience data could support interventions to enhance future participant experience and potentially increase engagement with research.

An example in a health context is the routine assessment of patient experience of primary care.^11,12 Done routinely at scale, experience can be assessed among individual practices, providing policy-makers with extensive data on experience over time and providing practices with data on strengths and weaknesses in their services which could be modified. There could be value in the trials community adopting a similar approach.

Existing measures of participant experience

Our scoping review of self-reported measures of participant experience concluded that there is no existing standardised measure.¹⁰ The review found a number of limitations in the available studies, including the lack of a formal definition of participant experience, and the use of measures without detailed data about their development.

Since 2015, the UK National Institute of Health and Care Research (NIHR) Clinical Research Network¹³ has embedded a measure (the ‘Participant Research Experience Survey’ - PRES) to assess participant experience of studies (including trials) within each local network. The survey uses a standard set of questions, but the survey is delivered within local networks and across multiple studies currently recruiting within that network. This provides a useful snapshot of experience within that local network but does not provide a systematic, comprehensive assessment of individual trials, and is not designed to provide the detail necessary to compare individual trials. Building in routine measurement and feedback of participant experience into all trials would arguably complement this approach.

Our aim was to develop a standardised measure of participant experience in trials through the application of qualitative research, cognitive testing and patient and public involvement and engagement (PPIE).

Study design

The study had two sequential phases, in line with measure development guidance,¹⁴ and the study flow is detailed in Figure 1. Phase 1 involved qualitative interviews to determine broad domains of participant experience and specific item content. Phase 2 involved cognitive testing to explore comprehension of the items. A PPIE advisory group (PAG) supported both phases. The 6 members of the PAG provided input into all stages of the project, as outlined in a strategy. The GRIPP-SF checklist¹⁵ details implementation of the strategy (Table 1). Three PAG members had previously taken part in a trial and four had experience of PPIE work. AD and CP were joint PAG co-ordinators. To guide development, we modified an existing definition of patient experience¹⁶ to make it applicable to trials, and defined participant experience as: (1) the sum of all interactions in the trial (2) shaped by an organisation’s culture (3) which influences participant perceptions (4) across the continuum of research.

Figure 1. Study structure.

In summary, we identified candidate domains of patient experience and candidate items from our previous work,¹⁰ and explored those through a focus group, semi-structured interviews, and PAG meetings. This generated (a) insights about measuring participant experience in trials (b) a preliminary list of candidate domains of participant experience, and candidate items mapped to those domains. These underwent two rounds of cognitive testing.

Phase 1: Measure development

Firstly, a focus group with clinical trials unit staff captured their perspectives on measuring participant experience of trials and their views on domains and items. Secondly, semi-structured interviews with trial professionals and participants explored the same issues. For pragmatic reasons, we limited the sample size to a maximum of 25 interviewees. Separate topic guides for professionals and participants were used incorporating questions and prompts covering trial experience and what the questionnaire might look like with core questions that can be asked of anyone participating in a trial, and to facilitate discussion, we presented candidate domains and items from a predefined list.¹⁰ Topic guides were piloted with members of the PAG who advised that the questions were appropriate. We have uploaded versions of the measure to the OSF repository.²⁶ The unformatted versions show the item content, which can be used with existing trial materials and matched to local formatting and style requirements. The formatted measure is a suggested format for presentation.

Participants

The study used purposive sampling. We recruited professionals who had worked on any type of trial within the previous 12 months, including clinical trials unit directors, principal investigators, trial funders and Clinical Research Network staff. We also recruited current or past (within the last 5 years) adult participants. We sought diversity in trial participation as well as type, age, gender, ethnicity and education. Recruitment sources were a local consent-to-contact database,¹⁷ the Greater Manchester Clinical Research Network and research team contacts using an email from the research team and an advert. Before conducting the research, the researcher and the Director of the Trials Unit and relevant CRN lead identified professional participants who could participate in the focus group by circulating the study information. Trial professional recruitment sources were aware of the researcher’s interest to improve participant experience in trials. There was no existing relationship with previous trial participants; people responded to the advert directly to the research team with no prior knowledge of the researcher’s interest.

Data collection

One focus group (n=6) was conducted with professionals by author CP in April 2018. Individual interviews with professionals were conducted in-person at the Trial Unit in the UK or via the telephone (n=10) at the University of Manchester by CP between April and July 2018. Trial participant interviews were also conducted in-person (n=10) or by telephone (n=4) by the lead author, NS, between May and July 2018. Each participant took part in one interview or focus group for approximately one-hour; only the participant/s and researcher were present. Both CP and NS are experienced, female, health services researchers, with PhDs, employed as Research Associates at the time of the study. In-person interviews took place in a convenient and private location, with travel expenses reimbursed and a £10 shopping voucher given as thanks. Of note, no participants refused to participate at interview.

Table 2 shows the focus group and interview participant characteristics for the study. Of the 11 trial professional participants interviewed, 36% were staff at the CRN, 27% were clinical trials unit directors, 18% were principal investigators, and 18% were trial funders. Of the 6 participants in the focus group, the majority were Trial Managers (n = 2) followed by a Clinical Trials Director, a Statistician, a Quality Assurance Trialist, and a Clinical Research Fellow. Of the 14 trial participants interviewed, 64% were male, with age range, 30-78 years, with 50% reporting a postgraduate qualification and 64% White English ethnicity.

Analysis

Interviews were recorded, field notes were made, and transcribed verbatim, then uploaded to NVivo (version 11) for coding and analysis by NS and CP. Research team members read early transcripts, suggesting codes, and offering avenues to explore with participants. PAG members each read one professional and one participant transcript to add their insights. Of note, transcripts were not returned to participants for comment/feedback and/or correction. Transcripts from trial participant interviews were analysed thematically by NS.¹⁸ Themes were generated, coded and categorized according to the developing framework. A document listing the codes from each transcript with excerpts of data relevant to each theme was developed to manage the developing framework. Data from trial professionals were coded separately from participant data by CP, and then compared. This and our previous work¹⁰ informed the drafting of the measure to be tested by cognitive testing (consisting of 9 candidate domains and 52 items).

Phase 2: Cognitive testing

Cognitive testing is a method used to assess the performance of a measure by collecting information about respondents’ thought processes, as they answer questions, using ‘think aloud’ techniques to inform adaptation of the measure.¹⁹

Participants

We invited trial participants from phase 1 and new participants using the same recruitment methods (via email and advert) and purposive sampling. There is no consensus around sample size for cognitive testing, as this depends on the type of cognitive process examined and the number of ‘rounds’ necessary to adjust the measure.²⁰ Here, a round was considered complete when clear problems were identified with items warranting adjustments to the items and measure.

Data collection

The study was conducted within the Centre for Primary Care and Health Services Research at The University of Manchester, UK. Cognitive interviews took place in August 2018 in a private room for approximately one hour with only the participant and researcher, NS, present. Interviews were audio-recorded, with note taking, but not transcribed. As before, participants received travel expenses and a shopping voucher.

NS received training on cognitive testing ahead of the study. The topic guide included a full explanation of the ‘think aloud’ task and open-ended questions and probes (spontaneous and pre-prepared) to assess different cognitive processes: comprehension; recall; judgement; and response.²¹ We opted for an ‘immediate retrospective think aloud’ approach whereby we asked participants to read each item while thinking aloud and to respond to each item. After doing a review of the topic guide, the research team and PAG judged that the questions and topic areas were appropriate.

Cognitive testing took place over two rounds (n= 13 and n = 6) where the results of round 1 informed round 2, with each respondent participating once. No participants refused to participate at interview. Of the 19 participants, 63% were male, with age range, 40–86 years, with 58% reporting a postgraduate qualification and 74% previous trial experience (see Table 3).

Analysis

For data analysis, we followed the three-stage process in a modified framework analysis,²⁰ involving: chronicling, condensing and using the data to improve the questionnaire; NS prepared the analysis. PAG members each read cognitive testing guidance prepared by NS to add insights to the analysis at the meeting to refine the items. The research team and PAG made decisions on changes, including changes to items, the items contained within each domain, the domain name (to reflect the revised content), the response scale and measure layout. We followed the published reporting framework for cognitive testing.²²

Ethical considerations

The study received Proportionate University Research Ethics Committee approval (Ref: 2018-2739-5920). Participants who expressed an interest in participating in the study via the advert were emailed a full Participant Information Sheet outlining study involvement and had the opportunity to consider the information and ask questions ahead of consenting to be interviewed. Those who agreed to participate gave written informed consent ahead of the interview, including how data collected would remain confidential; the use of the audio recording; any data collected may be archived and used as anonymous data as part of subsequent research (also known as secondary data analysis); and the use of anonymous quotes in academic books, reports or journals.

Phase 1: Measure development

We report themes and illustrative quotes from interviews and focus group participants, and insights from the PAG (each quote is tagged to indicate method of data collection and the participant’s role).

(a) Advantages and disadvantages of measuring participant experience

All participants mentioned that they thought the standardised measurement of participant experience had potential to improve trial experience.

Whether this benefit was realised as the trial was running or in future trials was discussed. Some interviewees described the potential to use the experience data, if it highlighted the benefits of participation, to market future trials to potential participants and to promote trials generally.

The main concern raised from trial professional interviewees centred around burden to participants, particularly in situations that were stressful.

The majority of trial interviewees focused on feasibility, in terms of when would be the easiest to do it for the participant and the usefulness of the experience data captured.

The interplay between the intervention, perception of benefit, and impact of receiving trial results was also a concern expressed by interviewees.

(b) Domains to be included in the measure

Candidate domains were identified by the research team for inclusion, relating specifically to the different phases in the trial (which the PAG described as ‘the participant journey’).

(b1) Early stage information provision and trial processes

Professionals were concerned that recall would make assessing information and processes occurring early on in the trial problematic if the measure was administered at the end of the trial. The majority of interviewees saw early stage processes as important to assess.

Trial participants spoke of having too much information with complicated text, and wanted specific information to manage their expectations of participation.

(b2) Perception of conduct of trial processes

How a trial was conducted and the practical issues (e.g., waiting times and travel) was seen as important by all interviewees:

(b3) Sharing trial results with participants and perception of trial processes

There was discussion around whether measures should be administered before or after the sharing of trial results, and frustration described by participants when results were not shared by the trial.

(b4) Engagement with trial team

This domain relates to the communication between the team and the participant across the trial journey and the extent to which participants ‘feel a part’ of the trial and have confidence and trust in staff. Having a relationship with the trial team was viewed as vital to participants.

(b5) Perceived benefits of participation for participants

This theme specifically relates to the perceived benefits of participation other than satisfaction to increase understanding of why participation in trials is important.

(b6) Perceived satisfaction with participation

Interviewees discussed the utility of a question on overall satisfaction. Some interviewees and PAG members felt that it would be preferable to ask whether a participant would take part in the same trial again compared to asking about taking part in research more generally.

Some trial professional interviewees referenced the assessment of patient experience of clinical care, which has become a mainstay of quality improvement in the NHS.

(c) Logistics of administering an experience measure

Research participants and PAG members raised a number of logistical considerations around delivering experience measures. These are summarised as a checklist (Table 4) with a brief summary presented here. The majority of interviewees felt that measuring experience in participants who are no longer in the trial (i.e. lost to follow-up or withdrawn) would provide important data on patient experience. The consensus was that the questionnaire needed to be easy to read and short (between 1-4, A4 sides, and take between 3-10 minutes to complete). A variety of modes were suggested (self-complete questionnaire: paper, electronic via a website or app). Different time-points were suggested by interviewees and can be broadly categorised as ‘during the trial’ or ‘end of trial’. There was no consensus as to which option was best, but flexibility was seen as key.

Phase 2: Cognitive testing

The cognitive testing highlighted issues around comprehension, recall and response to the candidate items. In summary, from 2 rounds of cognitive testing, we removed 30 items from the measure. We refined 46 items using respondents’ preferred terms to provide clarity and inserted examples grounded in participants’ experiences in 9 of these items, and 2 items were changed from statements to questions. The response options were enhanced to ensure the measure captured a range of trial experiences: binary responses were inserted in 9 items; graded responses were retained in 14 items; a ‘not applicable’ option was added to 6 response scales, a ‘no opinion’ to 4 scales; and free text was added to 4 items. Figure 1 summarises the development process of the measure, showing the domain and item refinement process throughout phase 1 and phase 2. 3 of 9 candidate domains were amalgamated and 3 were re-labelled.

By the end of the cognitive testing phase, consensus was reached among the research team that the final 6 core domains, and 25 items (including one final ‘full text’ response) captured a ‘meaningful’ trial participant experience. The short measure was subsequently developed based on those items viewed as preferable by respondents, with 1-9 items tapping each of the 6 domains, totalling 17 items. We have uploaded versions of the measure to the OSF repository.

Strengths and limitations

There are limitations to the study. First, we did not include participants in either the phase 1 or 2 sample who had not completed a trial they had been recruited to, either because they had withdrawn or were lost to follow-up. Our sample might be biased towards trial participants who may have had a more positive experience, although there were instances in the data where trial participants reported otherwise.

Most of our participants were white (88%) indicating the need for further work to explore under-represented groups. This supports wider initiatives underway to support trialists to design and conduct inclusive trials.⁷ We did not formerly assess the readability of the items instead relying on a proxy indicator of educational level for health literacy, and we were also less successful in recruiting a diverse group of participants. The PAG provided an important opportunity to integrate patient insights into development, but all were known to the team through existing networks and had interests in the research topic.

The data were collected over 4 years ago, with publication delayed by the pandemic and other issues. Although we expect that many of the issues remain pertinent, changes in the delivery of trials, such as moves to more remote methods,²³ would not have been captured.

Implications

The measurement of trial participant experience may be important for improving delivery of trials, exploring variation in experience in different sites and over time, and between groups within a trial. However, this will require feedback of the data and resulting change. Effective feedback processes (such as, guidance on how to interpret the results of the measure) needs to be co-produced with stakeholders to ensure it is acceptable and useful.²⁴ It will also be important to assess whether potential disadvantages of participation are realised.

Our measure has been piloted with three ongoing trials in the UK, with results reported in a forthcoming paper. Future work will also need to examine the factors that influence participant experience, and how much of the variation in experience is due to context (for example, condition experienced), trial type, participant characteristics, or aspects of the trial (for example, the intervention and trial procedures). It will also be important to explore the extent to which experience is relevant in all trials. For example, some trials have little active participation either because of the trial design (such as, cluster trials without individual consent) or because of the duration and extent of follow-up. We do not foresee any challenges translating the measure to a digital platform given there are substantial precedents for capturing patient experience and study experience data digitally.²⁵ We report a feasibility study of the use of the measure elsewhere.