Improving a trial participant information sheet through user testing in British South Asian women

Aim

The study aimed to test whether finding and understanding of key trial information could be improved using user testing methods.

Study design

Randomised trials are rightly considered the most robust method of assessing health care interventions. However, they are often not possible due to practical and logistical barriers, such as the high cost of such research. A non-randomised independent between-groups design was chosen, with participants in each group seeing only one version of the PIS.

In a non-randomised between-groups design, each participant was exposed to only one version of the PIS. One group reviewed the original PIS, and the other group reviewed the revised PIS. The advantage is, considering the target population; those participants enter the study fresh and naïve regarding procedures, avoiding practice effects. Each participant can only be tested once, as the knowledge gained during the 1st round of testing would prevent a fair test of the 2nd round of information (Bellemare et al., 2018).

Participants

The ROSHNI-2 trial aimed to evaluate the clinical- and cost-effectiveness of a culturally adapted group psychological intervention (Positive Health Programme, PHP) in primary care for British South Asian women with postnatal depression compared with treatment as usual. The current study aimed to apply the user testing methodology to the ROSHNI-2 PIS to test whether a revised PIS makes it more likely that British South Asian women participants can find and understand key information.

Inclusion and exclusion criteria are shown in Table 2.

Sampling

A purposive sampling strategy was used to select participant characteristics matched to the ROSHNI-2 trial cohorts in terms of gender, age, and education. In addition, as the study did not use random allocation, the aim was to ensure that the participants’ characteristics were similar in the two independent groups.

Sample size

There was no formal sample size calculation for the study. Based on data from similar research using user testing methods (Table 1), a sample size of 20-40 participants was recommended (Knapp et al., 2009c), and the current research sought a sample size of thirty participants (15 in each group).

Recruitment

Initially, the recruitment process involved an introductory meeting with various community organisations and mosques in the Manchester area. The aim was to raise awareness and promote the research. Recruitment took place from one community organisation and two mosques.

Data collection

User testing was conducted on the ROSHNI-2 PIS. User testing data was collected by participants answering 19 key questions from the PIS. Table 3 shows the list of key questions. The testing of the original and revised PIS was undertaken in two languages (English or Urdu). In response to each question, the participants were asked to find where the answer was located, then give the answer (using their own words when possible). If participants were not able to locate the answer, it was recorded as not found and not understood (because real trial participants can only be informed by content they can find in the document).

The times taken to read the PIS (‘reading time’) and complete all 19 user testing questions (‘comprehension time’) were recorded. Pilot user testing assessments were conducted with two community members before the first round of testing and no changes made to the study.

Outcomes

The user testing explored participants' ability to find and understand 19 key points of information in the PIS (Table 3). The 19 items were drawn from the four categories of information from previous PIS user testing research (Knapp et al., 2011):

Procedure for testing original, revised ROSHNI-2 PIS and interviews

The following procedures were in line with steps adopted in similar research (Knapp et al., 2011). Figure 1 shows the steps involved in the user testing of ROSHNI-2 original and revised PIS.

Figure 1. User testing of ROSHNI-2 original and revised participant information sheet (PIS).

PIS

ROSHNI-2

The ROSHNI-2 PIS comprised three single-sided A4 pages. The revised ROSHNI-2 PIS retained its meaning but with a revised layout, appearance, structure, and wording which was informed by the results from the first round of testing. Refer to Figures 2 and 3 for example sections of the PIS.

Figure 2. Original ROSHNI-2 participant information sheet (PIS) section.

Figure 3. Revised ROSHNI-2 participant information sheet (PIS).

Revision of ROSHNI-2 original PIS

YM followed good practice guidelines for translating and adapting the ROSHNI-2 PIS (Hall et al., 2018). Figure 4 provides an overview of the steps involved in the adaption process.

Figure 4. Revision of ROSHNI-2 participant information sheet (PIS) based on seven steps.

Revisions to the Revised ROSHNI-2 PIS

Revisions to the ROSHNI-2 PIS included dividing the pages into two columns, adding a brief table of contents to the front page, and emphasising the contact details and placing them on both the first and last pages. The main text of the information sheet was divided into various headings and bullet points were used. Questions were reworded and the text was arranged into smaller sections with shortened sentences and paragraphs. The revisions included: use of lay language; replacing difficult or technical words with a lay alternative; and use of flow diagram to illustrate various stages of the trial. Colours were used for the section headings and the flow diagram. The flow diagram was used to illustrate various stages of the trial.

Translation of ROSHNI-2 PIS

For the translation of the ROSHNI 2 PIS, it was decided to select Urdu. It is native to South Asia, in particular the subcontinent, the national language for Pakistan and the official language of several Indian states and the Jammu and Kashmir areas, and is spoken in all by more than 145 million people. (Lewis et al., 2016). Within the UK Urdu is spoken by over 270,000 individuals, making it the 4th most spoken language. The majority of these individuals are from ethnic minorities, particularly of South Asian descent (Lewis et al., 2016).

YM and WW were also bilingual, speaking both English and Urdu. The original ROSHNI-2 PIS was initially screened and reviewed in detail in the presence of WW, the technical wording about study concepts, definitions, processes, and methodology was marked and highlighted. A consensus was reached to adapt the marked terms in easy and understandable English language. WW and YM felt the terms used in the PISs did not conform to the British South Asian communities’ level of understanding of the research concepts.

A glossary of all the English terms and concepts was developed by YM, the technical terms and concepts were replaced by easy wording and explanations. The glossary of terms was based on explanations taken from other glossaries such as National Institute for Health Research (NIHR) dictionaries (Holosko & Thyer, 2014) and the wealth of expertise in translating and adapting for ethnic communities that WW has. To cater for the English-speaking lay British South Asian communities’ level of understanding, WW and YM decided to make changes to jargon and technical concepts. Therefore, specific language changes were applied to the English wording and terms.

Later, Urdu translation of information was carried out by YM and WW. YM translated the conceptual equivalent of a phrase, rather than a word-for-word translation, whilst considering the target audience. The language was kept simple, clear, and concise by avoiding long and complex sentences. Specialised terms and jargon were avoided. Back translation was carried out om the initial translation by MN. Later, two native speakers of the language provided feedback in developing the translated version and patient and public involvement (PPI) regarding formatting issues and language suitability was undertaken.

Data analysis

Comprehension was based on the participant's ability to find each of the 19 key questions in the information sheet and be able to convey an understanding of each of those points. The criterion of EU legislation on testing of medicine leaflet was used, which refers to questions found and understood by at least 80% of participants (Medicines and Healthcare products Regulatory Agency, 2005).

Stata statistical software was used to analyse the data (REF needed). Student t-tests were performed to test any difference in the reading and comprehension times for participants reading the original and revised PIS. Further chi-square tests were performed for found and understood data on individual questions on original and revised ROSHNI-2 PISs.

PPI

PPI was used at the stage of developing and translating the PIS for the ROSHNI-2 study. The PPI group gave suggestions to improve the layout and also suggested formatting preferences.

Ethical approval

The study received retrospective ethical approval following a review by the University of Manchester Research and Ethics Committee on 02 November 2018. All the participants provided written informed consent.

Demographic data

The original ROSHNI-2 PIS was tested on group A with 15 British South Asian women, who had a mean age of 37 years and were Pakistani (n = 14) or Bangladeshi (n = 1). The majority of the women completed education at the graduate level and required English materials.

The revised ROSHNI-2 PIS was user tested on group B (16 Pakistani women with a mean age of 35 years). The majority completed education at the graduate level and required English materials.

Table 4 presents the demographic information relating to groups A, B and the ROSHNI-2 study. The groups were broadly similar in age, education, and use of English or Urdu versions of the PIS.

Comprehension time and reading

Reading and comprehension times for the two groups are given in Table 5. Reading time was time taken to read the PIS whereas comprehension time was time taken to answer questions.

Overall, the time taken to read and comprehend 19 questions was less in the revised PIS when compared to the original PIS. There was a significant difference in reading time between the original PIS and the revised PIS (mean difference 3.84; t (29) 3.00 = 0.006; p < 0.05). Furthermore, there was a significant difference in comprehension time between the original PIS and the revised PIS (mean difference 3.70; t (29) 2.21 = 0.04; p < 0.05).

In the original PIS, questions under the categories of nature and purpose, trial procedure and safety and confidentiality had significant problems. Among the 19 questions, 2 questions (Q2, Q5) were understood by 80% of participants while 5 questions (Q4, Q8, Q16, Q1, Q9) were understood by 53-73% of participants. The rest of the questions were understood by fewer than 53% of the participants. Table 6 shows the results from the original and revised PISs.

Based on the findings from the checklist questions, testing, locating, and understanding answers to the questions, the time taken to answer questions, and participants' evaluative comments all indicated that the ROSHNI-2 original PIS was not performing well. Participants were not able to understand the content of the sheet and follow the information accurately.

The revised ROSHNI-2 PIS showed improvement. Questions under the categories of nature and purpose, trial procedure questions, and safety and confidentiality questions showed significant improvement. Among the 19 questions, nine (Q14, Q8, Q3, Q7, Q10, Q12, Q19, Q11, Q17) showed significant improvement at p < 0.05 (Table 6).

Summary of the findings

User testing of the original ROSHNI-2 trial PIS showed that it underperformed. Participants found it hard to find information, and even when information was found, it was not always understood. Overall, there was a lack of understanding about important aspects of the trial, including what is the trial trying to find out, what are the two different treatments, how will it be decided which patient in the trial receives which treatment, the number of follow-up visits and locating the emergency contact number. The findings are in line with previous user testing studies of trial information sheets (Table 1).

Furthermore, the study identified other issues such as the complexity of the structure and organisation of the original information, and the coherence of the information presented. The use of scientific terms and presentation of information in lengthy paragraphs led to less engagement and poor understanding of the study concepts. Also, findings reported that the original PIS had excessive wording with no apparent sequence in the flow of information.

Also, in the revised sheet, participant results reported that locating and understanding information was easier, and it was found that the time taken to read the sheet and then complete the checklist questionnaire was much shorter. The results emphasised the use of non-scientific language, improved structure, and better presentation of the PIS. These findings reinforce the existing view to consider language aspect as members of minority groups find difficulty in distinguishing clinical treatment from research (Dein, 2006).

Strengths and limitations

This is the first study to test understanding of a mental health trial PIS in ethnic minorities. Earlier research in this area utilised user testing on physical health conditions in the general population (Knapp et al., 2009a, 2009c, 2011).

As the current study was not recruiting participants for the ROSHNI-2 trial, several aspects of the study significantly differed from the actual recruitment situation. The participants were members of the public asked to imagine taking part in the ROSHNI-2 trial. They read the PIS at home before the testing appointment, and the user testing interviews took place in a quiet interview room rather than a busy clinic. Though these factors may have, to some extent, facilitated participants in their reading and understanding, it is unlikely to have biased the comparison of the original and revised PIS (Knapp et al., 2011). The demographic characteristics of the current study broadly matched the demographic characteristics of the ROSHNI-2 participants.

Although the analysis highlighted improvements in participants’ comprehension levels in the revised PIS, participants were not randomised. In relation to differences between groups A and B, education levels showed differences, with more participants at the graduate level in group B when compared to group A. Also, there were more postgraduate level participants in group A when compared to group B. It’s important to highlight that despite both groups being at least A-level educated, they also struggled to understand, showing a modest difference in improvement. Furthermore, there may have been other group differences that the study did not measure which also might have contributed to the difference in comprehension level found in the revised PIS.

There are reports that ethnic minority patients have considerable trust in their GPs and prefer to receive study-related information from them (Symonds et al., 2012) or other clinicians (D K Raynor et al., 2007). In contrast to an actual trial, where there is a recruiting clinician who assists participants with spoken and written information, in this study participants were asked to show an understanding of the ROSHNI-2 trial from the PIS alone. It is unclear whether the presence of a clinician would have impacted on the understanding of the PIS; this should be investigated in further research (Knapp et al., 2011).

The user testing method was tested in a study sample with a majority of participants at a graduate level, with a few at the high school level, while none of the participants had any formal research experience. Participants' level of education must have contributed to some level of understating the concepts, however overall understanding within this group was poor and therefore there is a need to assess this method with an ethnic minority sample where literacy in English is an issue.

Furthermore, the sample mainly comprised of Pakistani women, with only one participant from a Bangladeshi background. Future research requires testing the method in a more diverse British South Asian ethnic population.

At the time of testing participants, YM judged if the participant demonstrated understanding of the answers by cross-questioning participants. As YM was the only researcher collecting data and was not blinded to study aims, therefore there could be an aspect of bias influencing the data collection process.

Reporting the results of the individual items meant multiple statistical tests, which needs to be taken into account when interpreting p values, as multiple comparisons increase the likelihood that an individual value will be outside a 95% confidence limit or that the p-value for a single comparison will be less than 0.05 (Feise, 2002).

The user testing results from the participants showed that some questions improved more than others between the original and revised PISs. The questions linked to the nature of the trial, the process of taking part in the trial, management of treatment, and follow-up-based questions showed significant improvement. However, changes made did not affect user testing performance in some questions. Questions linked to interventions, and management of information did not show significant improvements, and further research is needed to seek greater understanding of these concepts.

A point to highlight is that user testing was used in this study as both a developmental tool and then an outcome measure with a small number of participants. Its use aimed to indicate problems with information, and it gives an indicator of when those problems have been fixed. In the current study, sample sizes of 15 per group are small and there can be a chance of Type 2 error.

Comparing the findings to the existing literature

The present finding confirms that the revision of the sheet by user testing with lay participants, through rewording, redesigning, and restructuring while retaining the meaning showed much-improved performance in almost all areas of the sheet. The results confirm the pattern reported in the studies reported in Table 1.

In terms of design, three studies used a stronger design (a randomised control trial [RCT] and a SWAT) whereas all other studies employed an independent group design and SWAT design (Knapp et al., 2011; Cockayne et al., 2017; Parker et al., 2018). In non-trials, there is a risk of participant variability (individual differences between groups) affecting the results, rather than solely manipulation of the independent variable. The sample size of the current study was lower compared to those conducted previously. Comparing the outcomes of the studies is difficult, as each study reported outcomes differently, although the general direction of the effect was consistent with a higher proportion of respondents answering questions correctly after user testing. As well as providing support for the general use of user testing, the current study is the first to evaluate user testing in a mental health setting, as the other studies recruited from a variety of physical health conditions.

Previously studies have reported that a shorter PIS led to a greater understanding (Knapp et al., 2011). However, the present study highlights that restructuring and changing the layout of information contributed to more effective engagement with the PIS.

Studies on the design of written medicine information have emphasised a need to involve users' perspectives in producing information for public usage (Lentz & De Jong, 1997; Raynor & Dickinson, 2009). The involvement of ethnic minorities in the development of trial materials may result in significant improvements to the information sheets, leading to better understanding and increased ethnic minority participation in trials. The present study supports involving users in the process of optimising and amending PISs. The study highlighted the problem areas that ethnic minority participants face when trying to read and understand trial-related information. User testing can help assure that the PIS can meet the needs of ethnic minority communities in understanding study-related information.

Policy and ethics implications

Modern research ethics committees often request highly detailed PISs, which may perversely reduce the quality of the consent decision by discouraging thorough reading and reducing assimilation of information. The current findings support this observation (Shukla et al., 2012).

It has been observed that in PISs there is more emphasis on the legal issues as compared to the communication aspect, as a result of this the length and complexity of the information sheets has increased (Fortun et al., 2008). Principal investigators and research ethics committees must balance legal and comprehensibility issues. In our study the most criticised aspect of the sheet was the structure of the information presented, and the complexity of the information presented. Other investigators have found that attempts to simplify PISs are rejected because of concerns of how text alterations might affect the legality of the statements (Koops & Lindley, 2002). The research ethics committee may review the standardised procedure for reviewing sheets, and this has implications for ethnic minority participant recruitment. Principal investigators and research ethics committees must consider other aspects to improve the readability and comprehensibility aspects of the information sheets without compromising on the length and legal aspects (Cockayne et al., 2015; Knapp et al., 2011).

Future work and implication

User testing methods may contribute to the recruitment of ethnic minority patients and the smooth running of the trial, with participants having a more complete understanding of the study (Hanley et al., 2001). Finally, it was not possible to evaluate the impact of revised ROSHNI-2 information sheets on the recruitment rate. However, the effect on actual trial participants is an important question. Further testing would require a study comparing the original and revised PIS with the random allocation of participants to test the effectiveness of the revised version on recruitment rates (Cockayne et al., 2015; Knapp et al., 2009b).

The Health Research Authority (HRA) in the UK has guidance on drafting PISs, the study highlights room for improvement in the guidelines, specifically addressing the needs of ethnic minority participants (Raynor et al., 2007).