SWAT 137: A 2x2 randomised factorial SWAT of the use of a pen and brief information leaflet to improve recruitment in a randomised controlled trial of a community-based clinic for patients with persistent physical symptoms

Introduction

Despite substantial funds being invested in the UK and internationally, many randomised controlled trials (RCT) in healthcare fail to recruit on time and to budget.¹ Many strategies are used by trialists to improve recruitment; however few such interventions have been rigorously evaluated.^2,3 Trials embedded in real-life ‘host’ trials (also known as ‘Studies within A Trial’ [SWATs]) are the most robust way of evaluating such interventions.

There is some evidence that using a pen as a nonmonetary incentive increases response rates and time to response for trial follow-up questionnaires.^3,4 The primary theory underlying the use of pen incentives is that of reciprocation.^5–8 In the context of trial recruitment, offering a potential participant a gift such as a pen may make the person more likely to take up the trial invitation to enrol. It is also possible that the convenience of having a pen to hand upon receipt of the invitation may increase the likelihood of the forms being completed, or the timeliness of return. One U.S. trial embedded in an observational study showed that including a pen with the study logo to a questionnaire mailed to women who had previously not responded, significantly improved recruitment rates.⁹ However, a SWAT embedded into a RCT testing yoga for older adults with multimorbidity found no improvement in randomisation or response rates with the inclusion of a pen incentive within postal recruitment packs.¹⁰

A common method of recruiting participants from general practices and other registries into trials is to send letters to potentially eligible patients inviting them to participate, along with the trial Participant Information Sheet (PIS). PISs are lengthy and increasingly complex¹¹ and being asked to read such a large document may act as a deterrent to potential participants. A shorter PIS (or participant information leaflet (PIL)) may be more appealing initially, as it is likely to provide a more manageable volume of information.¹² A Cochrane review of recruitment interventions identified two trials that have evaluated a brief PIL compared with a full PIS,^2,12,13 and found the brief PIL makes little or no difference to recruitment compared with a full PIS. RD = 0% (95% CI = -2% to 2%); GRADE: moderate. However, this needs replicating in different populations.

In this study we aimed to evaluate the effectiveness and cost effectiveness of a brief PIL (provided in addition to a standard length PIS) and a trial logo branded pen on recruitment and response rates in the Multiple Symptoms Study 3 (MSS3) host trial. The SWAT was part of the PROMETHEUS programme (MRC MR/R013748/1) designed to identify effective and cost-effective methods to improve recruitment to and retention in trials, and to identify if it is possible to routinely embed SWATs within trials.

Methods

The MSS3 SWAT was approved alongside the host MSS3 trial by the North West – Greater Manchester Central Research Ethics Committee (18/NW/0422). In line with that approval, patients were not informed about the SWAT and therefore, informed consent was not obtained. The MSS3 protocol was registered on ISRCTN (57050216) and has been published.¹⁴ The SWAT was registered with the Northern Ireland Network for Trials Methodology Research SWAT repository (SWAT137). The paper is reported in line with the Trial Forge Guidance: template for reporting the results of randomised Studies Within A Trial.¹⁵

Interventions

The pen was branded with the MSS3 logo and colours (Figure 1). The brief PIL consisted of an A4 sized sheet printed on high quality paper in colour and folded into three, in a leaflet style (Figure 2). It was designed to provide a more succinct and easy to read summary of the MSS3 trial than the standard PIL. The information and format were reviewed by the patient representative on the MSS3 Trial Management Group. It was provided alongside the standard PIL in the recruitment pack. The invitation letter explained that the brief PIL provides a summary of the research in order for the potential participant to decide if they might be interested in the study and that the standard PIL provides more details should they wish to read this before returning the form, but that they will have the opportunity to discuss the study with a member of the research team and ask questions later.

Figure 1. MSS3 branded pen.

Figure 2. MSS3 brief information leaflet.

Results

Participant flow from the initial postal invitation to randomisation in the MSS3 trial is detailed in Figure 3. Recruitment took place between October 2018 to December 2021. A total of 108 GP practices participated in the recruitment process, with 6978 patients initially identified as potentially eligible for invitation to the MSS3 host trial. There were 32 patients whose eligibility status changed between initial GP search and the point at which invitations were posted, so a total of 6946 invitations were sent, of which 318 (4.6%) were randomised following an initial invitation. The SWAT analysis excludes participants who responded to a reminder invitation - there were 2530 reminders sent over the course of the recruitment period and a further 36 participants randomised. The total proportion recruited to the MSS3 host study following both initial and reminder invitations was 354/6946 (5.1%).

Figure 3. Participant flow by arm.

PIL= brief participant information leaflet.

Participant numbers for factorial analysis ‘in the margins’ to determine average brief PIL and pen effects are presented in Table 1.

The proportion randomised per intervention group is shown in Table 2.

Interval estimates for both interventions include the null value, consistent with the hypothesis of no effect on recruitment to the host trial. Sensitivity analysis was conducted using random GP practice effects. The intra-class correlation coefficient was 0.074 and the fixed-effect estimates were consistent with the primary model.

Of the 6946 invited participants, 1086 (15.6%) returned an expression of interest form. Proportions expressing interest in participating in the MSS3 host trial are shown by intervention group in Table 3.

There was evidence of a higher rate of return in the brief PIL group, with estimated odds of return 14% greater than participants not receiving a brief PIL.

The additional cost of including a brief PIL in the invitation pack was £0.50. Given the 0.4% increase in participants randomised (shown in Table 2), 239 brief PIL invitations would need to be sent at a cost of £119.50 to recruit one additional participant to MSS3. The additional cost of including a pen in the invitation pack was £0.41. Cost-effectiveness was not calculated for the pen intervention, given the lower proportion randomised compared to the group of participants not receiving a pen.

There were 768 SWAT participants (70.7%) who returned an expression of interest form but did not go on to be randomised to the main trial. Reasons for non-randomisation are provided in Table 4, by arm and overall.

More than half the participants expressing interest but not going on to be randomised to the host study were found to be ineligible at one of the screening assessments. Of the 65 participants who completed the full screening process but did not consent to be randomised, 16 (24.6%) stated they were no longer interested in participating. The remaining reasons were listed as “other” or missing; free text notes implied the majority occurred due to participant non-attendance.

For participants who returned an expression of interest, the median return time was 16 days. Eight GP centres had partial postage dates imputed with the first day of the month. Five observations were excluded due to missing time-to-event data. Kaplan-Meier plots are presented in Figure 4.

Figure 4. Kaplan-Meier curves showing cumulative response rates by intervention group.

PIL = participant information leaflet.

For those who responded, the time to response was comparable between groups, but a higher proportion responded in the brief PIL and pen groups compared to their respective control groups. Hazard ratios are presented in Table 5. There was evidence of greater response rate (estimated 13% improvement) at any given time in the brief PIL group compared to no brief PIL.

Characteristics of participants returning an expression of interest form are provided in Table 6.

Demographic information for the group of participants successfully randomised is presented in Table 7. Age and sex distributions are comparable to the larger ‘expression of interest’ group.

Discussion

We observed no significant evidence of a difference in the proportion of participants randomised to the MSS3 trial, in those receiving either a pen or brief PIL compared to control. We did observe a significantly higher response rate in those receiving the brief PIL compared to control, but this did not result in an increased randomisation rate in this group. Response rates for those receiving the pen were also higher than the control group, but this was not statistically significant. Response rates were overall lower than had been anticipated when designing the MSS3 trial. There was no difference in time to response between the groups. We did not observe any differences in the characteristics of those expressing an interest versus those randomised, although very little data were available for this comparison.

It is interesting that inclusion of the brief PIL resulted in better response rates but not randomisation rates. One previous SWAT of a brief PIL¹² reported a similar difference in response rates between the brief PIL and control group, however this was not statistically significant. In our study, those returning an expression of interest in the SWAT intervention groups were more likely than those in the control group to not proceed at baseline due to no longer being interested or non-attendance at the appointment. It is possible that receipt of the pen and/or brief PIL may make return of the form more likely as a result of the hypothesised minor incentive, ease of form return or digestion of the study information – resulting in higher expression of interest rates – but that this does not then translate to randomisation when the requirements of participation are fully considered and understood. It is also possible that the brief PIL resulted in a small improvement in randomisation rate that we were not powered to detect, but that would nonetheless have been meaningful.

Although we did not observe evidence of statistically significantly higher randomisation rates, the SWAT interventions were low cost. It should be noted that improved recruitment rates may not be the only rationale for including similar interventions in future trials. For example, the provision of shorter, simplified recruitment materials may aid the representativeness of RCTs by reaching underserved populations.

Ideally, we would have collected demographic data on all participants who were sent an MSS3 invitation, in order to explore whether the SWAT interventions were of benefit to particular groups of participants - however this was not possible. Additionally, only limited data were collected on those returning an expression of interest form. We did not collect any qualitative data from participants with regards the SWAT interventions. This may have enabled us to explore the views of those receiving the interventions. If such data were collected, these would ideally include those who did not go on to participate in the host trial as well as those that did – although this would have been logistically challenging. The host MSS3 RCT was conducted both prior to and during the COVID-19 pandemic, with 59% of MSS3 invitations sent during the pandemic (post March 2020) – however, the number of participants in the SWAT arms were approximately equal pre-pandemic and during pandemic, so this is unlikely to have had an impact on the findings.

In conclusion, there was no significant evidence of effectiveness of the brief PIL intervention or the pen intervention on recruitment to the MSS3 study, but there was evidence of increased response rates to the initial invitation in the brief PIL group. The results of this SWAT should be combined with those from randomised SWATs testing the same interventions, in order to determine definitively whether they are an effective tool to improve recruitment rates.

Author roles

White D: Funding acquisition, Methodology, Writing – original draft preparation, Writing – review & editing; Sutton L: Formal analysis, Visualisation, Writing – original draft preparation, Writing – review & editing; Mooney C: Funding acquisition, Methodology, Project Administration, Writing – review & editing; Dawson J: Funding acquisition, Methodology, Validation, Writing – review & editing; Burton C: Funding acquisition, supervision, writing – review & editing.