Does handwriting the name of a potential trial participant on an invitation letter improve recruitment rates? A randomised controlled study within a trial

Jennifer McCaffery; Alex S. Mitchell; Caroline Fairhurst; Sarah Cockayne; Sara Rodgers; Clare Relton; David J. Torgerson; OTIS Study Team

doi:10.12688/f1000research.18939.1

Home Browse Does handwriting the name of a potential trial participant on an invitation...

ALL Metrics

-

Views

-

Downloads

Get PDF

Get XML

Export

▬

✚

Research Article

Clinical trial

Does handwriting the name of a potential trial participant on an invitation letter improve recruitment rates? A randomised controlled study within a trial

[version 1; peer review: 2 approved]

Jennifer McCaffery ¹, Alex S. Mitchell¹, Caroline Fairhurst¹, [...] Sarah Cockayne¹, Sara Rodgers¹, Clare Relton², David J. Torgerson¹, OTIS Study Team

Jennifer McCaffery ¹, Alex S. Mitchell¹, [...] Caroline Fairhurst¹, Sarah Cockayne¹, Sara Rodgers¹, Clare Relton², David J. Torgerson¹, OTIS Study Team

PUBLISHED 14 May 2019

Author details Author details

¹ Department of Health Sciences, University of York, York, North Yorkshire, YO10 5DD, UK
² SCHARR, University of Sheffield, Sheffield, South Yorkshire, S1 4DA, UK

Jennifer McCaffery
Roles: Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Alex S. Mitchell
Roles: Formal Analysis, Software, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Caroline Fairhurst
Roles: Conceptualization, Formal Analysis, Investigation, Methodology, Validation, Writing – Original Draft Preparation, Writing – Review & Editing

Sarah Cockayne
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Project Administration, Resources, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Sara Rodgers
Roles: Data Curation, Funding Acquisition, Project Administration, Resources

Clare Relton
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Resources, Writing – Review & Editing

David J. Torgerson
Roles: Conceptualization, Formal Analysis, Funding Acquisition, Methodology, Resources, Supervision, Validation, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Studies Within A Trial (SWAT) collection.

Abstract

Background: Randomised controlled trials (RCTs) often fail to recruit to target, resulting in a lack of generalisability of findings. A wide range of strategies for potentially increasing recruitment have been identified; however, their effectiveness has not been established. The aim of this study within a trial (SWAT) was to evaluate the effectiveness of handwritten personalisation of an invitation letter as part of a trial recruitment pack on recruitment to a host RCT.
Methods: A pragmatic, two-armed RCT was conducted, embedded within an existing falls prevention trial (OTIS) in men and women aged 65 years and over living in the community. Participants were randomised 1:1 to receive an OTIS recruitment pack containing an invitation letter on which their name was handwritten (intervention group), or one on which it was printed (control group). The primary outcome was randomisation into the host trial. Secondary outcomes related to trial eligibility and retention. Analyses were via logistic regression and Cox Proportional Hazards regression.
Results: Of the 317 SWAT participants, 12 (3.8%) were randomised into the OTIS trial: 3 (handwritten: 3/159 [1.9%]; printed: 9/158 [5.7%]; difference -3.8%, 95% CI -8.0% to 0.4%). There was weak evidence, against the intervention, of a difference in the likelihood of participants being randomised into the host trial between the two groups (OR 0.32, 95% CI 0.08 to 1.20, p=0.09). There were no statistically significant differences between the intervention and control groups on any of the secondary outcomes.
Conclusions: There was no evidence that personalisation of invitation letters improved recruitment to the OTIS trial. However, due to the small sample size, the results should be interpreted with caution. These findings need to be replicated across larger studies and wider populations.
Registration: ISRCTN22202133.

Keywords

Recruitment, Randomised Controlled Trial, Embedded trial, Personalisation, Handwritten

Corresponding author: Jennifer McCaffery

Competing interests: No competing interests were disclosed.

Grant information: The OTIS study was funded by the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme (Programme grant number: 14/49/149). The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. This SWAT was funded by York Trials Unit. The University of York is the study sponsor and has legal responsibility for the initiation and management of the trial (sponsor representative: Dr Michael Barber, Research and Enterprise Directorate, University of York, Ron Cooke Hub, Heslington, York, UK, YO10 5GE).
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2019 McCaffery J et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: McCaffery J, Mitchell AS, Fairhurst C et al. Does handwriting the name of a potential trial participant on an invitation letter improve recruitment rates? A randomised controlled study within a trial [version 1; peer review: 2 approved]. F1000Research 2019, 8:659 (https://doi.org/10.12688/f1000research.18939.1) First published: 14 May 2019, 8:659 (https://doi.org/10.12688/f1000research.18939.1) Latest published: 14 May 2019, 8:659 (https://doi.org/10.12688/f1000research.18939.1)

Introduction

Randomised controlled trials (RCTs) are regarded as the gold standard design to evaluate the effectiveness of interventions in health research^1–3. However, a recent review of RCTs funded by the National Institute of Health Research (NIHR) in the UK found that only 56% of RCTs reviewed achieved their planned sample size⁴.

Trialists are well aware of recruitment challenges and have adopted a wide range of strategies to achieve and retain their sample size, including gifts, reminders and enhanced cover letters^5,6. A recent Cochrane review has evaluated strategies used to improve recruitment to RCTs⁷. This review identified 68 trials involving more than 74,000 participants. The key conclusions from this review were that there was high-certainty evidence for three methods. The first is that informing participants what they will receive in the trial improves recruitment. The second is that phoning people who do not respond to postal invitations can be effective. Finally, using a tailored, user-tested information sheet makes little or no difference to recruitment. The review found that of the 72 strategies evaluated, only seven involved more than one study; therefore, additional studies are needed to evaluate the effectiveness of strategies to improve recruitment.

One potential strategy that has been found to be effective at improving the responses to postal questionnaires⁸ is making trial documentation more personal. In a review by Edwards et al.⁸, data from 58 trials demonstrated that the odds of returning a questionnaire was increased by more than one tenth (OR 1.14, 95% CI 1.07 to 1.22) when questionnaire material was made more personal. However, there was a wide range of ‘personalisation’ investigated within these trials (e.g. hand-addressing envelopes, signing letters personally etc.). Another systematic review looked specifically at the effect of personally addressed and/or hand-signed letters on questionnaire response. This review found that there was a positive effect on response rates when letters were personally addressed, which increased when letters were also hand-signed⁹.

Our aim was to undertake a study within a trial (SWAT) to determine whether the number of participants recruited to a trial can be improved by writing the potential participant’s name by hand, versus printing the name, on the invitation letter. By embedding a study within a RCT currently being coordinated by the York Trials Unit (YTU; University of York), the NIHR-funded OTIS study¹⁰, the effects of the intervention could be tested within a pragmatic context without the additional costs of participant recruitment.

Methods

Ethics approval

This trial was embedded within the NIHR Health Technology Assessment (HTA) funded Occupational Therapist Intervention Study (OTIS)¹⁰ (Programme grant number 14/49/149). The OTIS study is an RCT that aims to evaluate the clinical and cost effectiveness of an occupational therapist (OT) intervention to reduce falls in high risk older people. Ethical approval for the OTIS trial and this embedded study was given by the West of Scotland Research Ethics Service (WoSRES); Health Research Authority approval and the Department of Health Sciences Research Governance Committee at the University of York. This study within a trial was registered with the ISRCTN registry as part of the host trial registration (ISRCTN22202133; date registered: 20^th June 2016)

Participant recruitment

The OTIS trial is a modified cohort¹¹, pragmatic, multicentre, two-armed RCT. Detailed methods of the main trial have been published elsewhere¹⁰. Participants for OTIS were recruited by mail out of recruitment packs to: i) participants in the Yorkshire Health Study (YHS)¹²; ii) cohorts of previous trials held by the YTU^13–15; iii) potentially eligible patients identified in GP practice databases; and iv) via opportunistic screening. This embedded trial involved potential participants who were approached in the first mail out from the YHS (see Figure 1 for a participant flow diagram). Recruitment to the embedded trial began on the 20^th of April 2017 and follow-up ended on the 22^nd August 2018.

After receiving a recruitment pack (consisting of an invitation letter, participant information sheet, consent form, contact form and screening questionnaire) participants were asked to return their completed screening questionnaire and consent form to researchers based at the YTU if they wished to take part in the study. Participants were eligible to take part in the OTIS trial if they were aged 65 years or over, lived in the community, had fallen in the past 12 months or had a fear of falling, and were willing to receive a home visit from an OT. Participants were ineligible if they were unable to walk 10 feet (even with a walking aid), had dementia, lived in a residential or nursing home, had poor levels of English (with no access to assistance), had received an OT assessment for falls prevention in the last 12 months or were on a waiting list for an assessment, or had not returned a completed falls calendar. Participants who were eligible except for the fact that they had not fallen in the last 12 months and did not report a fear of falling were rescreened at intervals until they asked not to continue participation or they became eligible. Once eligible, participants were sent a baseline questionnaire and pack of falls calendars. Participants who completed their baseline questionnaire and falls calendars then became eligible for randomisation into the OTIS trial. For the main OTIS trial, participants were randomised to an environmental assessment by an OT or to the control group.

Figure 1. Participant flow diagram.

Randomisation and blinding

This trial was embedded in the OTIS trial, and included potential OTIS participants sent a recruitment pack as part of the first mail out to consenting members of the YHS. Recruitment packs were assigned a unique identification number. Block randomisation was used to allocate the recruitment packs in a 1:1 ratio to either the control group or the intervention group, using one large block the size of the mail out (n=317). Generation of the allocation sequence was undertaken by the OTIS trial statistician, who was not involved with the production of the recruitment packs, using Stata version 13.

Control group

The control group received the standard recruitment pack with an invitation letter with their name printed in the salutation. The letter was printed on one page of A4 and is shown in Extended data, Supplementary File 1¹⁶.

Intervention group

The intervention group received the standard recruitment pack with an invitation letter with their name handwritten in the salutation. The names were written by a researcher in in the form Mr/Ms Firstname Surname. The letter was printed on one page of A4 and is shown in Extended data, Supplementary File 2¹⁶.

Primary outcome

The primary outcome was the proportion of participants included in the embedded trial, who went on to be randomised to the host OTIS trial.

Secondary outcomes

The secondary outcomes were:

proportion of participants who returned a screening form
time to return screening form
proportion of participants who fulfilled the eligibility criteria on their initial screening form apart from the criterion relating to falls within past 12 months or fear of falling
proportion of participant who were eligible on their initial screening form
proportion of participants who remained in the trial at three months post randomisation (defined as returning at least the first three months’ worth of falls calendars from the date of randomisation).

Sample size

We randomised 317 participants who were due to be mailed out a recruitment pack about the OTIS trial by the YHS. This sample size is sufficient to detect a 10% absolute difference in the percentage of participants who go on to be randomised (from 10 to 20%) between the two groups at 80% power and a two-sided alpha level of 0.1.

Statistical analysis

Data were analysed on the basis of intention-to-treat and all hypothesis tests were two-sided at the 10% significance level. Categorical data were compared using logistic regression models and time to response data using a Cox proportional hazards model. An additional logistic regression model used to analyse whether participants remained in the trial 3 months post-randomisation was adjusted for the OTIS main trial group allocation (usual care or intervention). The odds ratio (OR) or hazard ratio (HR) from each model associated with the embedded trial allocation is presented along with the corresponding 95% confidence interval (CI) and p-value. All analyses were conducted using Stata version 15.

A completed CONSORT checklist is available at Open Science framework¹⁶.

Results

Primary outcome

Randomised to OTIS main trial. Of the 317 embedded trial participants, 12 (3.8%) were randomised into the OTIS trial (handwritten: 3/159 [1.9%]; printed: 9/158 [5.7%]; difference -3.8%, 95% CI -8.0% to 0.4%). There was weak evidence of a difference in the likelihood of embedded trial participants being randomised into OTIS between the two groups (OR 0.32, 95% CI 0.08 to 1.20, p=0.09), in favour of the control group.

Secondary outcomes

Returned screening form. In total, 49 (15.5%) of the 317 embedded trial participants returned a screening form (handwritten: 22/159 [13.8%]; printed: 27/158 [17.1%]; difference -3.3%, 95% CI -11.2% to 4.7%). There was no evidence of a difference in the likelihood of embedded trial participants returning a screening form between the two groups (OR 0.78, 95% CI 0.42 to 1.44, p=0.42).

Time to return of screening form. For the screening forms returned, the median time to return was 26 days (interquartile range [IQR] 20 to 60) in the handwritten arm and 26 days (IQR 20 to 204 days) in the printed arm. There was no evidence of a difference in the time to response between the two arms (HR 0.81, 95% CI 0.46 to 1.41, p=0.45).

Participants eligible apart from criterion relating to falls. Of the 317 embedded trial participants, 33 (10.4%) were initially ‘almost’ eligible for the OTIS trial (handwritten: 17/159 [10.7%]; printed: 16/158 [10.1%]; difference 0.6%, 95% CI -6.2% to 7.3%). There was no evidence of a difference in the likelihood of participants being eligible on initial screen except for the risk factors for falling between the two groups (OR 1.06, 95% CI 0.52 to 2.19, p=0.87).

Participants eligible for trial. In total, 13 (4.1%) of the 317 embedded trial participants were eligible for the OTIS trial on their initial screening form (handwritten: 3/159 [1.9%]; printed: 10/158 [6.3%]; difference -4.4%, 95% CI -8.8% to -0.1%). There was weak evidence of a difference in the likelihood of participants being fully eligible on initial screen between the two groups (OR 0.28, 95% CI 0.08 to 1.05, p=0.06), in favour of the control group.

Participants who remained in the trial at 3 months post randomisation (return first three falls calendars). Of the 317 embedded trial participants, 10 (3.2%) remained in the OTIS trial 3 months post-randomisation (handwritten: 2/159 [1.3%]; printed: 8/158 [5.1%]; difference -3.8%, 95% CI -7.6% to 0%). There was some evidence of a difference in the proportion of embedded trial participants remaining in the main OTIS trial between the two groups in favour of the control group (unadjusted OR 0.24, 95% CI 0.05 to 1.14, p=0.07). When the logistic regression was adjusted for main trial allocation (which reduced the included sample size to 12) the size of the effect was similar but the confidence interval was much wider and the p-value larger (handwritten: 2/3 [66.7%]; printed: 8/9 [88.9%]; adjusted OR 0.23, 95% CI 0.01 to 5.93, p=0.37).

Discussion

We found that recipients of a personalised invitation letter, on which their name had been handwritten, were three times less likely to be randomised into the host OTIS trial, and this difference was statistically significant at the 10% level.

Finding that personalisation did not increase recruitment to the OTIS trial provides a significant contribution to the limited literature on improving randomisation and recruitment to RCTs. The outcomes of the trial do not support the review by Edwards et al.⁸, which found that personalisation increased the odds of participants’ returning a questionnaire. However, not all of the studies included in the review observed an increase in response rate using personalisation. Moss and Worthern¹⁷ found that writing the potential participant’s name by hand significantly reduced response rates when compared to typing the name. The review by Edwards et al.⁸ included a wide range of studies within a wide range of contexts (e.g. teachers, students and individuals selected at random from the telephone directory). Moss and Worthern¹⁷ invited psychologists to provide their views on standardised assessments. They suggest that the reduction in response rates may have been due to handwriting being perceived as more personal but less professional. This supports Linsky’s theory that there are complex interactions between methods used to increase recruitment and the context in which they are used¹⁸. Our findings may also provide additional evidence for Linsky’s¹⁸ recommendation that in contexts requiring higher levels of confidentiality and professionalism, such as health research, personalisation may be contra-indicated.

There are some limitations to the study. Only a small number of potential participants were recruited and randomised to the OTIS study, indeed far fewer than anticipated, and so the embedded trial is severely underpowered. This reduces the reliability of our findings. This highlights the value of repeating this investigation with larger sample sizes and reviewing findings across similar studies. Potential participants were limited to individuals over the age of 65 years living in the community, as such the results are only applicable to this population. Further studies should substantiate the study results in other populations.

Conclusion

Our findings that personalisation does not improve recruitment lend weight to the argument that methods to improve recruitment may well be context specific. Given the small sample size the results should be interpreted with caution and highlight the need to replicate and extend this work across larger studies and wider populations.

Data availability

Underlying data

Open Science Framework: OTIS Invitation Letter SWAT. https://doi.org/10.17605/OSF.IO/KGH4S¹⁶.

This project contains the underlying data in CSV and SAV format, with a variable key in CSV format.

Extended data

Open Science Framework: OTIS Invitation Letter SWAT. https://doi.org/10.17605/OSF.IO/KGH4S¹⁶.

This project contains the following extended data:

Supplementary File 1. Invitation letter for the Control Group.
Supplementary File 2. Invitation letter for the Intervention Group.

Reporting guidelines

Open Science Framework: CONSORT checklist for study “Does handwriting the name of a potential trial participant on an invitation letter improve recruitment rates? A randomised controlled study within a trial”. https://doi.org/10.17605/OSF.IO/KGH4S¹⁶.

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Grant information

The OTIS study was funded by the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme (Programme grant number: 14/49/149). The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. This SWAT was funded by York Trials Unit. The University of York is the study sponsor and has legal responsibility for the initiation and management of the trial (sponsor representative: Dr Michael Barber, Research and Enterprise Directorate, University of York, Ron Cooke Hub, Heslington, York, UK, YO10 5GE).

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Acknowledgements

The authors would like to thank the embedded trial participants who returned trial recruitment documentation. This manuscript has been written by the authors on behalf of the OTIS Study Team. Sophie Boyes (York Teaching Hospital NHS Foundation Trust); Sarah Cockayne (University of York); Belen Corbacho (University of York); Shelley Crossland (Leicestershire Partnership NHS Trust); Avril Drummond (University of Nottingham); Caroline Fairhurst (University of York); Simon Gilbody (University of York); Catherine Hewitt (University of York); Sarah E Lamb (University of Oxford); Jennifer McCaffery (University of York); Alison Pighills (Mackay Base Hospital; Mackay Australia and James Cook University); Clare Relton (University of Sheffield); Sara Rodgers (University of York); and David J. Torgerson (University of York).

Faculty Opinions recommended

References

1. Djulbegovic B, Guyatt GH: Progress in evidence-based medicine: a quarter century on. Lancet. 2017; 390(10092): 415–23. PubMed Abstract | Publisher Full Text
2. Brownson RC, Fielding JE, Maylahn CM: Evidence-based public health: a fundamental concept for public health practice. Annu Rev Public Health. 2009; 30: 175–201. PubMed Abstract | Publisher Full Text
3. Greenhalgh T, Howick J, Maskrey N, et al.: Evidence based medicine: a movement in crisis? BMJ. 2014; 348: g3725. PubMed Abstract | Publisher Full Text | Free Full Text
4. Walters SJ, Bonacho Dos Anjos Henriques-Cadby I, Bortolami O, et al.: Recruitment and retention of participants in randomised controlled trials: a review of trials funded and published by the United Kingdom Health Technology Assessment Programme. BMJ Open. 2017; 7(3): e015276. PubMed Abstract | Publisher Full Text | Free Full Text
5. Caldwell PH, Hamilton S, Tan A, et al.: Strategies for increasing recruitment to randomised controlled trials: systematic review. PLoS Med. 2010; 7(11): e1000368. PubMed Abstract | Publisher Full Text | Free Full Text
6. Kearney A, Daykin A, Shaw ARG, et al.: Identifying research priorities for effective retention strategies in clinical trials. Trials. 2017; 18(1): 406. PubMed Abstract | Publisher Full Text | Free Full Text
7. Treweek S, Pitkethly M, Cook J, et al.: Strategies to improve recruitment to randomised trials. Cochrane Database Syst Rev. 2018; 2: MR000013. PubMed Abstract | Publisher Full Text
8. Edwards PJ, Roberts I, Clarke MJ, et al.: Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev. 2009; (3): MR000008. PubMed Abstract | Publisher Full Text
9. Scott P, Edwards P: Personally addressed hand-signed letters increase questionnaire response: a meta-analysis of randomised controlled trials. BMC Health Serv Res. 2006; 6: 111. PubMed Abstract | Publisher Full Text | Free Full Text
10. Cockayne S, Pighills A, Adamson J, et al.: Can occupational therapist-led home environmental assessment prevent falls in older people? A modified cohort randomised controlled trial protocol. BMJ Open. 2018; 8(9): e022488. PubMed Abstract | Publisher Full Text | Free Full Text
11. Relton C, Torgerson D, O'Cathain A, et al.: Rethinking pragmatic randomised controlled trials: introducing the "cohort multiple randomised controlled trial" design. BMJ. 2010; 340: c1066. PubMed Abstract | Publisher Full Text
12. Relton C, Bissell P, Smith C, et al.: South Yorkshire Cohort: a 'cohort trials facility' study of health and weight - protocol for the recruitment phase. BMJ Public Health. 2011; 11: 640. PubMed Abstract | Publisher Full Text | Free Full Text
13. Cockayne S, Adamson J, Clarke A, et al.: Cohort Randomised Controlled Trial of a Multifaceted Podiatry Intervention for the Prevention of Falls in Older People (The REFORM Trial). PLoS One. 2017; 12(1): e0168712. PubMed Abstract | Publisher Full Text | Free Full Text
14. Lewis H, Adamson J, Atherton K, et al.: CollAborative care and active surveillance for Screen-Positive EldeRs with subthreshold depression (CASPER): a multicentred randomised controlled trial of clinical effectiveness and cost-effectiveness. Health Technol Assess. 2017; 21(8): 1–196. PubMed Abstract | Publisher Full Text | Free Full Text
15. Shepstone L, Lenaghan E, Cooper C, et al.: Screening in the community to reduce fractures in older women (SCOOP): a randomised controlled trial. Lancet. 2018; 391(10122): 741–7. PubMed Abstract | Publisher Full Text
16. McCaffery J, Mitchell A, Fairhurst C, et al.: Does Handwriting the Name of a Potential Trial Participant on an Invitation Letter Improve Recruitment Rates? A randomised control trial within a trial. 2019.
17. Moss VD, Worthen BR: Do personalization and postage make a difference on response rates to surveys of professional populations? Psychol Rep. 1991; 68(2): 692–4. Publisher Full Text
18. Linsky AS: Stimulating responses to mailed questionnaires: A review. Public Opin Q. 1975; 39(1): 82–101. Publisher Full Text

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 14 May 2019