F1000ResearchF1000Research2046-1402F1000 Research LimitedLondon, UK10.12688/f1000research.23767.2Research ArticleArticlesIncluding a pen and/or cover letter, containing social incentive text, had no effect on questionnaire response rate: a factorial randomised controlled Study within a Trial
[version 2; peer review: 2 approved]
JamesSophieFormal AnalysisInvestigationMethodologyProject AdministrationResourcesWriting – Original Draft PreparationWriting – Review & Editinghttps://orcid.org/0000-0003-1878-2779a1ParkerAdwoaConceptualizationMethodologySupervisionWriting – Original Draft PreparationWriting – Review & Editing1CockayneSarahMethodologyProject AdministrationWriting – Original Draft PreparationWriting – Review & Editinghttps://orcid.org/0000-0002-1288-54971RodgersSaraProject AdministrationWriting – Review & Editinghttps://orcid.org/0000-0003-1757-541X1FairhurstCarolineData CurationFormal AnalysisMethodologyWriting – Original Draft PreparationWriting – Review & Editinghttps://orcid.org/0000-0003-0547-462X1TorgersonDavid J.ConceptualizationMethodologySupervisionWriting – Original Draft PreparationWriting – Review & Editinghttps://orcid.org/0000-0002-1667-42751RhodesSarahConceptualizationMethodologyResourcesWriting – Review & Editinghttps://orcid.org/0000-0002-5837-801X2CotterillSarahConceptualizationMethodologyResourcesWriting – Review & Editinghttps://orcid.org/0000-0001-5136-390X2
York Trials Unit, University of York, UK, York, Y010 5DD, UK
Centre for Biostatistics, University of Manchester, Manchester, M13 9PL, UKsophie.j.james@york.ac.uk
Background: Postal questionnaires are frequently used in randomised controlled trials to collect outcome data on participants; however, poor response can introduce bias, affect generalisability and validity, and reduce statistical power. The objective of this study was to assess whether a pen and/or social incentive text cover letter sent with a postal follow-up questionnaire increased response rates in a trial.
Method: A two-by-two factorial randomised controlled trial was embedded within the OTIS host trial. Participants due their 12-month (final) follow-up questionnaire were randomised to be sent: a pen; a social incentive text cover letter; both; or neither. The primary outcome measure was the proportion of participants in each group who returned the questionnaire. Secondary outcomes were: time to return, completeness of the questionnaire, necessity of a reminder letter, and the cost effectiveness.
Results: The overall 12-month questionnaire response rate was 721 out of 755 (95.5%). Neither the pen nor social incentive cover letter had a statistically significant effect on response rate: pen 95.2% vs. no pen 95.8%, adjusted OR 0.90 (95% CI 0.45 to 1.80; p=0.77); social incentive cover letter 95.2% vs. no social incentive cover letter 95.8%, adjusted OR 0.84 (95% CI 0.42 to 1.69, p=0.63). No statistically significant differences were observed between either of the intervention groups on time to response, need for a reminder or completeness. Therefore, neither intervention was cost-effective.
Conclusions: We found no evidence of a difference in response rates associated with the inclusion of a pen and/or social incentive cover letter with the final follow-up postal questionnaire of the host trial. However, when these results are combined with previous SWATs, the meta-analysis evidence remains that including a pen increases response rates. The social incentive cover letter warrants further investigation to determine effectiveness.
Trial registration:
ISRCTN22202133 (21
st June 2016).
Retentionpensocial incentivecover letterrandomised controlled trialembedded trialSWATpostal questionnaireresponse rateNational Institute for Health Research14/49/149The OTIS study was funded by the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme (Programme grant number: 14/49/149). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. This SWAT was funded by PROMETHEUS and the results from this project will contribute to the evidence towards trial methodology for improving retention of participants. This will study will be linked with a national research programme PROMETHEUS led by York Trials Unit (https://www.york.ac.uk/healthsciences/research/trials/research/swats/prometheus/) and findings will be combined with other studies in meta-analyses to detect small but cost effective differences. This will help future trials to be designed with effective interventions in place to maximise retention and avoid introduction of bias and reduced study power.
The University of York is the study sponsor and has legal responsibility for the initiation and management of the trial (sponsor representative: Dr Michael Barber, Research and Enterprise Directorate, University of York, Ron Cooke Hub, Heslington, York, UK, YO10 5GE).
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.Amendments from Version 1
We have added additional detail and clarification to the revised version of the article in response to comments raised by the reviewers. We have made clearer the definition and concept of social incentive which underpinned the intervention cover letter and the difference in this terminology from social pressure or social reward. We have also made minor amendments to Table 2 to remove BMI and EQ-5D-5L scores which the reviewers felt were not relevant. We added details about the PROMETHEUS programme and edited the description of the meta-analysis to address reviewer comments on the reason for the meta-analysis and the other studies that were included in this. We have also updated the discussion to also reflect the point that is mentioned in the methods that all participants were given a £5 monetary reward. Finally, we have updated two figures in the meta-analysis, due to a recently published new version (version 2) of one of the studies included in the meta-analysis ('Mitchell A, Cook L, Dean A, et al.: Using pens as an incentive for questionnaire return in an orthopaedic trial: an embedded randomised controlled retention trial') being published on F1000. This version made amendments to their results in light of a duplicate randomisation that was found in the host trial, these changes were negligible and did not affect the interpretation of the results. However this means that for accuracy the meta analysis figure of this paper has been updated to include these new figures.
Introduction
Randomised controlled trials (RCTs) are the gold standard to assess effectiveness of treatment options and to inform care decisions
1, yet only a few hundred studies exist to assess the effectiveness of different methods to improve retention or recruitment into RCTs
2.
Trial methodologists and funders have highlighted the need to evaluate participant recruitment and retention strategies in order to provide evidence on which to base decisions around the design and conduct of RCTs
3.
Postal questionnaires are frequently used in randomised controlled trials to collect outcome data on participants; however, poor response can introduce bias, affect generalisability and validity, and reduce statistical power. Several systematic reviews report on the topic of retention strategies, including improving response rates to questionnaires
4–
7. However, there remains a lack of definitive evidence regarding some commonly adopted practices such as sending a pen or using a cover letter with a questionnaire to encourage the participant to return it
8–
10. The results of a study within a trial (SWAT) evaluating these two strategies are reported here.
MethodsDesign
A two-by-two factorial RCT was embedded within the OTIS trial of occupational therapist-led home assessment and modification for the prevention of falls (
ISRCTN22202133)
11. OTIS recruited participants over the age of 65 years who were at risk of falling. Participants were randomised to receive an occupational therapist delivered visit or usual care. They were followed up for 12 months for falls data and were sent postal questionnaires at four, eight and 12 months. This SWAT was embedded at the 12-month time point. Ethical approval for this SWAT was received from the NHS West of Scotland Research Ethics Committee 3 (16/WS/0154) and Health Research Authority and Research Ethics approval in July 2018. Approvals were obtained from the University of York, Department of Health Sciences Research Governance Committee. Participants provided informed consent to be enrolled into the OTIS trial and to be sent study related information by post. Consent for the SWAT was therefore waived by the above-named ethics committee.
Participants
A total of 779 participants due to receive their 12-month questionnaire between 16
th October 2018 and 2
nd August 2019 were randomised into the SWAT in a single tranche in September 2018. Participants who had withdrawn from the OTIS study prior to this were excluded from randomisation.
The allocation sequence was generated by the OTIS statistician, who was not involved with the sending of the questionnaires, using STATA v15
12. The identification numbers of OTIS participants to be involved in the SWAT were randomised 1:1:1:1 in a single block. Because there were no descriptive details of the participants attached to the identification numbers this meant the randomisation was concealed.
Interventions
Table 1 details the combination of interventions sent in the post with the 12-month questionnaire. We included an unconditional £5 note with the questionnaire for all participants.
The non-standard cover letter offered a mild level of social incentive, in the form of a personalised table that indicated whether or not a questionnaire had been received from the participant at the earlier (4 and 8-month) time points. The concept of social incentive that underpinned the intervention for this study was that a social incentive is something that persuades people to behave in a certain way by the promise that their actions will be noticed or made public
10. Therefore, the cover letter was intended to highlight to the participant that their questionnaire responses are noted and valued
10.
Intervention groups.
PenYork Trials Unit branded pen, standard cover
letter
(Supplementary File 1)*
Control GroupNo pen, standard cover letter
(Supplementary File 4).
Pen and Social Incentive Cover LetterYork Trials Unit branded pen, social incentive
cover letter
(Supplementary File 3).
Social Incentive cover letterSocial incentive cover letter (
Supplementary File 2), no pen.
*Supplementary Files are available as Extended data
13.
Blinding and quality assurance
Participants were blind to their participation. Research administrators and research team members posting the questionnaire packs were not blind to the intervention; however, administrators who recorded the outcome data were blind to allocation.
Primary objective
To assess whether a pen and/or social incentive text cover letter sent with the 12-month questionnaire increased postal questionnaire response rates for participants in the OTIS trial.
Primary outcome
The primary outcome was response rate, defined as the proportion of participants in each group who returned the 12-month questionnaire.
Secondary outcomes
Time to return 12-month questionnaire
The completeness of the 12-month questionnaire
The requirement for a reminder letter to be sent
Cost effectiveness
Statistical analysis
The data were analysed in SPSS v25
14 using two-sided tests at the 5% significance level on an intention-to-treat basis. Participants who withdrew or died before the 12-month questionnaire was sent were excluded from the analysis. The primary outcome was compared using a logistic regression model adjusting for age (retention is generally higher in participants ˂75 years and older adults may respond differently to incentives
15), gender (to control for potential differences in anticipation of social incentives between males and females
16) and host trial treatment allocation. The presence of an interaction between the two interventions was tested by introducing the interaction term into the logisit model. Time to questionnaire return (calculated as days from questionnaire sent to return) was analysed using Cox Proportional Hazards regression, adjusting for the same covariates as in the primary analysis. The proportional hazards assumption was assessed using Schoenfeld residuals
17. Completeness of response (defined as number of items completed) was analysed by linear regression model and adjusted as for the primary analysis.
Cost effectiveness was calculated for each group using the total cost of the pen/letter/postage/stationary and staff time.
Due to SWATs typically being under-powered to show small effects, it is essential that the results are seen within the context of the wider literature. A fixed effect meta-analysis using the Mantel-Haenszel method was conducted using review manager v5.3
18 to pool the results of this study for enclosing a pen with the 12-month questionnaire with other RCT evidence. These were located utilising the Cochrane systematic review
7 search strategy (Supplementary file 14) in MEDLINE and EMBASE, along with hand searching of previous systematic reviews references, published retention research reference lists, conference papers and co-author personal knowledge of studies. The results of this study were pooled with four previous SWATs
8,
9,
19,
20 investigating the same intervention, with the same dichotomous outcome of response to the questionnaire or not. Pooled odds ratios and corresponding 95% CIs were calculated. Heterogeneity between trials was assessed using the Chi-squared and I
2 statistics. The meta-analysis was facilitated by the PROMoting THE USE of SWATs (PROMETHEUS) programme, which supports host trial teams to conducted SWATs and for data obtained to be collated and meta-analysed.
A meta-analysis of the results of the social incentive intervention was not undertaken as the only previous study using this was conducted within a cohort study rather than an RCT
10.
Results
Figure 1 depicts the recruitment and retention of participants in the embedded trial.
Table 2 presents summary statistics for the baseline characteristics of the SWAT participants.
Flow diagram depicting the recruitment and retention of participants in this embedded trial.
Baseline characteristics of the SWAT participants.
Pen only
(n=192)
Pen and social
incentive cover
letter
(n=184)
Social incentive
cover letter
only (n=189)
Standard 12-
month cover letter
(control) (n=190)
Age
n
192
184
189
190
Mean (SD)
80 (6.3)
80 (6.1)
79 (6.2)
80 (6.2)
Min, Max
67, 98
66, 98
65, 98
69, 94
Gender
n
192
184
189
190
Male
73 (38.0%)
56 (30.4%)
59 (31.2%)
69 (36.3%)
Female
119 (62.0%)
128 (69.6%)
130 (68.8)
121 (63.7%)
Host trial randomisation
n
192
184
189
190
OT visit (intervention)
61 (31.8%)
49 (26.6%)
59 (31.2%)
65 (34.2%)
GP standard care
131 (68.2%)
135 (73.4%)
130 (68.8%)
125 (65.8%)
Number of falls in 12 months prior to randomisation
n
145
139
149
135
Mean
2.2 (3.0)
1.8 (1.4)
2.0 (1.7)
2.2 (2.1)
Min, Max
1, 21
1, 11
1, 10
1, 15
#= How good or bad your health is today rated from 0 worst, 100 best.
Primary outcome
Between randomisation into the SWAT and being sent their 12-month questionnaire, 24 randomised participants either died or withdrew from the host trial and so were not sent the questionnaire. A total of 721/755 (95.5%) returned the 12-month questionnaire. The response rate was identical in the pen only group (184/192, 95.8%), social incentive cover letter only group (181/189, 95.8%) and control group (182/190, 95.8%). However, it was marginally lower in the pen and social incentive cover letter group (174/184, 94.6%).
No evidence of a difference in response rates was found between participants with or without pens (pen: 358/376 [95.2%]; no pen: 363/379 [95.8%]; adjusted OR 0.90, 95% CI 0.45 to 1.80, p=0.77) nor with or without the social incentive cover letter (cover letter: 355/373 [95.2%]; no cover letter: 366/382 [95.8%]; adjusted OR 0.84, 95% CI 0.42 to 1.69, p=0.63) (
Table 3).
Primary outcome results.
Primary
outcome
Group
Hazard ratio (HR)/
Odds ratio (OR)/Mean
difference (MD)
95%
Confidence
Interval
p-value
Other
Responserate
Pen received vs. not received
OR = 0.90
0.45, 1.80
0.77
Total of 721/755 (95.5%)
returned tde 12-month
questionnaire
Social incentive cover letter
received vs. not received
OR = 0.84
0.42, 1.69
0.29
Host trial allocation
(intervention vs. control)
OR = 1.40
0.64, 3.23
0.38
Age (per year)
OR = 0.96
0.91, 1.01
0.11
Gender (male vs. female)
OR = 0.71
0.35, 1.44
0.35
The interaction between the interventions was found to be non-significant (interaction effect size estimate OR 0.79 95% CI 0.20, 3.15 p = 0.74).
Secondary outcomes
Time to return. Median time to return the questionnaire was nine days, with a mean of 12.2 days. No statistically significant difference between the groups was found (
Table 4).
Secondary outcome results.
Secondary
outcome
Group
Hazard ratio (HR)/
Odds ratio (OR)/Mean
difference (MD)
95%
Confidence
Interval
p-value
Other
Time to return
Pen received vs. not
received
HR = 1.08
0.93, 1.25
0.30
Mean time for all participants to
return questionnaire = 12.2 days.
Median time for all participants to
return questionnaire = 9 days.
Social incentive cover
letter received vs. not
received
HR =1.101
0.87, 1.17
0.92
Host trial allocation
(intervention vs. control)
HR = 0.85
0.73, 1.00
0.05
Age (per year)
HR = 0.99
0.97, 1.00
0.02
Gender (male vs. female)
HR = 1.80
0.92, 1.26
0.35
Reminderssent
Pen received vs. not
received
OR = 0.89
0.56, 1.42
0.63
83/755 (11.0%) required a reminder
p value associated with the Kruskal-
Wallis test statistic p=0.190
Social incentive cover
letter received vs. not
received
OR = 0.92
0.58, 1.47
0.74
Host trial allocation
(intervention vs. control)
OR = 1.611
1.00, 2.59
0.05
Age (per year)
OR = 1.04
1.00, 1.08
0.03
Gender (male vs. female)
OR = 0.87
0.53, 1.42
0.57
Completenessof response
Pen received vs. not
received
MD = 0.14
-0.46, 0.74
0.65
Overall average completeness of the
questionnaires was 27.8/31 questions
(89.6% complete)
Social incentive cover
letter received vs. not
received
MD = 0.09
-0.69, 0.51
0.78
Host trial allocation
(intervention vs. control)
MD = -0.10
-0.55, 0.75
0.77
Age (per year)
MD = -0.10
-0.46, 0.74
0.65
Gender (male vs. female)
MD = -1.06
-1.69, -0.42
˂0.001
Reminders sent. In total, 83/755 (11.0%) participants required a reminder letter. The pen and social incentive cover letter group required the least reminders (19/184 10.3%) and the control group required the most reminders (24/190 12.6%). No statistically significant evidence was found of a difference of participants requiring a reminder between the groups (
Table 4).
Completeness of response. Overall average completeness of the questionnaires was 27.8/31 questions (89.6% complete) with no evidence of a difference in completeness of the questionnaire between pen received or not (
Table 4).
Cost effectiveness. Due to the non-statistically significant effect of the interventions on response rates calculating overall associated costs provides evidence of potential cost savings not to send the social incentive cover letter and/or pen (
Extended data: Supplementary File 9
13).
Meta-analysis
A fixed effect meta-analysis of enclosing a pen with a follow-up postal questionnaire on response rate was conducted (
Figure 2). This included five studies
8,
9,
21,
22 (n=13012 participants) and gave a statistically significant pooled OR favouring the intervention (1.21, 95% CI 1.09 to 1.35 p = 0.0004). Negligible heterogeneity was observed (chi-squared = 2.88 I
2= 0%). The risk of bias was low, as indicated by the Cochrane’s risk of bias tool assessment undertaken
23 (
Extended data: Supplementary File 10
13).
Meta-analysis of enclosing a pen with a questionnaire to increase response rate to a postal questionnaire.
Discussion
This SWAT found no evidence that sending a pen and/or a social incentive cover letter with a postal, trial follow-up questionnaire improved response rate, time to return, requirement for a reminder, or questionnaire completeness.
A limitation was the average age of the participants (79.9 years) giving a narrow age demographic thus restricting generalisability of results. Further investigation of the pen and social incentive cover letter in RCTs are required across more diverse populations.
The OTIS trial hosted three other methodological SWATs; therefore, there was a potential for contamination or interaction. It is preferable to plan all SWATs that will be undertaken in the early design stages
19, to ensure they are planned accordingly to reduce the potential of this.
The overall response rate of the 12-month postal questionnaire for all SWAT participants was 95.7%, which may have been helped by the inclusion of £5 to all participants as standard. This high response rate is therefore difficult to improve upon, furthermore the incentives may not have been as effective with participants who are very committed to the behaviour
10. The incentive required for committed participants may be different
10,
20. A learning point being that future SWATS testing these interventions should avoid doing so in trials with already high response rates.
Conclusion
Whilst neither the pen nor the social incentive cover letter showed an effect on response rate, the meta-analysis evidence remains that including a pen increases response rates. This reinforces that for interventions where small effects are likely, it is important to undertake a number of trials and combine these to be confident of an intervention’s effectiveness. Further investigation of the social incentive cover letter in RCTs is required to determine effectiveness.
Data availabilityUnderlying data
Open Science Framework: Pen and Social Incentive Cover Letter Retention SWAT,
https://doi.org/10.17605/OSF.IO/7TDRB13.
Extended data
Open Science Framework: Pen and Social Incentive Cover Letter Retention SWAT,
https://doi.org/10.17605/OSF.IO/7TDRB13.
This project contains the following extended data:
Full study protocol
Supplementary File 1- Cover letter for the Pen only group.
Supplementary File 2 - Cover letter for the Social incentive cover letter only group.
Supplementary File 3 - Cover letter for the Pen and social incentive cover letter group.
Supplementary File 4 - Cover letter for the control group.
Supplementary File 5 - Results table by intervention group
Supplementary File 6 - Graph Survival curve of pen vs no pen and time taken to return 12-month questionnaire.
Supplementary File 7 - Graph Survival curve of Social incentive cover letter vs no social incentive cover letter and time taken to return 12-month questionnaire.
Supplementary File 8- Survival curve of host trial allocation and time taken to return 12-month questionnaire.
Supplementary File 9 – Costings table
Supplementary File 10 – Cochrane Risk of bias tool assessments for Bell
et al., (2016)
8, Sharp
et al., (2006)
9, Cunningham-Burley
et al., (2020)
21, Mitchell
et al., (2020)
22 and James
et al., (2020).
Supplementary File 11 – Summary of all SWATs undertaken in the OTIS study
Supplementary File 12 – Copy of the OTIS reminder letter
Supplementary file 13 – Summary of studies included in the meta-analysis
Supplementary File 14 – Copy of the search strategies of Brueton
et al., (2014)
Reporting guidelines
Open Science Framework: CONSORT checklist for ‘Including a pen and/or cover letter, containing social incentive text, had no effect on questionnaire response rate: a factorial randomised controlled Study within a Trial’,
13.
Data are available under the terms of the
Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).
Acknowledgements
The authors would like to thank the embedded trial participants who returned 12-month questionnaires.
This manuscript has been written by the authors on behalf of the OTIS Study Team. Sophie Boyes (York Teaching Hospital NHS Foundation Trust); Belen Corbacho (University of York); Shelley Crossland (Leicestershire Partnership NHS Trust); Avril Drummond (University of Nottingham); Simon Gilbody (University of York); Catherine Hewitt (University of York); Sarah E Lamb (University of Oxford); Katie Whiteside (University of York); Jennifer McCaffery (University of York); Alison Pighills (Mackay Base Hospital; Mackay Australia and James Cook University); Clare Relton (University of Sheffield).
The results from this project will contribute to the evidence towards trial methodology for improving retention of participants. This will study will be linked with a national research programme PROMETHEUS led by York Trials Unit (
https://www.york.ac.uk/healthsciences/research/trials/research/swats/prometheus/) and findings will be combined with other studies in meta-analyses to detect small but cost effective differences. This will help future trials to be designed with effective interventions in place to maximise retention and avoid introduction of bias and reduced study power.
EbleABoonePElbourneD:
On minimizing the risk of bias in randomized controlled trials in economics. The World Bank.2016.
Reference SourceClarkLRonaldsonSDysonL:
Electronic prompts significantly increase response rates to postal questionnaires: a randomized trial within a randomized trial and meta-analysis.J Clin Epidemiol.2015;68(12):1446–50.
2570907610.1016/j.jclinepi.2015.01.016EdwardsPRobertsIClarkeM:
Increasing response rates to postal questionnaires: systematic review.BMJ.2002;324(7347):1183.
1201618110.1136/bmj.324.7347.1183111107NakashRAHuttonJLJØrstad-SteinEC:
Maximising response to postal questionnaires - A systematic review of randomised trials in health research.BMC Med Res Methodol.2006;6:5.
1650409010.1186/1471-2288-6-51421421EdwardsPJRobertsIGClarkeMJ:
Methods to increase response rates to postal questionnaires.Cochrane Database Syst Rev.2007; (2): MR000008.
1744362910.1002/14651858.MR000008.pub3EdwardsPJRobertsIClarkeMJ:
Methods to increase response to postal and electronic questionnaires.Cochrane Database Syst Rev.2009; (3): MR000008.
1958844910.1002/14651858.MR000008.pub4BruetonVCTierneyJFStenningS:
Strategies to improve retention in randomised trials: a Cochrane systematic review and meta-analysis.BMJ Open.2014;4(2):e003821.
2449669610.1136/bmjopen-2013-0038213918995BellKClarkLFairhurstC:
Enclosing a pen reduced time to response to questionnaire mailings.J Clin Epidemiol.2016;74:144–150.
2673897110.1016/j.jclinepi.2015.12.004SharpLCochranCCottonSC:
Enclosing a pen with a postal questionnaire can significantly increase the response rate.J Clin Epidemiol.2006;59(7):747–54.
1676527910.1016/j.jclinepi.2005.10.014CotterillSHowellsKRhodesS:
The effect of using social pressure in cover letters to improve retention in a longitudinal health study: an embedded randomised controlled retention trial.Trials.2017;18(1):341.
28728603 10.1186/s13063-017-2090-55520383CockayneSPighillsAAdamsonJ:
Can occupational therapist-led home environmental assessment prevent falls in older people? A modified cohort randomised controlled trial protocol.BMJ Open.2018;8(9):e022488.
3020608610.1136/bmjopen-2018-0224886144405StataCorp:
Stata Statistical Software: Release 15. College Station, T.S.L.2017.
Reference SourceJamesS:
Pen and Social Incentive Cover Letter Retention SWAT.2020.
http://www.doi.org/10.17605/OSF.IO/7TDRBCorp I:
IBM SPSS Statistics for Windows, Version 25.0. Armonk, NY: IBM Corp. Released (2017).
Reference SourceChhatreSJeffersonACookR:
Patient-centered recruitment and retention for a randomized controlled study.Trials.2018;19(1):205.
2958780510.1186/s13063-018-2578-75870194SpreckelmeyerKNKrachSKohlsG:
Anticipation of monetary and social reward differently activates mesolimbic brain structures in men and women.Soc Cogn Affect Neurosci.2009;4(2):158–65.
1917453710.1093/scan/nsn0512686229SchoenfeldD:
Partial residuals for the proportional hazards regression model.Biometrika.1982;69(1):239–241.
10.1093/biomet/69.1.239Review Manager (RevMan) [Computer program]. Version 5.3. Copenhagen: The Nordic Cochrane Centre, T.C.C,2014.BowerPBruetonVGambleC:
Interventions to improve recruitment and retention in clinical trials: a survey and workshop to assess current practice and future priorities.Trials.2014;15:399.
2532280710.1186/1745-6215-15-3994210542O'KeefeDJ:
Persuasion: Theory and Research. SAGE publishers,2002.
Reference SourceCunningham-BurleyRRocheJFairhurstC:
Enclosing a pen to improve response rate to postal questionnaire: an embedded randomised controlled trial [version 1; peer review: awaiting peer review].F1000Res.2020;9:577.
3372804010.12688/f1000research.23651.17941094MitchellACookLDeanA:
Using pens as an incentive for questionnaire return in an orthopaedic trial: an embedded randomised controlled retention tria.F1000Res.2021;9:321.HigginsJPTAltmanDGGøtzschePC:
The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials.BMJ.2011;343:d5928.
2200821710.1136/bmj.d5928319624510.5256/f1000research.77829.r99236Reviewer response for version 2DuncanEilidh1Refereehttps://orcid.org/0000-0002-1009-9732LawrieLouisa1Co-refereeCoffeyTaylor1Co-referee
Health Services Research Unit, University of Aberdeen, Aberdeen, UK
Competing interests: No competing interests were disclosed.
Is the work clearly and accurately presented and does it cite the current literature?
Yes
If applicable, is the statistical analysis and its interpretation appropriate?
Yes
Are all the source data underlying the results available to ensure full reproducibility?
Yes
Is the study design appropriate and is the work technically sound?
Yes
Are the conclusions drawn adequately supported by the results?
Yes
Are sufficient details of methods and analysis provided to allow replication by others?
Yes
Reviewer Expertise:
Health services research, design and evaluation of interventions to change behaviour; the application of behavioural theory to trial recruitment and retention.
We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
10.5256/f1000research.77829.r99235Reviewer response for version 2Grylka-BaeschlinSusanne1Refereehttps://orcid.org/0000-0001-5652-6886
Research Unit for Midwifery Science, Zurich University of Applied Sciences, Katharina-Sulzer-Platz, Winterthur, Switzerland
Competing interests: No competing interests were disclosed.
Thank you for thoroughly revising this manuscript. The limitations of the study are now addressed in the discussion, nevertheless diminish the importance of the results. However, the significance of the study was increased by the meta-analysis. I only have one very minor comment:
Table 3, column “Other”: should it be “the…” instead of “tde 12-month questionnaire…”?
Is the work clearly and accurately presented and does it cite the current literature?
Yes
If applicable, is the statistical analysis and its interpretation appropriate?
Yes
Are all the source data underlying the results available to ensure full reproducibility?
Yes
Is the study design appropriate and is the work technically sound?
Partly
Are the conclusions drawn adequately supported by the results?
Partly
Are sufficient details of methods and analysis provided to allow replication by others?
Yes
Reviewer Expertise:
Midwifery, Epidemiology
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
10.5256/f1000research.26227.r91091Reviewer response for version 1Grylka-BaeschlinSusanne1Refereehttps://orcid.org/0000-0001-5652-6886
Research Unit for Midwifery Science, Zurich University of Applied Sciences, Katharina-Sulzer-Platz, Winterthur, Switzerland
Competing interests: No competing interests were disclosed.
Thank you very much for giving me the opportunity to review this article. Studies Within A Trial (SWAT) are important and a good opportunity to make optimal use of resources. Despite their disadvantage that the original trial has not been designed for the purpose of the SWAT, they provide important indications for further studies. In this sense, the results of this SWAT are important for further research.
General impression:
This SWAT was planned at the same time as the original trial. The methods are described in detail and are the strongest part of the article. Nevertheless, there are some important concerns. Participants were rewarded with a 5-pound note. This is in itself also an intervention to increase response rates. As the authors themselves note, the response rate is generally very high. Possibly the additional reward and together with the high response rate made it impossible to show an effect of the two tested interventions.
Abstract:
Background: Is it still true that postal questionnaires are used frequently in randomized controlled trials? One could imagine that a swift towards online surveys can be observed.
Background:
This very short background is the weakest part of the manuscript.
The rational for the SWAT differs between the abstract and the main text. The general problem of low response rate for the generalizability of results seems important also for the article, not only for the abstract.
Methods:
This is the strongest part of the article and mostly very clear.
It was a good idea to combine the results of this SWAT with results of other studies in a meta-analysis. However, it could be meaningful to provide further details about the literature research for the articles, which were included. There is not enough information to decide if the research was really systematic or not. Especially the comment that co-author personal knowledge of studies was considered for finding studies raises some questions. Did the included articles test the same interventions?
Results:
Results are presented in a clear way with nice tables. Especially Figure 1 which gives very good overviews about the study process.
Are differences between the study groups described in Table 2 significant or not?
Discussion:
The enclosed 5-pound note represents a further limitation and should be mentioned.
Is the work clearly and accurately presented and does it cite the current literature?
Yes
If applicable, is the statistical analysis and its interpretation appropriate?
Yes
Are all the source data underlying the results available to ensure full reproducibility?
Yes
Is the study design appropriate and is the work technically sound?
Partly
Are the conclusions drawn adequately supported by the results?
Partly
Are sufficient details of methods and analysis provided to allow replication by others?
Yes
Reviewer Expertise:
Midwifery, Epidemiology
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
JamesSophieUniversity of York, UK, UK
Competing interests: No competing interests were disclosed.
2092021
We thank the reviewer for the helpful comments and points raised. We hope that our responses are satisfactory and the article has been updated in response to the points raised from both reviewers (v2).
General impression:
This SWAT was planned at the same time as the original trial. The methods are described in detail and are the strongest part of the article. Nevertheless, there are some important concerns. Participants were rewarded with a 5-pound note. This is in itself also an intervention to increase response rates. As the authors themselves note, the response rate is generally very high. Possibly the additional reward and together with the high response rate made it impossible to show an effect of the two tested interventions.
Response: Thank you, these are valid points. Brueton et al., (2013) report that the addition of a monetary incentive was more effective than no incentive (RR 1.31; 95% CI 1.11 to 1.55, p=0.002) so the £5 monetary reward was included as standard in the OTIS trial at the 12 month time point. All participants received the £5 so this was not felt to be a limitation in assessing the effectiveness of the 2 interventions investigated by this SWAT, but it may have added to the high response rate and this is now noted in the discussion. At the time this SWAT was designed and added to the OTIS trial the high response rates were not known. The high response rate is mentioned in the discussion of the paper with a learning point for future SWATs, as we acknowledge that this limits the scope for the interventions to have a large benefit.
Abstract:
Background: Is it still true that postal questionnaires are used frequently in randomized controlled trials? One could imagine that a swift towards online surveys can be observed.
Response: We believe it is still true, indeed most of the trials we conduct in York Trial Unit still use postal questionnaires; however, we agree that the use of online data collection has increased since this SWAT was designed 4 years ago. With RCTs in this demographic/population and for participants in RCTs who do not have internet provision, in order to ensure inclusivity, there is likely to remain a need for future RCTs to still have a postal paper questionnaire option available and so keeps this relevant.
Background:
This very short background is the weakest part of the manuscript.
The rational for the SWAT differs between the abstract and the main text. The general problem of low response rate for the generalizability of results seems important also for the article, not only for the abstract.
Response: Thank you for this, we have updated the paper v2 to include this information from the abstract into the introduction to ensure consistency. We were limited by the word count of the article to be able to provide any further detail in this section.
Methods:
This is the strongest part of the article and mostly very clear.
It was a good idea to combine the results of this SWAT with results of other studies in a meta-analysis. However, it could be meaningful to provide further details about the literature research for the articles, which were included. There is not enough information to decide if the research was really systematic or not. Especially the comment that co-author personal knowledge of studies was considered for finding studies raises some questions. Did the included articles test the same interventions?
Response: All the studies included in the meta-analysis are now published and peer reviewed with the references provided so that these can be located and a summary of the key information of the studies included in the meta analysis is now provided in Supplementary file 13.
[Added to paper v2: The Cochrane systematic review search strategy (supplementary file 14) was used to search MEDLINE and EMBASE. Additionally previous systematic reviews references were hand searched, along with published retention research reference lists, conference papers and co-author personal knowledge of studies.]
Results:
Results are presented in a clear way with nice tables. Especially Figure 1 which gives very good overviews about the study process.
Are differences between the study groups described in Table 2 significant or not?
Response: Table 2 presents summary statistics for the baseline characteristics of the SWAT participants. According to the CONSORT statement, significance testing of baseline differences in randomized controlled trials should not be performed. Indeed, this practice has been discouraged by numerous authors.
(Reference for the above information = Michiel R de Boer, Wilma E Waterlander, Lothar DJ Kuijper, Ingrid HM Steenhuis, andJos WR Twisk(2015) Testing for baseline differences in randomized controlled trials: an unhealthy research behavior that is hard to eradicate. Int J Behav Nutr Phys Act. 2015; 12: 4.Published online 2015 Jan 24. doi: 10.1186/s12966-015-0162-z)
Discussion:
The enclosed 5-pound note represents a further limitation and should be mentioned.
Now mentioned in the discussion
10.5256/f1000research.26227.r66639Reviewer response for version 1DuncanEilidh1Refereehttps://orcid.org/0000-0002-1009-9732CoffeyTaylor1Co-refereeLawrieLouisa1Co-referee
Health Services Research Unit, University of Aberdeen, Aberdeen, UK
Competing interests: No competing interests were disclosed.
The present article by James
et al. details a study within a trial (SWAT) of including a pen and/or cover letter with a postal questionnaire. A two-by-two factorial randomized control trial was conducted that assessed the effect of including a branded pen, a ‘social incentive’ cover letter, both a pen and ‘social incentive’ cover letter, or a standard cover letter as control. The primary outcome measure was the return of a postal questionnaire with secondary outcomes of time to return the questionnaire, completeness of the questionnaire, requirement for a reminder letter to be sent, and overall cost effectiveness. Overall, there was no effect of any intervention on return of questionnaires and no effect on secondary outcome measures. A meta-analysis was conducted with respect to including a pen with the questionnaire and its effect on response rate, with pooled results supporting the inclusion of pens.
We would firstly like to commend the authors for presenting the methods clearly and providing supplementary material which enhances replicability. Below we indicate points which could be addressed and identify areas of improvement to further enhance the quality of this article.
Major points to address: The peer review team would like to highlight the following areas for clarification.
Potentially confounding incentives
The ‘social incentive’ intervention cover letter is described as including a “mild level of social incentive”. In referencing the supporting citation for the cover letter (Cotterill
et al., 2017)
1, the social incentive used is social pressure. Social pressure is described as conveying to participants that “information about their behavior will be noticed (Cotterill
et al., 2017)
1”. The present paper then describes the cover letter as “intended to highlight to the participant that their questionnaire responses are noted and valued”. It is apparent that reminder letters were also sent to some participants. A reminder letter could be argued to be another “mild level of social incentive” as it is similarly conveying to participants that their behaviour (i.e. not returning the questionnaire) is being noticed. Can the authors comment on the potential for the reminder letters to confound the results? The reminder is not included among the supplementary documents, but we would be interested in the authors’ thoughts on the potential for unintended social pressure cues or behaviour change techniques being delivered through the reminder letter.
In reviewing both the standard and social incentive cover letter, it is noted that they are identical apart from the personalized table indicating the participants’ previous responses. Similarly to the above point regarding unintended social pressure cues, we wonder if the phrases “We would be very grateful if you could return the completed questionnaire to us[…]” and “Your input to this trial is very important to us[…]”can be unintentionally introducing confounding social pressure by highlighting that their questionnaire responses are “noted and valued”. Can the authors reflect on how the addition of these phrases (and thereby the possibility that unintended ‘social incentives’ were applied within the control group) may have potentially affected results?
Under the section, “Interventions”, on page 3, it is mentioned that an unconditional £5 note was included with the questionnaire for all participants. Due to the financial incentive present, we would just like to clarify if the inclusion of the £5 note was controlled for in the analysis.
It is referenced in the “Discussion” section that the host trial included three other methodological SWATs that have a potential for contamination or interaction. If possible, it would be good to offer brief descriptions or links to other SWATS in the supplementary materials to better understand their potential to confound intervention effects.
Theoretical foundation of cover letter
Applying the Behaviour Change Techniques (BCT) taxonomy definitions to the intervention letter raises the possibility that the intervention being tested is more akin to “feedback on behaviour” (see Michie, 2013)
2, given that the letter provides informative feedback on the frequency of questionnaire return, rather than the BCT taxonomy definition of ‘social incentive’ (i.e. “Inform that a verbal or nonverbal reward will be delivered if and only if there has been effort and/or progress in performing the behaviour"). It is worth noting that some readers may interpret the meaning of 'social incentives' quite differently than how it is conceptualised here. Can the authors offer some clarification on this for the reader and on the theoretical underpinnings of the intervention? Adding citation(s) to clarify their conceptualisation of 'social incentive' would be helpful.
It is also unclear whether the authors regard the social incentive cover letter as a form of “social reward”: noted on page 4, the authors state that gender was controlled for within the logistic regression model to “control for potential differences in anticipation of social rewards…”. It does not appear that participants are presented with additional verbal or non-verbal reward above the standard letter with just the inclusion of the response table.
It is noted in Cotterill
et al. (2017)
1 that social pressure is “unlikely to be effective among those who are very committed to the behavior” which appears to be the case given that response rates across groups are >95%. Given that there are two prior postal response timepoints, was identifying if this group is highly committed to questionnaire response considered?
Minor points to address:
Regarding Table 2, are BMI, EQ-55D-5L score, and number of falls relevant to include? The table is already quite extensive and this data may detract the reader’s attention from other variables (e.g. age, sex) more pertinent to the study.
The pen is described as York Trials Unit branded. Was there branding specific to the host trial? It might be that participants would be quicker to recognize the trial versus the CTU. How might this affect the social incentive?
Could it be mentioned briefly in the main body of the paper that the meta-analysis was undertaken as part of PROMETHEUS? We notice it is mentioned in the acknowledgments but one of the review team members was confused about the inclusion of the meta-analysis.
Overall assessment: We would like to commend the authors on a concise and well-articulated presentation of the present study. The study is well designed and it is clearly presented with accompanying data and figures making it highly reproducible. We recognize the importance of each individual SWAT’s contribution to a larger body of evidence and the conclusions drawn and the recommendations for future work are appropriate. We have suggested potential limitations and concerns with the theoretical foundations of the social incentive cover letter and the presence of potential confounding incentives but our overall assessment is that the article meets its objectives and its strengths merit indexing.
Contribution to this peer review:
This peer review was conducted by Taylor Coffey, Dr Louisa Lawrie and Dr Eilidh Duncan as part of TC and LL’s training. All reviewers contributed to writing the report.
Is the work clearly and accurately presented and does it cite the current literature?
Yes
If applicable, is the statistical analysis and its interpretation appropriate?
Yes
Are all the source data underlying the results available to ensure full reproducibility?
Yes
Is the study design appropriate and is the work technically sound?
Yes
Are the conclusions drawn adequately supported by the results?
Yes
Are sufficient details of methods and analysis provided to allow replication by others?
Yes
Reviewer Expertise:
Health services research, design and evaluation of interventions to change behaviour; the application of behavioural theory to trial recruitment and retention.
We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above.
References:
The effect of using social pressure in cover letters to improve retention in a longitudinal health study: an embedded randomised controlled retention trial.
Trials.2017;18(1) :
10.1186/s13063-017-2090-510.1186/s13063-017-2090-5:
The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions.Ann Behav Med.2013;46(1) :
10.1007/s12160-013-9486-681-952351256810.1007/s12160-013-9486-6JamesSophieUniversity of York, UK, UK
Competing interests: No competing interests were disclosed.
2092021
Major points to address: The peer review team would like to highlight the following areas for clarification.
Potentially confounding incentives
The ‘social incentive’ intervention cover letter is described as including a “mild level of social incentive”. In referencing the supporting citation for the cover letter (Cotterill
et al., 2017)1, the social incentive used is social pressure. Social pressure is described as conveying to participants that “information about their behavior will be noticed (Cotterill
et al., 2017)1”. The present paper then describes the cover letter as “intended to highlight to the participant that their questionnaire responses are noted and valued”. It is apparent that reminder letters were also sent to some participants. A reminder letter could be argued to be another “mild level of social incentive” as it is similarly conveying to participants that their behaviour (i.e. not returning the questionnaire) is being noticed. Can the authors comment on the potential for the reminder letters to confound the results? The reminder is not included among the supplementary documents, but we would be interested in the authors’ thoughts on the potential for unintended social pressure cues or behaviour change techniques being delivered through the reminder letter.
Thank you to the reviewers for their valued comments.
For this paper the term ‘social incentive’ was used rather than ‘social pressure’ on the advice of two of the co-authors of this paper who were involved in the original Cotterill et al., (2017) embedded trial to test the social incentive cover letter intervention within a cohort study. They noted that following this paper the term ‘social pressure’ is not commonly used outside of politics literature.
A copy of the reminder letter that was sent to 83 participants has been added to the OSF as supplementary document 12.
Whilst it could be argued that the reminder letter provided another mild level of social incentive, it was not felt that it would provide additional social incentive as per the definition used for this study (discussed in relation to the next comment). Furthermore, the number of these sent out was evenly distributed across the randomised and analysed groups: 83 reminder letters sent were sent out of 755 and these were evenly distributed across the 4 groups (pen and social incentive cover letter group 19/184 (10.3%), control group 24/190 (12.6%), social incentive cover letter only group 20/189 (10.5%) and pen only group 20/192 (10.4%). So it would be unlikely that these additional reminder letters would confound the results.
There was no statistically significant evidence of a difference in the likelihood of participants requiring a reminder between the groups (pen vs no pen: adjusted OR = 0.89 [95% CI 0.56, 1.42 p = 0.63]; social incentive cover letter vs no social incentive cover letter: adjusted OR = 0.92 [95% CI 0.58, 1.47 p = 0.74].
In reviewing both the standard and social incentive cover letter, it is noted that they are identical apart from the personalized table indicating the participants’ previous responses. Similarly to the above point regarding unintended social pressure cues, we wonder if the phrases “We would be very grateful if you could return the completed questionnaire to us[…]” and “Your input to this trial is very important to us[…]”can be unintentionally introducing confounding social pressure by highlighting that their questionnaire responses are “noted and valued”. Can the authors reflect on how the addition of these phrases (and thereby the possibility that unintended ‘social incentives’ were applied within the control group) may have potentially affected results?
The standard text for this cover letter was the exact text used in the cover letters previously sent to participants with their 4 and 8 month questionnaires for the OTIS trial. The text of both the ‘control’ and ‘social incentive’ cover letters were identical the only difference was the addition of the table therefore it was not felt that this would affect the results. The example sentences noted could fall into some definitions for social incentive, but they would not meet ‘persuades people to behave in a certain way by the promise that their actions will be noticed or made public’ (Cotterill et al., 2017). Clarification of this definition has been added to the paper to help to clarify the comments around this.
Under the section, “Interventions”, on page 3, it is mentioned that an unconditional £5 note was included with the questionnaire for all participants. Due to the financial incentive present, we would just like to clarify if the inclusion of the £5 note was controlled for in the analysis.
Since all participants received the £5 it was neither necessary nor appropriate to control for this in the analysis.
It is referenced in the “Discussion” section that the host trial included three other methodological SWATs that have a potential for contamination or interaction. If possible, it would be good to offer brief descriptions or links to other SWATS in the supplementary materials to better understand their potential to confound intervention effects.
A supplementary file 11 has been added to OSF outlining the other SWATs that were undertaken within the OTIS Trial.
Theoretical foundation of cover letter
Applying the Behaviour Change Techniques (BCT) taxonomy definitions to the intervention letter raises the possibility that the intervention being tested is more akin to “feedback on behaviour” (see Michie, 2013)2, given that the letter provides informative feedback on the frequency of questionnaire return, rather than the BCT taxonomy definition of ‘social incentive’ (i.e. “Inform that a verbal or nonverbal reward will be delivered if and only if there has been effort and/or progress in performing the behaviour"). It is worth noting that some readers may interpret the meaning of 'social incentives' quite differently than how it is conceptualised here. Can the authors offer some clarification on this for the reader and on the theoretical underpinnings of the intervention? Adding citation(s) to clarify their conceptualisation of 'social incentive' would be helpful.
Although recent work has pointed towards the importance of using behaviour change theory to support and influence behavioural actions required by participants in RCTs (Gilles et al., 2018), behaviour change theory and associated research is such a vast area that this is a valid point that different concepts can be defined in different ways by different theories/authors. To help to clarify this in the paper sentences have been added to provide further detail and detail how the concept of social incentive was interpreted in relation to this study.
[added to paper v2: ‘The concept of social incentive that underpinned the intervention for this study was that a social incentive is something that persuades people to behave in a certain way by the promise that their actions will be noticed or made public [10].’
It is also unclear whether the authors regard the social incentive cover letter as a form of “social reward”: noted on page 4, the authors state that gender was controlled for within the logistic regression model to “control for potential differences in anticipation of social rewards…”. It does not appear that participants are presented with additional verbal or non-verbal reward above the standard letter with just the inclusion of the response table.
Error in terminology of the paper, social reward was used interchangeably with social incentive in error. Social reward removed from this sentence and replaced with social incentive with a further reference added.
[changed paper v2 to read: ‘gender, to control for potential difference in the effect of social incentives between males and females]
It is noted in Cotterill
et al. (2017)1 that social pressure is “unlikely to be effective among those who are very committed to the behavior” which appears to be the case given that response rates across groups are >95%. Given that there are two prior postal response timepoints, was identifying if this group is highly committed to questionnaire response considered?
Thank you, this is a valid point. At the time this SWAT was designed and added to the OTIS trial the high response rates of the previous time points were not known. This is currently noted in the paper as a consideration for future research to consider when using this type of social incentive cover letter.
Minor points to address:
Regarding Table 2, are BMI, EQ-55D-5L score, and number of falls relevant to include? The table is already quite extensive and this data may detract the reader’s attention from other variables (e.g. age, sex) more pertinent to the study.
These variables have been removed from Table 2 as suggested.
The pen is described as York Trials Unit branded. Was there branding specific to the host trial? It might be that participants would be quicker to recognize the trial versus the CTU. How might this affect the social incentive?
The pen had the logo of York Trials Unit and was not specific to the OTIS trial. This would be an interesting follow on question for future work, whether the branding of the pen sent with a questionnaire had any effect on response rate. The pen was not considered to provide a social incentive for this SWAT, only the cover letter.
Could it be mentioned briefly in the main body of the paper that the meta-analysis was undertaken as part of PROMETHEUS? We notice it is mentioned in the acknowledgments but one of the review team members was confused about the inclusion of the meta-analysis.
[Added to paper v2: ‘Due to SWATs typically being under-powered to show small effects, it is essential that the results are seen within the context of the wider literature (added to methods section under statistical analysis)
The results of this study were pooled with four previous SWATs [8,9,22,23] investigating the same intervention, with the same dichotomous outcome of response to the questionnaire or not. Pooled odds ratios and corresponding 95% CIs were calculated. Heterogeneity between trials was assessed using the Chi-squared and I 2 statistics. The meta-analysis was facilitated by the PROMoting THE USE of SWATs (PROMETHEUS) programme, which supports host trial teams to conducted SWATs and for data obtained to be collated and meta-analysed.’(Summary details for the studies included in the meta-analysis are provided in Supplementary file 13)]