Enclosing a pen to improve response rate to postal questionnaire: an embedded randomised controlled trial

Background: Poor response to questionnaires collecting outcome data in randomised controlled trials (RCTs) can affect the validity of trial results. The aim of this study within a trial (SWAT) was to evaluate the effectiveness of including a pen with a follow-up postal questionnaire on response rate. Methods: A two-armed RCT was embedded within SSHeW (Stopping Slips among Healthcare Workers), a trial of slip-resistant footwear to reduce slips in NHS staff. Participants were randomised 1:1 to receive a pen or no pen with their follow-up questionnaire. The primary outcome was the proportion of participants who returned the questionnaire. Secondary outcomes were: time to response, completeness of response, and whether a postal reminder notice was required. Data were analysed using logistic regression, linear regression and Cox proportional hazards regression. Results: Overall, 1466 SSHEW trial participants were randomised into the SWAT. In total, 13 withdrew from the host trial before they were due to be sent their follow-up questionnaire, 728 participants received a pen with their questionnaire, and 725 did not receive a pen. A questionnaire was returned from 67.7% of the pen group and 64.7% of the group who did not receive a pen. There was no significant difference in return rates between the two groups (OR 1.15, 95% CI 0.92 to 1.43, p=0.22), nor level of completeness of the questionnaires (AMD -0.01, 95% CI 0.06 to 0.05, p=0.77). There was weak evidence of a reduction in the proportion of participants requiring a reminder and in time to response in the pen group. Conclusion: Inclusion of a pen with the follow-up postal questionnaire sent to participants in the SSHeW trial did not statistically significantly increase the response rate. These results add to the body of evidence around improving response rates in trials. Trial registration: ISRCTN 33051393 (for SSHEW). Registered on 14/03/2017.


Introduction
Randomised controlled trials (RCTs) are key to evaluating the effectiveness of interventions and often use postal questionnaires to collect outcome data. However, low response rates can limit the validity of the trial findings by reducing the power of the study and introducing bias 1 .
Numerous strategies to increase response rates have been studied 2,3 including sending a pen with the questionnaire. The pen acts both as a facilitator to aid completion of the questionnaire, and an incentive to return it 4,5 . The effectiveness of this intervention is equivocal with some studies reporting an increase in response rate 5-7 whilst others failed to show a positive impact 4,8 . These studies displayed considerable heterogeneity and only two were embedded in RCTs 6,7 . A Study within a Trial (SWAT) is a self-contained study embedded within a host trial that can be used to evaluate strategies designed to improve trial efficiency 9 . This SWAT evaluated the effectiveness of enclosing a pen with a follow-up postal questionnaire on response rates in the SSHeW trial 10 .

Design
This two-armed RCT was embedded in the SSHeW trial, a trial evaluating the effectiveness of slip-resistant footwear to reduce slips in NHS staff 10 . The SSHeW trial was registered (ISRCTN 33051393) and the trial protocol has been published 10 .

Participants
The SWAT was conducted in seven NHS Trusts in England and included all eligible participants in the SSHeW trial who were due to be sent their 14-week postal questionnaire between 04.07.2018 and 12.02.2019.

Intervention
The intervention group were sent a York Trials Unit, University of York branded pen with their questionnaire. The control group did not receive a pen.

Outcomes
The SWAT outcomes are outlined in Table 1.

Sample size
As is usual with an embedded trial, a formal sample size calculation was not undertaken as the sample size was determined by the number of participants due to receive their 14-week questionnaire.
We anticipated that randomising 2,000 participants into the SWAT would provide 80% power to detect an absolute difference of 6% (two-sided α=0.05) in response rates between the two groups, assuming a control rate of 60%.

Randomisation
Participants were allocated to either the intervention (pen) or control (no pen) group using simple randomisation in a 1:1 ratio. The allocation sequence was generated by the SSHeW trial statistician, who was not involved in sending out the questionnaires.

Blinding
Participants were not aware of their involvement in this SWAT but due to the nature of the intervention participants and study team members could not be blinded to group allocation.

Approvals
This SWAT was approved by the Department of Health Sciences Research Ethics Committee at the University of York and the Health Research Authority (HSRGC/2016/187/A).

Statistical analysis
Data were analysed using Stata version 15 11 on an intentionto-treat basis, using two-sided tests at the 5% significance level. The models used for each outcome are given in Table 2, the values associated with the pen allocation from each model is presented with its 95% confidence interval and p-value. All models were adjusted for main trial group allocation (slip-resistant footwear or wait-list control) and pen sub-study allocation (pen or no pen).

Costing
The total cost of a standard SSHeW questionnaire pack was £2.42 (envelope and postage: £0.86; questionnaire and cover Completeness of response Continuous (0-5) Number of completed responses to 5 key questions on the 14-week questionnaire.
Reminder notice sent Binary (sent/not sent) Proportion of participants sent a reminder questionnaire (sent three weeks after the initial questionnaire if no response had been received, no additional pens were sent with reminders).

Cost Continuous
Consideration of cost effectiveness of pen inclusion

Results
A total of 1466 participants were included in the SWAT (pen, n=733; no pen, n=733). In total, 13 participants withdrew from the main SSHeW trial after they had been randomised into the SWAT but before being sent their follow-up questionnaire, leaving 1453 participants ( Figure 1). Baseline characteristics are summarised descriptively in Table 3.
Results are presented in Table 4 (Figure 2), and in the proportion of participants requiring a reminder (OR 0.83, 95% CI 0.68 to 1.02, p=0.08).

Costing
A 3% difference in questionnaire response rate and an absolute difference in the percentage of participants who required a reminder of 1.1% were found. Considering these to be true effects, in order to receive one additional questionnaire, 33 participants would have to be sent a pen, at a cost of approximately 33x32p=£10.56. Approximately 91 participants would need to be sent a pen to prevent one reminder mailing and therefore to save £2.42. Hence, roughly one reminder is required per three retained participants, and the cost per retained participant is approximately £10.

Discussion
Whilst the results of all outcomes in this SWAT favoured the pen group, we found that the addition of a pen did not statistically significantly increase the response rate to, or completeness of, a follow-up questionnaire sent at 14 weeks post-randomisation among participants of the SSHeW trial. There was some evidence  It may be that, in this group of participants, the pen failed to act as a facilitator or was not a sufficient incentive to return the questionnaire, given the fact that participants in the trial already received a free pair of shoes (although offer of shoes was not conditional on returning the questionnaire).
However, the trial ultimately only had about 40% power to detect a difference of 3% in response rates (from 64.7 to 67.7%) and is therefore at risk of a type II error. Another potential weakness is that, due to the select population of healthcare workers, the results may not be generalisable to other populations or contexts.
The strength of this study is that it was a randomised trial.

Conclusion
This SWAT suggests that enclosing a pen in a questionnaire mail out may be an effective method to increase response rates but was likely underpowered to detect a statistically significant difference of the 3% observed. Since pens are inexpensive, even a small difference is likely to be cost-effective. The results contribute to the body of evidence regarding this intervention and may be included in future meta-analyses to improve power.

Karen Bracken
NHMRC Clinical Trials Centre, The University of Sydney, Sydney, NSW, Australia This paper describes the result of a SWAT embedded within the SSHeW trial. The SWAT aimed to evaluate whether including a pen with a follow-up postal questionnaire would impact questionnaire response rate. The authors indicate that previous studies have yielded mixed results. The results of this SWAT add to the body of existing evidence. The paper is clear and concise and addresses a common problem faced by trials; low response rates to questionnaires. The paper provides adequate detail for other trialists to replicate the intervention within their own trial. The inclusion of cost analyses is particularly useful. I was interested to see that the cost of including the pen was very low and would have liked an explanation of the components of the cost (cost of pen, additional cost of postage and posting materials). Like many SWATs, this SWAT was unavoidably constrained in sample size by the host trial, making conclusions difficult due to lack of power. It is unclear from the paper whether the authors expected that the sample size of 1466 was sufficient to detect an operationally important difference between groups. If not, then I suggest the conclusion be updated to make this clear, rather than stating that a statistically significant increase was not found. Would the authors suggest that further SWATs are needed in this area based on the Trial Forge guidance? (https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-019-3980-5 1 ) Also, I suggest removing the observed power calculation from the discussion (https://lesslikely.com/statistics/observed-power-magic/ provides an explanation as to why observed power calculations should be avoided).
The current host trial was Stopping Slips among Healthcare Workers (SSHeW), a trial testing slipresistant footwear with NHS Staff to reduce slips at work. The SWAT intervention was to include a pen with the follow-up questionnaire to test whether participants who received a pen were more likely to return their questionnaire than those participants who did not receive a pen. The authors also looked at whether inclusion of a pen with the questionnaire affected the time taken to send back the questionnaire; how complete they questionnaire was answered and; whether a reminder had to be sent for return of the questionnaire. The SWAT was run on the 14 week post randomisation follow-up questionnaire. All SSHEW participants were randomised into the SWAT (n=1466). There were no significant difference in return rates between the two groups but weak evidence that including a pen might make participants return the questionnaire sooner and less will need a reminder.
This study reports findings contributing to the evidence base for how trial teams can increase questionnaire response rates in their trials, using the gold standard randomised comparison. This is a hugely important area as low response rates affect the conclusions that can be drawn from trial findings. Unfortunately, low questionnaire response rates are something which continues to affect a lot of trials.
We have a few comments which should be addressed: ABSTRACT No comments.

INTRODUCTION
It would be interesting to know if the authors expected there to be an issue with the questionnaire return rate in the host trial and what was the basis of this expectation (e.g. low response rate to previous questionnaires)? Please consider adding what the expected response rate for the host trial was. This could be added in the introduction or methods according to the authors preference. Commonly, SWATs are reactive so some background information to the approach in this SWAT would provide a helpful context.
The authors mention that the pen would act as facilitator and incentive, it would be helpful to have any reference to this use as a behavioural motivator or barrier.

METHODS
It would be interesting to know if the investigators found any issues with posting a pack including a pen, eg additional bulk causing any issues with standard franking machines. Did any changes have to be made to the process of posting out to include the pen?

RESULTS
No comments.

DISCUSSION
Please include some consideration as to whether the timing of the incentive to return questionnaires might have affected the findings. In many trials participants are followed-up for two years or longer and that is the point when it gets really tough to keep the response rate high (reviewers' personal experiences). It would be really valuable to have the authors thoughts on this for trial teams looking for help.
Paragraph 3 -The authors list a lack of power as a limitation. We'd suggest highlighting that many SWATs will be underpowered because they are generally unable to change the size of the host trial; they are made for meta-analysis really.

CONCLUSION
The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias • You can publish traditional articles, null/negative results, case reports, data notes and more • The peer review process is transparent and collaborative • Your article is indexed in PubMed after passing peer review • Dedicated customer support at every stage • For pre-submission enquiries, contact research@f1000.com