Using pens as an incentive for questionnaire return in an orthopaedic trial: an embedded randomised controlled retention trial [version 2; peer review: 1 approved, 1 approved with reservations]

Background: We did a ‘study within a trial’ (SWAT), evaluating the effectiveness of the inclusion of a pen with a postal questionnaire, compared to no pen being included, on the retention rate in a large orthopaedic trial. Methods: The SWAT was embedded in the KReBS trial. The primary outcome was the proportion of 12-month questionnaires returned. Secondary outcomes were the proportion of questionnaires completed and time to questionnaire return. Binary data were analysed using logistic regression and time to return using Cox proportional hazards regression. Odds ratios (OR) and hazard ratios (HR) are presented, with associated 95% confidence intervals and p-values. Results: In total, 2305 participants were randomised into the SWAT. In the pen group, 1020/1145 (89.1%) of participants returned a questionnaire, compared to 982/1147 (85.6%) in the no pen group. The absolute difference in questionnaire return rate was 3.5% (95% CI: 0.8% to 6.2%; p=0.01). There were statistically significant differences in questionnaire return


Introduction
Recruitment and retention of participants have been identified as serious issues for randomised controlled trials (RCTs) 1,2 .
Incentives, both monetary and non-monetary, are used by UK clinical trials units as part of recruitment and retention strategies 3 .
In particular, the use of pens has recently been evaluated as a non-monetary incentive for recruitment 4 , while in 2016, Bell et al. assessed whether pens were effective in improving retention 5 . The study by Bell and colleagues indicated that the inclusion of a pen with postal follow-up questionnaires may increase return rates; however, the authors stated that there was a need to test the intervention in more diverse patient populations, and in addition the results were of borderline statistical significance In response to this uncertainty, we did a 'study within a trial' (SWAT) evaluating the effectiveness of the inclusion of a pen with a postal questionnaire, compared to no pen being included, on the retention rate in a large orthopaedic trial.

Design
This paper details the methods and results of a SWAT embedded within the prospectively registered KReBS RCT (ISRCTN87127065). KReBS evaluated the effectiveness of a two-layer compression bandage compared with a standard wool and crepe bandage applied post-operatively on patient-reported outcomes in total knee replacement patients 6 .

Participants
The SWAT was conducted in 26 NHS hospital trust sites and was implemented after the start of KReBS follow-up. All KReBS participants were eligible for this SWAT provided they were not deceased or withdrawn from follow-up before being due to be sent their 12-month follow-up postal questionnaire.

Intervention
Participants in the SWAT intervention group received a pen (branded with the York Trials Unit and University of York logos) with their 12-month questionnaire.
All SWAT participants received pre-planned retention strategies within KReBS. This consisted of a reminder letter and additional copy of the questionnaire if the participant had not returned a completed copy 4 weeks after sending out the original copy. If there was still no response following the postal reminder, participants were contacted by telephone to obtain the patient-reported outcomes.

Outcomes
The primary outcome was the proportion of participants who returned a 12-month questionnaire. Secondary outcomes were proportion of participants who completed the questionnaire and time to questionnaire return. A questionnaire was considered complete if the participant had answered 11 or more questions of the 12-item host trial primary outcome, the Oxford Knee Score 7 .

Sample size
Since this was an embedded trial, the sample size was determined by the number of participants in the main KReBS trial 6 , which aimed to recruit 2600 participants.

Randomisation
Participants were randomised into the SWAT in two batches, using a 1:1 allocation ratio, in a single large block the size of the batch. The SWAT was embedded at the 12 month time point and was only intended to include participants who had not died or expressly withdrawn from data collection up to this point. It was not logistically feasible to randomise participants in real time just before they were sent the questionnaire, and yet by randomising participants into the SWAT too early there was a risk of participants withdrawing from the trial before they were sent the 12 month questionnaire. We balanced minimising the time between randomisation into the SWAT and being sent the questionnaire, and the resources required to randomise participants, by choosing to randomise two large sets ('batches') of participants. Shortly before the first participant was due to be sent their 12 month questionnaire we randomised participants into the SWAT (i.e. the study statistician was sent a list of participant ids to allocate to receive a pen or not), then any remaining participants were included in the second batch later on. The allocation schedule for each batch was generated by a statistician at York Trials Unit using Stata v15 8 .

Blinding
Participants were not informed of their explicit participation in the SWAT, but due to the nature of the intervention could not be blinded to receipt (or not) of a pen with their questionnaire. Similarly, it was not possible to blind research staff to SWAT allocation.

Approvals
The SWAT was incorporated into the host trial protocol and approved as part of Substantial Amendment 2 by the Research Ethics Committee North East -Newcastle and North Tyneside on 13/04/2018. As the SWAT was deemed to be low risk, and to avoid disappointment for participants who did not receive

Amendments from Version 1
We have provided more information in the introduction justifying the need for this study. We have provided detail on the preplanned retention strategies used, and also given more detail on how participants were randomised into the study. We have also made minor amendments to the results in light of a duplicate randomisation that was found in the host trial, meaning that 2334 rather than 2335 participants were randomised to the host trial. This duplicate randomisation was provided with an allocation for the SWAT, and therefore the results of the SWAT have changed, however, these changes are negligible and do not affect the interpretation of the results. We have added a new version of the flowchart to take these changes into account. the additional incentive, informed consent was not obtained for participation in this SWAT.

Statistical analysis
Analyses were carried out using Stata v16.0 9 . A diagram detailing the flow of participants through the SWAT is provided, and baseline characteristics are presented by SWAT allocation.
Outcomes are summarised descriptively. Statistical tests were two-sided, used a 5% significance level, and were done on an intention to treat basis. All analyses (except the calculation of the absolute difference in return rate) used mixed effects, adjusting for SWAT allocation and host trial allocation as fixed effects and trial site as a random effect. The absolute difference in return rates was estimated using the two-sample test of proportions. Relevant parameter estimates are presented with associated 95% confidence intervals (CI) and p-values.
The proportion of participants who returned a 12-month questionnaire was analysed using logistic regression. Questionnaire completion was analysed in the same manner.
Time to questionnaire return was analysed using a Cox proportional hazards shared frailty model. Participants who did not return a questionnaire were censored at 90 days.

Results
In total, 2334 participants were recruited into the KReBS trial and 2305 were randomised into the SWAT ( Figure 1). The average age was 69.0 years and 55.2% were female (Table 1). A further 13 participants died or withdrew following randomisation and as a result, 1145 participants in the pen group, and 1147 in the no pen group, were sent a 12-month questionnaire and were included in the analysis.
In the pen group, 1020 (89.1%) of participants returned a questionnaire, compared to 982 (85.6%) in the no pen group ( Table 2). The absolute difference in return rate was 3.5% (95% CI: 0.8% to 6.2%; p=0.01). There was a statistically significant difference between the groups in the likelihood of returning a questionnaire (OR 1.36; 95% CI: 1.06 to 1.74; p=0.02), and also in the likelihood of returning a complete questionnaire (OR 1.40; 95% CI: 1.11 to 1.78; p<0.01). In addition, there was evidence of a reduction in time to return in favour of the pen group (HR 1.17; 95% CI: 1.07 to 1.27; p<0.01). See Underlying data for full, individual-level data 10 .

Discussion
There is strong evidence that the use of a pen as a nonmonetary incentive in the KReBS trial increased the proportion of questionnaires returned and completed, and also decreased the time to return. The decrease in time to return provides support to the findings of Bell et al. 5 . However, completion rate was calculated as a proportion of all SWAT participants rather than all SWAT participants who returned a questionnaire, and as a result questionnaire completion was highly correlated with questionnaire return. On the other hand, the large sample size of this SWAT means the results can be generalised to other orthopaedic trials.

Conclusion
This SWAT adds to the growing evidence base for whether pens are effective as an incentive for retention and indicates their potential effectiveness.

Data availability
Underlying data Open Science Framework: Underlying data and CONSORT checklist for Using pens as an incentive for questionnaire return in an orthopaedic trial: an embedded randomised controlled retention trial. https://doi.org/10.17605/OSF.IO/ BEPN3 10 .
This project contains the underlying data in CSV, SAS and V8XPT formats, alongside a data key.

Open Peer Review
Summary of the article: The article reports the results of a SWAT within the KReBS trial. The SWAT aimed to investigate the effectiveness of the inclusion of a pen with a postal questionnaire compared to no pen being included on the questionnaire return rate in a large orthopaedic trial. The results provide evidence for a higher questionnaire return rate, higher questionnaire completion rate, and shorter time until questionnaire return with the inclusion of a pen.

Abstract:
For the conclusion I would suggest being more precise by rephrasing it as follows: "This SWAT adds to the growing evidence base for whether pens are effective as an incentive for returning a postal questionnaire and participant retention, and indicates their potential effectiveness."

Introduction:
Did the authors conduct a systematic search for similar SWATs in terms of research question? If yes, please provide search details in an appendix. If no, please do so or explain why you think this is not necessary. In the SWAT store there is SWAT 138, for instance, which could/should also be mentioned in the introduction.

Methods:
Outcomes: Please clarify whether the primary outcome includes also questionnaires that were completed over the phone, because there was no response from participants otherwise. In addition, I think it would be of interest for readers to know how many participants in each group (pen vs no pen) needed a reminder letter, and how many needed to be contacted by telephone to obtain patient-reported outcomes. This could be done in post-hoc analyses and would reveal something about how/at which level the intervention worked.

Results:
Please see my suggestion above for some additional analyses.

Discussion:
I think in terms of generalizability it should be clearly acknowledged that the results are generalizable to other Orthopaedic trials with respect to an increased return rate of a postal questionnaire, but not necessarily in generally increasing retention of participants in orthopaedic trials. If participants need to come to the clinic for a follow-up visit instead of filling a questionnaire, then sending a pen might not be useful.

Conclusion:
Based on what I said in the paragraph above, I suggest rephrasing the conclusion as for the abstract -just to make it clear that results cannot automatically extrapolated to any form of participant retention.

If applicable, is the statistical analysis and its interpretation appropriate? Partly
Are all the source data underlying the results available to ensure full reproducibility? Yes

Are the conclusions drawn adequately supported by the results? Partly © 2021 Shiely F. This is an open access peer review report distributed under the terms of the Creative Commons
Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

HRB Clinical Research Facility and School of Public Health, University College Cork, Cork, Ireland
Thank you to the authors for their considered revision of this SWAT, which has given clarity on the methods which will allow replication.

If applicable, is the statistical analysis and its interpretation appropriate? Partly
Are all the source data underlying the results available to ensure full reproducibility? Partly

Introduction Section
In the introduction section, you mention the Bell

Randomisation
What do you mean by two batches, and why were there two batches?

Conclusion
Your conclusion is not supported by your results. Your conclusion should be that pens are effective as an incentive for returning questionnaires in postal studies, and also effective at increasing the time to return. You did not measure retention, as this was a postal questionnaire and not a follow-up of the patients at the centre. It's a stretch to extrapolate that filling out a questionnaire and returning it is a measure of retention in a clinical trial. I would be interested to know how many of those that received the pen return to the CTU for the main trial for their next appointment. Please modify your conclusions.

Is the work clearly and accurately presented and does it cite the current literature? Partly
Is the study design appropriate and is the work technically sound? Yes

Are sufficient details of methods and analysis provided to allow replication by others? Partly
If applicable, is the statistical analysis and its interpretation appropriate? Partly al. Please amend.
The results from Bell et al. were uncertain in that the primary outcome of questionnaire return was of borderline statistical significance, and the authors indicated that research across a range of participant groups was required.
All SWAT participants received pre-planned retention strategies within KReBS. This sentence is left hanging there with no further explanation. What were these retention strategies? When you say all SWAT participants received these retention strategies, do you mean the SWAT comparator group also? IF so, why is this sentence under the "intervention" heading? It implies that you are speaking about the intervention group only. Perhaps you'd be better saying both SWAT intervention and SWAT comparator groups received pre-planned retention strategies if that is what you meant. Please qualify.
Thank you for making this point. We can confirm that SWAT participants in both the intervention and comparator groups received pre-planned retention strategies. Participants were sent a reminder letter and a further copy of the questionnaire if they had not returned a completed copy 4 weeks after sending. If there was still no response following the postal reminder, participants were contacted by telephone to obtain the patient reported outcomes. We accept your point that the theory of how the inclusion of a pen influences completion rates is not immediately obvious. If a participant were to use their own pen, they may have had to spend some time trying to find one. This may lead to the participant becoming frustrated with the questionnaire, and thus decrease the probability of them completing it correctly. We hoped that by including a pen with the questionnaire, we would make the process easier for the participant.
We disagree that the completion rate should be removed as an outcome as this was prespecified in the published KReBS protocol. While we agree with your point on its limitations, we have already outlined these in the discussion section, and believe this is a more scientifically sound approach than removing it from the paper.

What do you mean by two batches, and why were there two batches?
This SWAT was embedded at the 12 month time point and was only intended to include participants who had not died or expressly withdrawn from data collection up to this point.
It was not logistically feasible to randomise participants in real time just before they were sent the questionnaire, and yet by randomising participants into the SWAT too early we risked participants withdrawing from the trial before they were sent the 12 month questionnaire. We balanced minimising the time between randomisation into the SWAT and being sent the questionnaire, and the resources required to randomise participants, by choosing to randomise two large sets ('batches') of participants. Shortly before the first participant was due to be sent their 12 month questionnaire we randomised participants into the SWAT (i.e. the study statistician was sent a list of participant ids to allocate to receive a pen or not), then any remaining participants were included in the second batch later on.
Your conclusion is not supported by your results. Your conclusion should be that pens are effective as an incentive for returning questionnaires in postal studies, and also effective at increasing the time to return. You did not measure retention, as this was a postal questionnaire and not a follow-up of the patients at the centre. It's a stretch to extrapolate that filling out a questionnaire and returning it is a measure of retention in a clinical trial. I would be interested to know how many of those that received the pen return to the CTU for the main trial for their next appointment. Please modify your conclusions.
Thank you for this comment, however we think we are justified in retaining use of the term 'retention' in the context of this trial. There were no clinic or YTU follow up appointments in the KReBS trial; it exclusively used postal follow up questionnaire to obtain outcome data. The primary outcome in KReBS was the participant reported Oxford Knee Score collected at 12 months. Therefore, we used a broader definition of retention i.e. the return of outcome data via any means. We believe this blog post by Shaun Treweek and Katie Gillies shows that this is in line with the definition of retention used in the SWAT community ( https://blogs.biomedcentral.com/on-medicine/2017/09/01/trial-retention-its-time-forcinderella-to-go-to-the-ball/ ).

Competing Interests:
No competing interests were disclosed.

Comments on this article Version 1
Reviewer Response 25 Aug 2020 Sarah Cotterill, University of Manchester, Manchester, UK This is a well-designed study and it has been written up very clearly. It follows on from a couple of other studies on this topic, and confirms those results, finding that the addition of a pen is an effective way of increasing response to questionnaire sent to participants in a clinical trial.
Is it appropriate to describe a pen that is sent with a questionnaire as an 'incentive'? According to the Behaviour Change Technique Taxonomy, an incentive is a where someone is promised in advance that 'money/vouchers or other valued objects will be delivered if and only if there has been effort and/or progress in performing the behaviour' 1 . In this experiment, the pen was provided to all participants, regardless of whether or not they performed the behaviour (return of the questionnaire), and perhaps that would be better described, not as an incentive, but rather as something else, such as 'adding objects to the environment', which is defined as 'Add objects to the environment in order to facilitate performance of the behaviour.' (Examples from the BCTT: Provide free condoms to facilitate safe sex; Provide attractive toothbrush to improved tooth brushing technique). Competing Interests: I have previously co-authored a paper with one of this paper's authors The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias • You can publish traditional articles, null/negative results, case reports, data notes and more • The peer review process is transparent and collaborative • Your article is indexed in PubMed after passing peer review • Dedicated customer support at every stage • For pre-submission enquiries, contact research@f1000.com