Previously titled: SCIMITAR+ Trial: A randomised study within a trial (SWAT) of a contingent financial reward to improve trial follow-up

Objectives To evaluate the effectiveness of a contingent financial incentive (£10 note in addition to a routinely provided £10 voucher) versus no contingent financial incentive, on improving the retention rate in a randomised controlled trial (RCT). Methods A two arm ‘Study within a Trial’ (SWAT) embedded within a host RCT (SCIMITAR+). Participants were randomised to the SWAT using a 2:1 (intervention:control) allocation ratio. The primary outcome measure was the proportion of participants completing a CO breath measurement at the first SCIMITAR+ follow up time point (6 months). Secondary outcomes were withdrawing from follow-up after contact and time from assessment due date to completion.  Analyses were conducted using logistic or Cox Proportional Hazards regression as appropriate. Results A total of 434 participants were randomised into this SWAT. Completion of the CO breath measurement at 6 months was 88.5% (n=247) in the intervention arm of the SWAT and 85.4% (n=123) in the control arm (Difference 3.1%, OR 1.29, 95% CI 0.71-2.33, p=0.41). There was also no evidence of a difference in the proportion of participants withdrawing from follow-up after contact (intervention n=7 (2.5%), control n=5 (3.5%); OR 0.76, 95% CI 0.23-2.44, p=0.64), nor in terms of proximity of 6-month visit completion to due date (HR 1.07, 95% CI 0.86-1.33, p=0.55). Conclusion It is unclear if contingent financial incentives increased rates of faceOpen Peer Review


v2 Abstract Objectives
To evaluate the effectiveness of a contingent financial incentive (£10 note in addition to a routinely provided £10 voucher) versus no contingent financial incentive, on improving the retention rate in a randomised controlled trial (RCT). Methods A two arm 'Study within a Trial' (SWAT) embedded within a host RCT (SCIMITAR+). Participants were randomised to the SWAT using a 2:1 (intervention:control) allocation ratio. The primary outcome measure was the proportion of participants completing a CO breath measurement at the first SCIMITAR+ follow up time point (6 months). Secondary outcomes were withdrawing from follow-up after contact and time from assessment due date to completion. Analyses were conducted using logistic or Cox Proportional Hazards regression as appropriate.

Conclusion
It is unclear if contingent financial incentives increased rates of face-

Reviewer Status
Invited Reviewers

Amendments from Version 1
Changes have been made to the manuscript as follows: -Manuscript title amended to make clear this manuscript relates to the SWAT and not the SCIMITAR+ Trial in the main which may have been implied by the original wording.
-Two P values were inadvertently included in the Abstract Results section. The incorrect P value has been removed.
-Given the limited power of SWATs, reference to statistical significance may be misleading. All references to 'statistical significance' have been amended to reflect the lack of clarity around the estimate due to wide confidence intervals.
-Addition of further justification of SWAT in Introduction to make clear the evidence base on which this SWAT builds.
-Revision of 'reward' to 'incentive' throughout the manuscript for consistency.
-Addition of clarification on randomisation process (Methods -Randomisation; Results), to justify exclusion of some participants post randomisation -Addition of detail in relation to proportional hazards assumptions assessment in relation to secondary outcome (proximity to due date).

Introduction
Attrition is a major problem for randomised controlled trials (RCTs) with 25% experiencing more than 10% attrition 1 . Contingent incentives have shown to be effective in increasing questionnaire response rate, but have not been tested in relation to face-to face-visits. This SWAT evaluated the effectiveness of a contingent financial incentive -£10 cash in addition to a routinely provided £10 voucher -versus no contingent financial incentive, on improving the retention rate in the SCIMITAR+ trial.

Design
This SWAT was embedded within the SCIMITAR+ RCT which evaluated the effectiveness of a bespoke, individuallytailored, smoking cessation programme, compared to usual care, for adult smokers with severe mental ill health conditions 4 . The SCIMITAR+ Trial was registered prospectively: ISRCTN72955454 This paper refers to the methods and results of the SWAT only.

Participants
The SWAT 5 was conducted in 21 NHS Trusts and 16 primary care settings and was implemented after the start of SCIMITAR+ follow-up. Participants were eligible for this SWAT if they reached the SCIMITAR+ 6-month follow-up on or after 31 st September 2016.

Intervention
When participants in the SWAT intervention group were contacted by the research team to arrange their follow-up appointment, they were advised of the potential of receiving £10 cash contingent on providing a carbon monoxide (CO) breath measure as part of their 6-month face-to-face study appointment, in addition to the £10 gift voucher routinely provided to all participants. Participants in both groups received all other pre-planned retention strategies within SCIMITAR+.

Outcomes
The primary outcome for the SWAT was the proportion of participants completing a CO breath measurement at the SCIMITAR+ 6 month follow-up time-point. Secondary outcome measures were: i) the proximity of visit completion to visit due date; ii) the proportion of participants withdrawing from follow-up in the two months after initial contact was made to arrange the 6-month visit.

Sample size
The sample size was determined by the number of participants followed-up at 6 months in SCIMITAR+ from the point at which this SWAT was embedded.

Randomisation
Simple randomisation using random numbers was carried out by an independent statistician at the York Trials Unit using Stata v13 6 . All potentially participants eligible for inclusion, at the time of randomisation, were allocated with a 2:1 allocation ratio (intervention:control) due to the anticipated effectiveness of financial incentives increasing questionnaire response rates.

Blinding
It was not possible to blind research staff to the participant's allocation. Participants were not informed about the SWAT so were blind to the study hypothesis.

Approvals
The SWAT was approved by the Research Ethics Committee Yorkshire and Humber -Leeds East (15/YH/0051). As the SWAT was deemed to be low risk, and to avoid disappointment for participants who did not receive the additional incentive, informed consent was not obtained for participation in this SWAT.

Statistical analysis
Analyses were conducted using Stata v15 7 on an intention to treat basis using two-sided statistical tests at the 5% significance level, adjusting for host trial allocation.
The proportion of participants who provided a 6-month CO breath measure was analysed using logistic regression. The odds ratio (OR), 95% confidence interval (CI) and p-value are presented.
The 6-month appointment due date was 183 days after randomisation. Participants who withdrew a month either side of the 6-month appointment due date were classed as withdrawn. The proportion of participants withdrawing from SCIMITAR+ in the two months after contact were analysed in the same way as the primary outcome.
A Cox Proportional Hazard model compared the proximity of the visit completion to visit due date (time in days). Participants who completed their visit before or on the due date had their time-to-visit set to 0.1.

Results
In total, 434 participants were randomised into this SWAT (n=286, 65.9% intervention group; n=148, 34.1% control group). Due to randomisation occurring at a single time point, eleven participants were excluded from analysis as they withdrew from SCIMITAR+ following randomisation but prior to being contacted for their 6-month visit. There were 423 eligible participants (intervention group n=279, 66.0%; control group n=144, 34.0%) ( Figure 1).
Overall, 87.5% (n=370) of participants completed the CO breath measurement at 6 months and it was unclear if there was a difference between intervention (88.5%, n=247) and control groups (85.4%, n=123) (3.1% difference, OR 1.29, 95% CI 0.71-2.33, p=0.41). It was unclear if there was any difference in withdrawals between trials arms (intervention n=7, Unable to receive intervention allocation (withdrew before 6-month visit) n=7 Unable to receive control allocation (withdrew before 6-month visit) n=4 Able to receive intervention allocation n=279 Able to receive control allocation n=144 Did not complete 6-month CO breath measure n=32 Did not complete 6-month CO breath measure n=21 Withdrawal within 2 months of contact n=7 Withdrawal within 2 months of contact n=5 Completed 6-month CO breath measure n=247 Completed 6-month CO breath measure n=123 Analysed

Discussion
It is unclear whether an additional £10 in cash increases the likelihood of participants completing a face-to-face followup, the proportion of the participants withdrawing, or have an effect on the proximity of the visit to the due date.

Strengths and limitations
A small positive difference was observed; however, despite the large sample size, the study was underpowered to confidently rule out a small 'true' effect. Due to the small effect size (3.1% increase in response) the cost per additional person attending would be in excess of £300.
Due to the sample size of this SWAT, it is most likely generalisable to the larger host trial population of patients with severe mental ill health disorders.
Data was not collected on how study staff followed the guidance on discussing the contingent £10 note to intervention group participants when arranging follow up visits. This may have diluted the effect of the intervention.

Conclusion
It is unclear if contingent financial incentives increase rates of face-to-face follow-up completion in this trial. However, there were sample size and power limitations. Future SWATs are needed to add to the evidence base. This is a paper about the financial incentive SWAT and not the SCIMITAR+ trial -the title seems to suggest that this is the trial report for the SCIMITAR+ trial. Something like 'The effect of contingent financial reward to improve retention; A randomised study embedded within the SCIMITAR+ trial' might be better.

2.
This SWAT has minimal power to answer a research question by itself, but as the authors rightly say it is an important addition to the body of evidence. I find the term 'not statistically significant' potentially misleading because it seems to suggest that the intervention wasn't effective whereas the confidence intervals are very wide, and appear consistent with previous results. I'd recommend removing reference to 'statistical significance' and instead using terms like 'it is unclear whether….', given the very wide confidence intervals. 3.
The description 'This SWAT evaluated the effectiveness of a contingent financial reward -£10 cash in addition to a routinely provided £10 voucher -versus no contingent financial reward, on improving the retention rate in the SCIMITAR+ trial' is difficult to understand. It seems like the intervention is the offer of a contingent financial reward rather than the reward itself (which happens after outcome measurement) -should this perhaps be 'the effectiveness of the offer of a contingent financial reward'? The £10 voucher happens in both arms so perhaps this should say 'versus the routinely rewarded £10 voucher alone'unless I'm misunderstanding what the control group is here.

4.
It says 'contingent financial reward' in some places and 'financial incentive' in others -are these the same thing? It will be helpful if one term were used consistently.

5.
It says that analyses were 'on an intention to treat basis' but 11 randomised participants that withdrew before the 6-month visit were excluded from the analysis; this is not really compatible with a true 'intention to treat' analysis which aims to preserve randomised groups. Perhaps clearly describe the analysis population rather than calling it 'intention to treat' -e.g. the analysis population was participants randomised to the SWAT who had not withdrawn by time X days before due visit. Consider a sensitivity analysis including all randomised participants.

6.
The observed effect size was 3.1%; there is lots of uncertainty in this estimate so ideally it shouldn't really be used to infer the likely cost effectiveness of future studies -instead, I'd recommend considering each end of the confidence interval (or perhaps the upper end as a best-case scenario) when considering cost per additional person retained.

7.
I don't understand the sentence 'Due to the sample size of this SWAT, it is most likely generalisable to the larger host trial population of patients with severe mental ill health disorders.' A large sample size does not guarantee generalisability, and as noted the sample size is not really large enough to draw conclusions about this population. I think this point needs to be constructed more clearly.

8.
For the secondary outcome of 'proximity to due date', I'm really interested to know whether 9. the proportional hazards assumption holds. It would be helpful to include a Kaplain-Meier curve and/or some comment about assumption checking.

If applicable, is the statistical analysis and its interpretation appropriate? Partly
Are all the source data underlying the results available to ensure full reproducibility? Yes

Are the conclusions drawn adequately supported by the results? Partly
Competing Interests: No competing interests were disclosed.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
Author Response 15 Jun 2020 Catherine Arundel, University of York, UK, York, UK There seems to be an error in this sentence in the abstract -'The difference (3.1%) was not statistically significant (p=0.36; OR 1.29, 95% CI 0.71-2.33, p=0.41).' -two different p-values are reported and the incorrect one should be removed.

We thank the reviewer for this observation. The error has been corrected accordingly
This is a paper about the financial incentive SWAT and not the SCIMITAR+ trial -the title seems to suggest that this is the trial report for the SCIMITAR+ trial. Something like 'The effect of contingent financial reward to improve retention; A randomised study embedded within the SCIMITAR+ trial' might be better. We thank the reviewer for this observation. The title has been amended for clarity.
This SWAT has minimal power to answer a research question by itself, but as the authors rightly say it is an important addition to the body of evidence. I find the term 'not statistically significant' potentially misleading because it seems to suggest that the intervention wasn't effective whereas the confidence intervals are very wide, and appear consistent with previous results. I'd recommend removing reference to 'statistical significance' and instead using terms like 'it is unclear whether….', given the very wide confidence intervals. We agree with the reviewers thoughts here. The manuscript has therefore been updated to remove 'statistical significance' and to make clear the lack of certainty. P values have however been retained for completeness.
The description 'This SWAT evaluated the effectiveness of a contingent financial reward -£10 cash in addition to a routinely provided £10 voucher -versus no contingent financial reward, on improving the retention rate in the SCIMITAR+ trial' is difficult to understand. It seems like the intervention is the offer of a contingent financial reward rather than the reward itself (which happens after outcome measurement) -should this perhaps be 'the effectiveness of the offer of a contingent financial reward'? The £10 voucher happens in both arms so perhaps this should say 'versus the routinely rewarded £10 voucher alone'unless I'm misunderstanding what the control group is here. We have amended the manuscript in line with the reviewer's suggestion to make clearer the control group.
It says 'contingent financial reward' in some places and 'financial incentive' in others -are these the same thing? It will be helpful if one term were used consistently. We apologise for this inconsistency. The manuscript has been updated throughout to ensure consistency, It says that analyses were 'on an intention to treat basis' but 11 randomised participants that withdrew before the 6-month visit were excluded from the analysis; this is not really compatible with a true 'intention to treat' analysis which aims to preserve randomised groups. Perhaps clearly describe the analysis population rather than calling it 'intention to treat' -e.g. the analysis population was participants randomised to the SWAT who had not withdrawn by time X days before due visit. Consider a sensitivity analysis including all randomised participants. We can see why the reviewer feels that this is not strictly ITT. The study randomisation was done in a single batch, rather than when the participant reached the Month 6 time point, and so participants received an allocation when they were not technically eligible for the SWAT, hence the exclusion.
As a result, we suggest that the analysis in the paper is ITT in the sense that the analysis includes everyone who would have been eligible for the SWAT. We have sought to make this clearer in the paper for avoidance of confusion.
For added confidence, the analysis has been re-run including those who withdrew post randomisation, but pre visit. When these are included the findings do not change.
The observed effect size was 3.1%; there is lots of uncertainty in this estimate so ideally it shouldn't really be used to infer the likely cost effectiveness of future studies -instead, I'd recommend considering each end of the confidence interval (or perhaps the upper end as a best-case scenario) when considering cost per additional person retained.
We thank the reviewer for making this important point. As the manuscript states, the cost of retaining an additional participant would be over £300. This statement is still correct in the worst case scenario, and as such covers the uncertainty; whilst also covering the estimate that we have found in this study.
I don't understand the sentence 'Due to the sample size of this SWAT, it is most likely generalisable to the larger host trial population of patients with severe mental ill health disorders.' A large sample size does not guarantee generalisability, and as noted the sample size is not really large enough to draw conclusions about this population. I think this point needs to be constructed more clearly. For clarity, this section has been amended to reflect that results are only applicable to the study population. This was a SWAT within the SCIMITAR+ trial. The purpose of the SWAT was to establish if £10 cash + £10 voucher would increase retention rates when compared to £10 voucher only. SCIMITAR+ was a smoking cessation trial. The authors conclude that contingent financial incentives did not statistically significantly increase rates of face-to-face follow-up completion within the SCIMITAR+ trial.

Frances Shiely
In the background section of the abstract, you describe the SWAT as "…versus no contingent financial incentive" which is misleading. There is a contingent financial incentive because you have a £10 voucher as standard. I presume it's contingent on them turning up to the appointment? I had to go to the body of the paper to ensure I had the correct interpretation. You should describe the SWAT more accurately as "to establish if £10 cash + £10 voucher would increase retention rates when compared to £10 voucher only" and change this in the main body of the paper also. I struggled with the design of this SWAT and what the investigators hoped to achieve by it. I don't think the primary outcome was suitable to measure what they intended. The conclusion drawn in the abstract is unrelated to the primary outcome. The purpose of the SWAT was to look at retention. The participants were eligible for the SWAT if they reached the SCIMITAR+ 6-month follow-up. I'd like the authors to comment on this in relation to their SWAT. It could be argued that if they were already coming for the 6 month follow-up they were motivated to continue regardless of the extra financial incentive. It seems like 100% of those contacted for the 6-month follow-up turned up to their appointment. Therefore, 100% were retained. I don't see the point then in making the primary outcome contingent on the CO breath measurement and was not surprised there was no statistically significant difference between the two groups. Your secondary outcome, the proportion withdrawing after the contact would be more relevant. This is supported by the fact you used the proportion turning up for the appointment as your concluding statement.
Comparing two incentivised arms at 6 month follow-up doesn't appear to be the best use of the money. What would have been interesting to find out is if the additional money at 12 months would make a difference to retention? Perhaps let the intervention arm know at 6-months that at 12 months, in addition to the voucher they would get £10 cash. Then you could establish if the additional incentive was effective.
I would have liked to know what you told the comparator group when you phoned them to come to their 6-month visit. Did you tell them they were getting a £10 voucher to give a CO. Did you make them aware of what the intervention group were getting or did you tell them about the SWAT at all?
The participants were already incentivised with the £10 voucher. What is the evidence base for thinking that an additional £10 cash would increase retention at 6 months? I note the Bailey et al. 2013 study but this was in relation to response rates for a postal questionnaire. 1 Having reviewed, we believe that the conclusion does relate to the primary outcome given it was unclear, due to the wide confidence intervals if there was any difference in completion of a CO breath measure between the two groups. Following review of this, and comments from Reviewer 2, we can see there has been some misunderstanding. The study randomisation was done in a single batch, rather than when the participant reached the Month 6 time point, therefore at the time of randomisation it was no known if the participant would attend their 6 month visit. We have made this clearer in the randomisation and results sections. We are clear how the reviewer has derived this information. Figure 1 demonstrates that 32 intervention and 21 control participants did not attend a face to face visit and so did not provide a CO breath measure.
Comparing two incentivised arms at 6 month follow-up doesn't appear to be the best use of the money. What would have been interesting to find out is if the additional money at 12 months would make a difference to retention? Perhaps let the intervention arm know at 6months that at 12 months, in addition to the voucher they would get £10 cash. Then you could establish if the additional incentive was effective. The impact of the contingent reward on subsequent follow up time points is an important consideration. The effects were not assessed in this study, but this would be important to include in any subsequent studies of this nature.
I would have liked to know what you told the comparator group when you phoned them to come to their 6-month visit. Did you tell them they were getting a £10 voucher to give a CO. Did you make them aware of what the intervention group were getting or did you tell them about the SWAT at all? As noted in the intervention description, during the call to arrange a follow up appointment intervention participants were advised of the contingent £10 if they were willing to meet face to face and provide the primary outcome. The intervention section has been reviewed to make this clearer.