SWAT 76 evaluation: randomised evaluation of sending pre-notification cards to trial participants before a face-to-face primary outcome measurement to increase attendance

Background: Retention is considered the second highest trial methods priority in the UK after recruitment. Methods: This Study Within A Trial (SWAT) evaluated whether sending a pre-notification card around one month before a face-to-face primary outcome measurement visit compared to not sending the card increased trial retention. The SWAT was a two-arm, parallel randomised (1:1 allocation ratio), stratified by centre, study. It was embedded within the ActWELL host trial, which evaluated whether women receiving lifestyle change counselling from volunteer coaches improved outcomes including weight and physical activity. The SWAT primary outcome was the difference in the proportion of participants attending the host trial primary outcome measurement visit. The secondary outcome was the direct cost of sending cards. Host trial participants and research staff at the primary outcome visit were blind to the SWAT. Analysis was intention-to-treat. Certainty of evidence was assessed using GRADE. Results: 558 host trial participants took part in the SWAT and were included in the analysis. Of the 274 women sent a card, 231 attended the primary outcome visit (84.3%) compared to 230/284 (81.0%) for those not receiving a card. Sending a pre-notification card may result in a slight increase in attendance at a face-to-face primary outcome measurement visit at 1-year: risk difference = 3.3% (95% confidence interval = -3.0% to 9.6%). This is GRADE low certainty evidence. A recording error meant it was unclear whether 17 participants allocated to the card were actually sent one but a sensitivity analysis did not change the overall result or conclusion. The direct cost of producing and sending the cards was £192 GBP (€213 EUR; $260 USD), or £21.33 (€23.55; $28.77 per additional retained participant. Discussion: Trialists could consider using pre-notification as they may gain a slight increase in retention to face-to-face trial measurement visits but further evaluations are needed.


Introduction
Retention is considered the second highest trial methods priority in the UK after recruitment. 1 A recent UK study found that the median loss-to-follow-up in a sample of 151 trials was 11%. 2 This reduces the amount of trial data available for analysis, which is especially problematic for the trial's most important outcome -the primary outcome -because this is the outcome that will be used to judge whether the trial intervention is effective. Ensuring that retention is high is therefore of great important to trialists. Reminders are generally an effective way of increasing response rates to questionnaires and there is evidence that pre-notification (contacting participants to say that they will soon be sent a questionnaire) is beneficial, although it is not high certainty evidence. 3 There is no clear evidence from the Cochrane systematic review of trial retention interventions that pre-notification is effective for trial retention for face-to-face visits. 4 However, a Study Within A Trial (SWAT) in a trial involving women aged between 70 and 84 years at high risk of osteoporotic fractures did find that sending a newsletter to participants approximately six weeks before a trial questionnaire increased retention by 1.6% (P = 0.05). 5

SWAT question
Does sending a pre-notification card around one month before a face-to-face primary outcome measurement visit compared to not sending the card increase trial retention?

Host trial
This SWAT evaluation was embedded in the ActWELL trial (ISRCTN11057518). 6 ActWELL evaluates whether women who receive two, face-to-face lifestyle change sessions from volunteer coaches followed by up to nine telephone calls over a year, compared to no counselling, improves a range of lifestyle outcomes. The two primary outcomes were weight change and change in physical activity at 12-months. Women were invited to take part in ActWELL when they attended their routine mammography appointment (all women aged 50 -70 in Scotland receive an offer of mammography every three years) at one of four Scottish National Health Service Breast Screening centres. A total of 560 women were randomised into the ActWELL trial.

Participants
All host trial participants who had not withdrawn were eligible.

Intervention
The intervention is a pre-notification card sent around one month before the face-to-face primary outcome measurement visit. Women were thanked to make them feel valued, were told their data were valuable regardless of how things had gone in the trial and the number of other women in the trial was highlighted. The card was signed by the Chief Investigator of the host trial and the Chief Executive of Breast Cancer Now, the charity involved in delivering the host trial intervention. The card is shown in Figure 1.

Comparator
No pre-notification card.

Outcomes
Primary outcome: the difference in the proportion of participants attending the host trial primary outcome measurement visit (i.e., retention).

REVISED Amendments from Version 1
The main changes are: relative risk data are now presented alongside absolute risks; the total number of women who attended the host trial primary outcome visit has been given in Table 1; we have added reference to SWAT 77 and SWAT 86 in our Discussion; we have added some context to the Discussion around the cost of the SWAT intervention; minor text clarifications throughout.
Any further responses from the reviewers can be found at the end of the article  Secondary outcome: the direct cost of sending pre-notification cards.

Sample size
The sample size was determined by the host trial so no sample size calculation was done. See Trial Forge Guidance 1 for more information about SWAT sample size calculation. 7 Randomisation Two-arm, parallel randomised with a 1:1 allocation ratio, stratified by centre. One of the authors (ST) prepared a central randomisation list for each centre for up to 150 participants using https://www.random.org/sequences/. This was then passed to the trial manager and trial administrator who sent out the pre-notification cards.

Blinding
Women in the host trial had no knowledge of the SWAT. Host trial primary outcome visits were organised and done by research nurses, who had no knowledge of the SWAT or host trial allocation. The SWAT primary outcome, retention, was objective.

Approvals
The study was approved by East of Scotland Research Ethics Service REC 1 as part of the host ActWELL trial (17/ES/0073). The low risk nature of the SWAT evaluation meant that individual informed consent from host trial participants to be involved was not required by the ethics committee, in line with most SWATs in the UK.

Statistical analysis
The difference in the proportion of attended visits between groups was calculated using fixed effects in Comprehensive Meta-Analysis Version 3 (https://www.meta-analysis.com/). GRADE was used to assess the certainty of the evidence. 8 In addition to the numerical result, the result is summarised as an informative statement as per GRADE Guidelines 26. 9

Results
Two host trial participants withdrew before the 12-month host trial primary outcome measurement meaning 558 were included in the SWAT, which ran between March 2018 and July 2019 ( Figure 2; Table 1). One host trial centre recruited 151 participants, which was beyond its recruitment target and one participant beyond the randomisation list for that centre. The extra participant was manually allocated to the comparator arm. A discrepancy between the randomisation log (which indicated who should get a card) and the host trial's tracker system (which confirmed that a card had been sent to a participant) meant that we could not confirm whether 17 participants (3.0% of the total) who should have been sent the pre-notification card were actually sent one. Three further participants who should have received a card are known to have not been sent a card because the participant was called in for a host trial measurement visit before the card could be sent.
The summary statement below and the primary analysis in Table 1 are intention-to-treat as per the randomisation schedule. A sensitivity analysis done using the tracker data is also shown.
Summary statement of result: Sending a pre-notification card may result in a slight increase in attendance at a face-to-face primary outcome measurement visit at 1-year. Risk difference = 3.3% (95% confidence interval = -3.0% to 9.6%). GRADE = low certainty evidence.

Direct costs
The direct costs of printing the cards was £72 GBP. Design work was extremely modest, bundled with other host trial design work and not charged separately. Second class (i.e., delivery within two days) postage costs were run through the University of Dundee mailroom at an estimated cost of £120 GBP. The total direct cost was therefore £192 GBP (€212 EUR; $259 USD).

Cost per additional participant retained
If the 274 participants who received the card had attended at the same rate as those who did not, a total of 222 participants would have attended the next visit. In fact, 231 attended, meaning an extra nine participants were retained. The cost per additional participant retained was £192/9, or £21.33 (€23.55; $28.77). Table 1. Attendance at the 12-month primary outcome measurement visit for those sent a pre-notification card and those not sent a card.
Attendance at 12-months -Intention-to-treat analysis (SWAT primary analysis) Total number of participants who attended the primary outcome visit = 461 Allocated to be sent a pre-notification card (n = 274)* Allocated to not be sent a notification card (n = 284)

Attended visit
Did *This includes three participants who were allocated to receive a card but who are known not to have been sent one.
Attendance at 12-months -As per tracker system (sensitivity analysis) Total number of participants who attended the primary outcome visit = 461

Study limitations
No serious issues.

Inconsistency of results
Downgrade 1 level because this is a single study (sparsity of data).

Indirectness of evidence
No serious issues.

Imprecision
Downgrade 1 level because confidence interval is wide and crosses risk difference of 0.

Reporting bias
No serious issues.

Discussion
Sending a simple card about one month prior to a face-to-face primary outcome measurement visit may result in a slight improvement in attendance. This is GRADE low certainty evidence because there is just this single evaluation and it is imprecise.
We are not aware of other pre-notification interventions that target face-to-face trial visits. The 2021 update of the Cochrane retention review 4 found no additional completed pre-notification studies (ST is a co-author of this update). There are at least two pre-notification protocols (SWAT 77 and SWAT 86) registered on the SWAT repository (https://www.qub.ac.uk/sites/ TheNorthernIrelandNetworkforTrialsMethodologyResearch/SWATSWARInformation/Repositories/SWATStore/) in addition to our own SWAT 76, meaning there may be additional studies underway but not yet complete. Mitchell and colleagues 5 added their evaluation to a meta-analysis of pre-notification evidence done outside trials and healthcare. As might be expected, there was substantial heterogeneity but the overall direction of effect was also in favour of prenotification.

Strengths and limitations
We had a record-keeping error, which means we cannot say with confidence that all participants who should have been sent a pre-notification card were sent one. However, the number of participants affected was relatively small and our overall results and conclusion remain the same regardless of whether we analyse according to the randomisation schedule or the tracker system, which suggests the results are robust in the face of this error. There are currently only two evaluations of pre-notification in trials, our own of cards aimed at face-to-face visits and that of Mitchell and colleagues of a newsletter to increase questionnaire response. 5 This does not provide a broad range of contexts for this evidence base. Both evaluations were done in the UK and in women only. Finally, although the text on the card did target factors thought to influence attendance we did not use formal behavioural change theory to identify these factors, or to suggest text that might target these factors.

Implications for trial practice
Given the paucity of evidence to support retention decisions, 4 trialists could consider using pre-notification as they may gain a slight increase in retention to face-to-face trial measurement visits at 1-year. Trials are expensive and in that context pre-notification cards are a very cheap intervention that may provide a small increase in the proportion of primary outcome data a trial team obtains. We had no negative reaction to them from participants (i.e. there were no complaints) and our cost of around £21 per additional retained participant is likely to be substantially cheaper that recruiting an additional participant to replace these lost primary outcome data.

Implications for SWAT research
Looking at the existing evidence and referring to Trial Forge Guidance 2 as to whether further SWATs evaluating this intervention are required, 10 we conclude that more evaluations are needed because the GRADE certainty in the evidence is not high, there is only a single evaluation meaning cumulative meta-analysis cannot converge and few host trial contexts are included.
Further evaluations of pre-notification in trials could target either face-to-face or questionnaires but should aim to add new host trial contexts. Future host trials should involve men. Formal approaches to developing intervention content may increase effect sizes.

Open Peer Review
to an in-person primary outcome research visit improved retention within the multicentre ActWELL randomized trial of lifestyle coaching. 558 participants from the host trial were sent a pre-notification card. Of this cohort, 84% who were allocated a pre-notification card attended the primary outcome visit, compared to 81% who were not allocated a pre-notification card. This is a well-written, well-designed, and well-reported study. It will make important contributions to further understanding ways to improve retention to clinical trials. I have some considerations for the authors.

Moderate issues:
Results -To help the reader understand the impact on the trial, I recommend the authors first report the total number of women who attended their primary outcome follow-up visit, then report the proportion by group.

1.
Results -I initially found Table 1 confusing. Since those who did not attend the visit is the complement of those who did attend a visit, I suggest reporting only those who attended the visit. The risk difference represents the difference between those who attended the visit. I also suggest adding the 95% CI around the estimates, and report 1 decimal point for consistency with the risk difference.

2.
Can the authors report whether patients recalled receiving the pre-notification card before their outcomes visit?

3.
Implications for trial practice -I suggest adding text to further contextualize the implications of the relative cost of this intervention, the overall investment costs in the enrolled patients to-date, and the importance of ascertaining the primary outcome. Even a 3% improvement in primary outcome ascertainment could affect trial results.

1.
Introduction -I suggest adding a statement regarding the importance of the primary outcome in a trial and the need for interventions to optimize ascertainment.

2.
Intervention -I suggest moving the text regarding behavioural change theory in the development of the pre-notification card from the methods to the limitations section of the manuscript.

3.
Results -Data error for 17 participants (3%) -suggest also reporting % of total group to help the reader easily evaluate the small number. Interestingly, the risk difference was also 3%. The authors conducted a sensitivity analysis, and the results were similar. I suggest this is also a strength of the study.

4.
Implications for trial practice. I suggest framing the study findings specific to 1-year followup measures.

If applicable, is the statistical analysis and its interpretation appropriate? Partly
Are all the source data underlying the results available to ensure full reproducibility? Yes

Are the conclusions drawn adequately supported by the results? Yes
Competing Interests: No competing interests were disclosed.
Reviewer Expertise: Critical care clinical trials; health research methodology; rehabilitation; physiotherapy

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
Author Response 05 Jul 2021 Shaun P. Treweek, University of Aberdeen, Aberdeen, UK

Response to comments on F1KR00CDE F1R-VER53982-A 5/7/2021
Thanks for the reviewers' comments on our manuscript. Our responses are below. We have made the necessary changes to the manuscript using track changes so that they can be easily seen. We have also submitted a clean version with the changes accepted.

Introduction: clearly state that the previous SWAT of the impact of a newsletter on retention was statistically significant (p<0.05 … just!)
Done. We have also added the exact difference of 1.6% rather than saying 'around 1%'.

I would query the statement that "All trial participants were eligible."
We have changed our text to: 'All host trial participants who had not withdrawn were eligible.'

The primary outcome would be better specified as a binary outcome measure (i.e. at the participant level e.g. "attendance at the host trial primary outcome measurement visit").
We have thought about this but since the primary result we report is the difference in attendance between those who got the card and those who did not (i.e. 3.3%) and not the actual attendance for each group (84% and 81%), we think the way we describe the primary is correct.

Statistical analysis: the method of analysis should be stated, in addition to the software used
We have added that we used a fixed effects model.

Present relative relative measures of effect as well.
Good idea, apologies for not doing it originally. Done.

I would like to see more information on the economic analysis. For example, how much did the intervention cost per participant retained?
We only collected direct costs so our economic analysis is very limited. However, we liked the suggestion to calculate a cost per (extra) participant retained, which means we have added the text below to the cost information. We have also added the cost per additional participant retained to the abstract. References 11 and 12 were added by the publisher, not us, but they do work as they should in the version I downloaded yesterday (8/6/2021). There may perhaps have been an error in the version the reviewer was sent that has now been corrected by the publisher.

Reviewer #2
1. Results -To help the reader understand the impact on the trial, I recommend the authors first report the total number of women who attended their primary outcome follow-up visit, then report the proportion by group.
We have added this number to the header row of Table 1 and we have done this for both the intention-to-treat and sensitivity analyses. Table 1 confusing. Since those who did not attend the visit is the complement of those who did attend a visit, I suggest reporting only those who attended the visit. The risk difference represents the difference between those who attended the visit. I also suggest adding the 95% CI around the estimates, and report 1 decimal point for consistency with the risk difference.

Results -I initially found
We have thought about the reviewer's comment re. only showing the number who attended a visit but have decided that we would prefer to be explicit and give both the number who did attend and the number who did not. The main reason is that it avoids the reader having to do a calculation and all the raw numbers are there for readers to do other calculations should they wish.
There is no confidence interval for the proportions attending a visit: the numbers are not an estimate but the actual number attending. The risk difference (and now relative risk in response to Reviewer 1, Comment #5) does have a confidence interval around it. We have changed our presentation so that we now consistently report to 1 decimal place as requested by the reviewer.

Can the authors report whether patients recalled receiving the pre-notification card before their outcomes visit?
Participants were not asked whether they recalled getting a card so we cannot unfortunately report on this. Both the participants and the research nurses who took measurements at the primary outcome visits did not know that we were running this study (i.e. they were blind) and to maintain the blind, we did not ask about the cards. It is possible that some participants mentioned this to the nurse but we have no record of this.
4. Implications for trial practice -I suggest adding text to further contextualize the implications of the relative cost of this intervention, the overall investment costs in the enrolled patients todate, and the importance of ascertaining the primary outcome. Even a 3% improvement in primary outcome ascertainment could affect trial results. Good point. We have re-written the Implications for trial practice section: 'Given the paucity of evidence to support retention decisions [Reference 4], trialists could consider using pre-notification as they may gain a slight increase in retention to face-to-face trial measurement visits. Trials are expensive and in that context pre-notification cards are a very cheap intervention that may provide a small increase in the proportion of primary outcome data a trial team obtains. We had no negative reaction to them from participants (i.e. there were no complaints) and our cost of around £21 per additional retained participant is likely to be substantially cheaper that recruiting an additional participant to replace these lost primary outcome data.'

Abstract -I suggest reporting absolute values in the intervention and control groups, then the risk difference.
Done.

Introduction -I suggest adding a statement regarding the importance of the primary outcome in a trial and the need for interventions to optimize ascertainment.
We have added the following text to the Introduction: 'This reduces the amount of trial data available for analysis, which is especially problematic for the trial's most important outcome-the primary outcome-because this is the outcome that will be used to judge whether the trial intervention is effective. Ensuring that retention is high is therefore of great importance to trialists.' 7. Intervention -I suggest moving the text regarding behavioural change theory in the development of the pre-notification card from the methods to the limitations section of the manuscript. Done.
8. Results -Data error for 17 participants (3%) -suggest also reporting % of total group to help the reader easily evaluate the small number. Interestingly, the risk difference was also 3%. The authors conducted a sensitivity analysis, and the results were similar. I suggest this is also a strength of the study.
We have added the proportion (3%) to the Results and also mentioned that the similarity of the sensitivity analysis results to the intention-to-treat results shows the results are robust in the face of this recording error.

Implications for trial practice. I suggest framing the study findings specific to 1-year follow-up measures.
We have made the change suggested and also added 'at 1-year' to our evidence summary statement in the Abstract and Results.
analysis is appropriate, although I do suggest additional analyses and results be presented to enhance interpretation and generalisability.
The source data underlying the results are available. The conclusions are drawn adequately and supported by the results, with an appropriate sensitivity analysis performed to investigate the potential impact of some degree of failure to definitively send the pre-notification card to 20 participants allocated to receive the card.
Specific comments: Introduction: clearly state that the previous SWAT of the impact of a newsletter on retention was statistically significant (p<0.05 … just!). 1.
I would query the statement that "All trial participants were eligible." From my perspective, the eligibility criteria were (a) Trial participant of the ActWELL trial (b) remaining in ActWELL at 1 month prior to the 12-month final follow-up.

2.
The primary outcome would be better specified as a binary outcome measure (i.e. at the participant level e.g. "attendance at the host trial primary outcome measurement visit").

3.
Statistical analysis: the method of analysis should be stated, in addition to the software used.

4.
In line with CONSORT 2010, for binary outcome measures both absolute and relative measures of effect are recommended. Across different trial populations, interventions, duration of follow-up etc., it is likely that there are different 'control' rates of retention and the impact of any intervention is probably more likely to be on a relative rather than an absolute scale. For example, the newsletter trial referenced in the Introduction (Mitchell et al., (2012)) had a control group retention rate of 94.6% (and an absolute increase of 1.6% in the intervention group), compared with this trial which had a control group retention rate of 81.0% (and an absolute increase of 3.3%). It is therefore recommended that the results are also presented as a Risk Ratio (Relative Risk) or Odds Ratio.

5.
I would like to see more information on the economic analysis. For example, how much did the intervention cost per participant retained? 6.
In the Discussion and Strengths and Limitations, the authors state that there are no additional pre-notification studies; it would be better to stated that there are no additional completed pre-notification studies (there are at least two other pre-notification trials registered on the SWAT Repository Store (SWAT77 and SWAT86).

7.
Reference list: references 11 and 12 re identical (Reference 12 should refer to the CONSORT checklist for this SWAT).

Is the study design appropriate and is the work technically sound?
We have thought about this but since the primary result we report is the difference in attendance between those who got the card and those who did not (i.e. 3.3%) and not the actual attendance for each group (84% and 81%), we think the way we describe the primary is correct.

Statistical analysis: the method of analysis should be stated, in addition to the software used
We have added that we used a fixed effects model. References 11 and 12 were added by the publisher, not us, but they do work as they should in the version I downloaded yesterday (8/6/2021). There may perhaps have been an error in the version the reviewer was sent that has now been corrected by the publisher.

Reviewer #2
1. Results -To help the reader understand the impact on the trial, I recommend the authors first report the total number of women who attended their primary outcome follow-up visit, then report the proportion by group.
6. Introduction -I suggest adding a statement regarding the importance of the primary outcome in a trial and the need for interventions to optimize ascertainment. We have added the following text to the Introduction: 'This reduces the amount of trial data available for analysis, which is especially problematic for the trial's most important outcome-the primary outcome-because this is the outcome that will be used to judge whether the trial intervention is effective. Ensuring that retention is high is therefore of great importance to trialists.' 7. Intervention -I suggest moving the text regarding behavioural change theory in the development of the pre-notification card from the methods to the limitations section of the manuscript. Done.
8. Results -Data error for 17 participants (3%) -suggest also reporting % of total group to help the reader easily evaluate the small number. Interestingly, the risk difference was also 3%. The authors conducted a sensitivity analysis, and the results were similar. I suggest this is also a strength of the study.
We have added the proportion (3%) to the Results and also mentioned that the similarity of the sensitivity analysis results to the intention-to-treat results shows the results are robust in the face of this recording error.

Implications for trial practice. I suggest framing the study findings specific to 1-year follow-up measures.
We have made the change suggested and also added 'at 1-year' to our evidence summary statement in the Abstract and Results.

Competing Interests:
No competing interests were disclosed.
The benefits of publishing with F1000Research: Your article is published within days, with no editorial bias • You can publish traditional articles, null/negative results, case reports, data notes and more • The peer review process is transparent and collaborative • Your article is indexed in PubMed after passing peer review • Dedicated customer support at every stage • For pre-submission enquiries, contact research@f1000.com