ALL Metrics
-
Views
-
Downloads
Get PDF
Get XML
Cite
Export
Track
Brief Report
Revised

SWAT 76 evaluation: randomised evaluation of sending pre-notification cards to trial participants before a face-to-face primary outcome measurement to increase attendance

[version 2; peer review: 2 approved]
PUBLISHED 15 Jul 2021
Author details Author details
OPEN PEER REVIEW
REVIEWER STATUS

This article is included in the Studies Within A Trial (SWAT) collection.

Abstract

Background: Retention is considered the second highest trial methods priority in the UK after recruitment.
  Methods: This Study Within A Trial (SWAT) evaluated whether sending a pre-notification card around one month before a face-to-face primary outcome measurement visit compared to not sending the card increased trial retention. The SWAT was a two-arm, parallel randomised (1:1 allocation ratio), stratified by centre, study. It was embedded within the ActWELL host trial, which evaluated whether women receiving lifestyle change counselling from volunteer coaches improved outcomes including weight and physical activity.
The SWAT primary outcome was the difference in the proportion of participants attending the host trial primary outcome measurement visit. The secondary outcome was the direct cost of sending cards. Host trial participants and research staff at the primary outcome visit were blind to the SWAT. Analysis was intention-to-treat. Certainty of evidence was assessed using GRADE.
Results: 558 host trial participants took part in the SWAT and were included in the analysis. Of the 274 women sent a card, 231 attended the primary outcome visit (84.3%) compared to 230/284 (81.0%) for those not receiving a card. Sending a pre-notification card may result in a slight increase in attendance at a face-to-face primary outcome measurement visit at 1-year: risk difference = 3.3% (95% confidence interval = -3.0% to 9.6%). This is GRADE low certainty evidence. A recording error meant it was unclear whether 17 participants allocated to the card were actually sent one but a sensitivity analysis did not change the overall result or conclusion. The direct cost of producing and sending the cards was £192 GBP (€213 EUR; $260 USD), or £21.33 (€23.55; $28.77 per additional retained participant.
  Discussion: Trialists could consider using pre-notification as they may gain a slight increase in retention to face-to-face trial measurement visits but further evaluations are needed.

Keywords

SWAT, retention, pre-notification cards, randomised trial

Revised Amendments from Version 1

The main changes are: relative risk data are now presented alongside absolute risks; the total number of women who attended the host trial primary outcome visit has been given in Table 1; we have added reference to SWAT 77 and SWAT 86 in our Discussion; we have added some context to the Discussion around the cost of the SWAT intervention; minor text clarifications throughout.

See the authors' detailed response to the review by Christopher Sutton
See the authors' detailed response to the review by Michelle E Kho

Introduction

Retention is considered the second highest trial methods priority in the UK after recruitment.1 A recent UK study found that the median loss-to-follow-up in a sample of 151 trials was 11%.2 This reduces the amount of trial data available for analysis, which is especially problematic for the trial's most important outcome - the primary outcome - because this is the outcome that will be used to judge whether the trial intervention is effective. Ensuring that retention is high is therefore of great important to trialists. Reminders are generally an effective way of increasing response rates to questionnaires and there is evidence that pre-notification (contacting participants to say that they will soon be sent a questionnaire) is beneficial, although it is not high certainty evidence.3

There is no clear evidence from the Cochrane systematic review of trial retention interventions that pre-notification is effective for trial retention for face-to-face visits.4 However, a Study Within A Trial (SWAT) in a trial involving women aged between 70 and 84 years at high risk of osteoporotic fractures did find that sending a newsletter to participants approximately six weeks before a trial questionnaire increased retention by 1.6% (P = 0.05).5

SWAT question

Does sending a pre-notification card around one month before a face-to-face primary outcome measurement visit compared to not sending the card increase trial retention?

Methods

Host trial

This SWAT evaluation was embedded in the ActWELL trial (ISRCTN11057518).6 ActWELL evaluates whether women who receive two, face-to-face lifestyle change sessions from volunteer coaches followed by up to nine telephone calls over a year, compared to no counselling, improves a range of lifestyle outcomes. The two primary outcomes were weight change and change in physical activity at 12-months. Women were invited to take part in ActWELL when they attended their routine mammography appointment (all women aged 50 - 70 in Scotland receive an offer of mammography every three years) at one of four Scottish National Health Service Breast Screening centres. A total of 560 women were randomised into the ActWELL trial.

Participants

All host trial participants who had not withdrawn were eligible.

Intervention

The intervention is a pre-notification card sent around one month before the face-to-face primary outcome measurement visit. Women were thanked to make them feel valued, were told their data were valuable regardless of how things had gone in the trial and the number of other women in the trial was highlighted. The card was signed by the Chief Investigator of the host trial and the Chief Executive of Breast Cancer Now, the charity involved in delivering the host trial intervention. The card is shown in Figure 1.

ba512ca1-fbe5-4e0f-af40-c5162ac38bfa_figure1a.gifba512ca1-fbe5-4e0f-af40-c5162ac38bfa_figure1b.gif

Figure 1. The ActWELL 12-month pre-notification card.

Comparator

No pre-notification card.

Outcomes

Primary outcome: the difference in the proportion of participants attending the host trial primary outcome measurement visit (i.e., retention).

Secondary outcome: the direct cost of sending pre-notification cards.

Sample size

The sample size was determined by the host trial so no sample size calculation was done. See Trial Forge Guidance 1 for more information about SWAT sample size calculation.7

Randomisation

Two-arm, parallel randomised with a 1:1 allocation ratio, stratified by centre. One of the authors (ST) prepared a central randomisation list for each centre for up to 150 participants using https://www.random.org/sequences/. This was then passed to the trial manager and trial administrator who sent out the pre-notification cards.

Blinding

Women in the host trial had no knowledge of the SWAT. Host trial primary outcome visits were organised and done by research nurses, who had no knowledge of the SWAT or host trial allocation. The SWAT primary outcome, retention, was objective.

Approvals

The study was approved by East of Scotland Research Ethics Service REC 1 as part of the host ActWELL trial (17/ES/0073). The low risk nature of the SWAT evaluation meant that individual informed consent from host trial participants to be involved was not required by the ethics committee, in line with most SWATs in the UK.

Statistical analysis

The difference in the proportion of attended visits between groups was calculated using fixed effects in Comprehensive Meta-Analysis Version 3 (https://www.meta-analysis.com/).

GRADE was used to assess the certainty of the evidence.8 In addition to the numerical result, the result is summarised as an informative statement as per GRADE Guidelines 26.9

Results

Two host trial participants withdrew before the 12-month host trial primary outcome measurement meaning 558 were included in the SWAT, which ran between March 2018 and July 2019 (Figure 2; Table 1). One host trial centre recruited 151 participants, which was beyond its recruitment target and one participant beyond the randomisation list for that centre. The extra participant was manually allocated to the comparator arm. A discrepancy between the randomisation log (which indicated who should get a card) and the host trial's tracker system (which confirmed that a card had been sent to a participant) meant that we could not confirm whether 17 participants (3.0% of the total) who should have been sent the pre-notification card were actually sent one. Three further participants who should have received a card are known to have not been sent a card because the participant was called in for a host trial measurement visit before the card could be sent.

ba512ca1-fbe5-4e0f-af40-c5162ac38bfa_figure2.gif

Figure 2. Flow diagram summarising the flow of participants through the SWAT evaluation.

The summary statement below and the primary analysis in Table 1 are intention-to-treat as per the randomisation schedule. A sensitivity analysis done using the tracker data is also shown.

Summary statement of result: Sending a pre-notification card may result in a slight increase in attendance at a face-to-face primary outcome measurement visit at 1-year. Risk difference = 3.3% (95% confidence interval = -3.0% to 9.6%). GRADE = low certainty evidence.

Table 1. Attendance at the 12-month primary outcome measurement visit for those sent a pre-notification card and those not sent a card.

Attendance at 12-months – Intention-to-treat analysis (SWAT primary analysis)
Total number of participants who attended the primary outcome visit = 461
Allocated to be sent a pre-notification card (n = 274)*Allocated to not be sent a notification card (n = 284)
Attended visitDid not attend visitAttended visitDid not attend visitRisk difference (95% confidence interval)Risk ratio (95% confidence interval)
231 (84.3%)43 (15.7%)230 (81.0%)54 (19.0%)3.3% (−3.0% to 9.6%)1.04 (0.97 to 1.12)
*This includes three participants who were allocated to receive a card but who are known not to have been sent one.
Attendance at 12-months – As per tracker system (sensitivity analysis)
Total number of participants who attended the primary outcome visit = 461
Tracker registers participant was sent a pre-notification card (n = 254)Tracker registers participant was not sent a notification card (n = 304)
Attended visitDid not attend visitAttended visitDid not attend visitRisk difference (95% confidence interval)Risk ratio (95% confidence interval)
213 (83.9%)41 (16.1%)248 (81.6%)56 (18.4%)2.3% (−4.0% to 8.6%)1.03 (0.95 to 1.11)
GRADE assessment
Study limitationsNo serious issues.
Inconsistency of resultsDowngrade 1 level because this is a single study (sparsity of data).
Indirectness of evidenceNo serious issues.
ImprecisionDowngrade 1 level because confidence interval is wide and crosses risk difference of 0.
Reporting biasNo serious issues.

Direct costs

The direct costs of printing the cards was £72 GBP. Design work was extremely modest, bundled with other host trial design work and not charged separately. Second class (i.e., delivery within two days) postage costs were run through the University of Dundee mailroom at an estimated cost of £120 GBP. The total direct cost was therefore £192 GBP (€212 EUR; $259 USD).

Cost per additional participant retained

If the 274 participants who received the card had attended at the same rate as those who did not, a total of 222 participants would have attended the next visit. In fact, 231 attended, meaning an extra nine participants were retained. The cost per additional participant retained was £192/9, or £21.33 (€23.55; $28.77).

Discussion

Sending a simple card about one month prior to a face-to-face primary outcome measurement visit may result in a slight improvement in attendance. This is GRADE low certainty evidence because there is just this single evaluation and it is imprecise.

We are not aware of other pre-notification interventions that target face-to-face trial visits. The 2021 update of the Cochrane retention review4 found no additional completed pre-notification studies (ST is a co-author of this update). There are at least two pre-notification protocols (SWAT 77 and SWAT 86) registered on the SWAT repository (https://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/SWATSWARInformation/Repositories/SWATStore/) in addition to our own SWAT 76, meaning there may be additional studies underway but not yet complete. Mitchell and colleagues5 added their evaluation to a meta-analysis of pre-notification evidence done outside trials and healthcare. As might be expected, there was substantial heterogeneity but the overall direction of effect was also in favour of pre-notification.

Strengths and limitations

We had a record-keeping error, which means we cannot say with confidence that all participants who should have been sent a pre-notification card were sent one. However, the number of participants affected was relatively small and our overall results and conclusion remain the same regardless of whether we analyse according to the randomisation schedule or the tracker system, which suggests the results are robust in the face of this error. There are currently only two evaluations of pre-notification in trials, our own of cards aimed at face-to-face visits and that of Mitchell and colleagues of a newsletter to increase questionnaire response.5 This does not provide a broad range of contexts for this evidence base. Both evaluations were done in the UK and in women only. Finally, although the text on the card did target factors thought to influence attendance we did not use formal behavioural change theory to identify these factors, or to suggest text that might target these factors.

Implications for trial practice

Given the paucity of evidence to support retention decisions,4 trialists could consider using pre-notification as they may gain a slight increase in retention to face-to-face trial measurement visits at 1-year. Trials are expensive and in that context pre-notification cards are a very cheap intervention that may provide a small increase in the proportion of primary outcome data a trial team obtains. We had no negative reaction to them from participants (i.e. there were no complaints) and our cost of around £21 per additional retained participant is likely to be substantially cheaper that recruiting an additional participant to replace these lost primary outcome data.

Implications for SWAT research

Looking at the existing evidence and referring to Trial Forge Guidance 2 as to whether further SWATs evaluating this intervention are required,10 we conclude that more evaluations are needed because the GRADE certainty in the evidence is not high, there is only a single evaluation meaning cumulative meta-analysis cannot converge and few host trial contexts are included.

Further evaluations of pre-notification in trials could target either face-to-face or questionnaires but should aim to add new host trial contexts. Future host trials should involve men. Formal approaches to developing intervention content may increase effect sizes.

Data availability

Underlying data

Open Science Framework: SWAT 76 data for host trial ActWELL, https://doi.org/10.17605/OSF.IO/N64HU.11

This project contains the following underlying data:

  • Primary analysis 8-1-2021 (public).csv

  • Sensitivity analysis 8-1-2021 (public).csv

Reporting guidelines

Open Science Framework: CONSORT checklist for ‘SWAT 76 evaluation: randomised evaluation of sending pre-notification cards to trial participants before a face-to-face primary outcome measurement to increase attendance’, https://doi.org/10.17605/OSF.IO/B78JT.12

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Comments on this article Comments (0)

Version 2
VERSION 2 PUBLISHED 08 Feb 2021
Comment
Author details Author details
Competing interests
Grant information
Copyright
Download
 
Export To
metrics
Views Downloads
F1000Research - -
PubMed Central
Data from PMC are received and updated monthly.
- -
Citations
CITE
how to cite this article
Treweek S, Gallant S and Anderson AS. SWAT 76 evaluation: randomised evaluation of sending pre-notification cards to trial participants before a face-to-face primary outcome measurement to increase attendance [version 2; peer review: 2 approved]. F1000Research 2021, 10:84 (https://doi.org/10.12688/f1000research.50890.2)
NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.
track
receive updates on this article
Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?
Key to Reviewer Statuses VIEW
ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.
Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions
Version 2
VERSION 2
PUBLISHED 15 Jul 2021
Revised
Views
3
Cite
Reviewer Report 11 Aug 2021
Christopher Sutton, Centre for Biostatistics, University of Manchester, Manchester, UK 
Approved
VIEWS 3
I am pleased to have had the opportunity to review and comment on the revised article and ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Sutton C. Reviewer Report For: SWAT 76 evaluation: randomised evaluation of sending pre-notification cards to trial participants before a face-to-face primary outcome measurement to increase attendance [version 2; peer review: 2 approved]. F1000Research 2021, 10:84 (https://doi.org/10.5256/f1000research.58441.r89686)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
Version 1
VERSION 1
PUBLISHED 08 Feb 2021
Views
14
Cite
Reviewer Report 27 Apr 2021
Michelle E Kho, School of Rehabilitation Science, McMaster University, Hamilton, ON, Canada 
Approved
VIEWS 14
Treweek et al. conducted a study within a trial to evaluate whether a post card sent 1 month prior to an in-person primary outcome research visit improved retention within the multicentre ActWELL randomized trial of lifestyle coaching. 558 participants from ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Kho ME. Reviewer Report For: SWAT 76 evaluation: randomised evaluation of sending pre-notification cards to trial participants before a face-to-face primary outcome measurement to increase attendance [version 2; peer review: 2 approved]. F1000Research 2021, 10:84 (https://doi.org/10.5256/f1000research.53982.r79341)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
  • Author Response 15 Jul 2021
    Shaun P. Treweek, Health Services Research Unit, University of Aberdeen, Aberdeen, UK
    15 Jul 2021
    Author Response
    Response to comments on F1KR00CDE F1R-VER53982-A
    5/7/2021

    Thanks for the reviewers’ comments on our manuscript. Our responses are below.  We have made the necessary changes to the manuscript using ... Continue reading
COMMENTS ON THIS REPORT
  • Author Response 15 Jul 2021
    Shaun P. Treweek, Health Services Research Unit, University of Aberdeen, Aberdeen, UK
    15 Jul 2021
    Author Response
    Response to comments on F1KR00CDE F1R-VER53982-A
    5/7/2021

    Thanks for the reviewers’ comments on our manuscript. Our responses are below.  We have made the necessary changes to the manuscript using ... Continue reading
Views
17
Cite
Reviewer Report 01 Mar 2021
Christopher Sutton, Centre for Biostatistics, University of Manchester, Manchester, UK 
Approved
VIEWS 17
Overall, this is an excellent manuscript and an extremely welcome and useful addition to the literature on trial retention.  The work provides a clear citation of the relevant literature.

The methodology is sound and the trial performed ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Sutton C. Reviewer Report For: SWAT 76 evaluation: randomised evaluation of sending pre-notification cards to trial participants before a face-to-face primary outcome measurement to increase attendance [version 2; peer review: 2 approved]. F1000Research 2021, 10:84 (https://doi.org/10.5256/f1000research.53982.r79342)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
  • Author Response 15 Jul 2021
    Shaun P. Treweek, Health Services Research Unit, University of Aberdeen, Aberdeen, UK
    15 Jul 2021
    Author Response
    Response to comments on F1KR00CDE F1R-VER53982-A
    5/7/2021

    Thanks for the reviewers’ comments on our manuscript. Our responses are below.  We have made the necessary changes to the manuscript using ... Continue reading
COMMENTS ON THIS REPORT
  • Author Response 15 Jul 2021
    Shaun P. Treweek, Health Services Research Unit, University of Aberdeen, Aberdeen, UK
    15 Jul 2021
    Author Response
    Response to comments on F1KR00CDE F1R-VER53982-A
    5/7/2021

    Thanks for the reviewers’ comments on our manuscript. Our responses are below.  We have made the necessary changes to the manuscript using ... Continue reading

Comments on this article Comments (0)

Version 2
VERSION 2 PUBLISHED 08 Feb 2021
Comment
Alongside their report, reviewers assign a status to the article:
Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.
Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions
Sign In
If you've forgotten your password, please enter your email address below and we'll send you instructions on how to reset your password.

The email address should be the one you originally registered with F1000.

Email address not valid, please try again

You registered with F1000 via Google, so we cannot reset your password.

To sign in, please click here.

If you still need help with your Google account password, please click here.

You registered with F1000 via Facebook, so we cannot reset your password.

To sign in, please click here.

If you still need help with your Facebook account password, please click here.

Code not correct, please try again
Email us for further assistance.
Server error, please try again.