Keywords
SWAT, retention, pre-notification cards, randomised trial
This article is included in the Studies Within A Trial (SWAT) collection.
SWAT, retention, pre-notification cards, randomised trial
The main changes are: relative risk data are now presented alongside absolute risks; the total number of women who attended the host trial primary outcome visit has been given in Table 1; we have added reference to SWAT 77 and SWAT 86 in our Discussion; we have added some context to the Discussion around the cost of the SWAT intervention; minor text clarifications throughout.
See the authors' detailed response to the review by Christopher Sutton
See the authors' detailed response to the review by Michelle E Kho
Retention is considered the second highest trial methods priority in the UK after recruitment.1 A recent UK study found that the median loss-to-follow-up in a sample of 151 trials was 11%.2 This reduces the amount of trial data available for analysis, which is especially problematic for the trial's most important outcome - the primary outcome - because this is the outcome that will be used to judge whether the trial intervention is effective. Ensuring that retention is high is therefore of great important to trialists. Reminders are generally an effective way of increasing response rates to questionnaires and there is evidence that pre-notification (contacting participants to say that they will soon be sent a questionnaire) is beneficial, although it is not high certainty evidence.3
There is no clear evidence from the Cochrane systematic review of trial retention interventions that pre-notification is effective for trial retention for face-to-face visits.4 However, a Study Within A Trial (SWAT) in a trial involving women aged between 70 and 84 years at high risk of osteoporotic fractures did find that sending a newsletter to participants approximately six weeks before a trial questionnaire increased retention by 1.6% (P = 0.05).5
This SWAT is registered on the SWAT repository as SWAT 76. See: http://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/FileStore/Filetoupload,864300,en.pdf
This SWAT evaluation was embedded in the ActWELL trial (ISRCTN11057518).6 ActWELL evaluates whether women who receive two, face-to-face lifestyle change sessions from volunteer coaches followed by up to nine telephone calls over a year, compared to no counselling, improves a range of lifestyle outcomes. The two primary outcomes were weight change and change in physical activity at 12-months. Women were invited to take part in ActWELL when they attended their routine mammography appointment (all women aged 50 - 70 in Scotland receive an offer of mammography every three years) at one of four Scottish National Health Service Breast Screening centres. A total of 560 women were randomised into the ActWELL trial.
The intervention is a pre-notification card sent around one month before the face-to-face primary outcome measurement visit. Women were thanked to make them feel valued, were told their data were valuable regardless of how things had gone in the trial and the number of other women in the trial was highlighted. The card was signed by the Chief Investigator of the host trial and the Chief Executive of Breast Cancer Now, the charity involved in delivering the host trial intervention. The card is shown in Figure 1.
Primary outcome: the difference in the proportion of participants attending the host trial primary outcome measurement visit (i.e., retention).
Secondary outcome: the direct cost of sending pre-notification cards.
The sample size was determined by the host trial so no sample size calculation was done. See Trial Forge Guidance 1 for more information about SWAT sample size calculation.7
Two-arm, parallel randomised with a 1:1 allocation ratio, stratified by centre. One of the authors (ST) prepared a central randomisation list for each centre for up to 150 participants using https://www.random.org/sequences/. This was then passed to the trial manager and trial administrator who sent out the pre-notification cards.
Women in the host trial had no knowledge of the SWAT. Host trial primary outcome visits were organised and done by research nurses, who had no knowledge of the SWAT or host trial allocation. The SWAT primary outcome, retention, was objective.
The study was approved by East of Scotland Research Ethics Service REC 1 as part of the host ActWELL trial (17/ES/0073). The low risk nature of the SWAT evaluation meant that individual informed consent from host trial participants to be involved was not required by the ethics committee, in line with most SWATs in the UK.
The difference in the proportion of attended visits between groups was calculated using fixed effects in Comprehensive Meta-Analysis Version 3 (https://www.meta-analysis.com/).
GRADE was used to assess the certainty of the evidence.8 In addition to the numerical result, the result is summarised as an informative statement as per GRADE Guidelines 26.9
Two host trial participants withdrew before the 12-month host trial primary outcome measurement meaning 558 were included in the SWAT, which ran between March 2018 and July 2019 (Figure 2; Table 1). One host trial centre recruited 151 participants, which was beyond its recruitment target and one participant beyond the randomisation list for that centre. The extra participant was manually allocated to the comparator arm. A discrepancy between the randomisation log (which indicated who should get a card) and the host trial's tracker system (which confirmed that a card had been sent to a participant) meant that we could not confirm whether 17 participants (3.0% of the total) who should have been sent the pre-notification card were actually sent one. Three further participants who should have received a card are known to have not been sent a card because the participant was called in for a host trial measurement visit before the card could be sent.
The summary statement below and the primary analysis in Table 1 are intention-to-treat as per the randomisation schedule. A sensitivity analysis done using the tracker data is also shown.
Summary statement of result: Sending a pre-notification card may result in a slight increase in attendance at a face-to-face primary outcome measurement visit at 1-year. Risk difference = 3.3% (95% confidence interval = -3.0% to 9.6%). GRADE = low certainty evidence.
The direct costs of printing the cards was £72 GBP. Design work was extremely modest, bundled with other host trial design work and not charged separately. Second class (i.e., delivery within two days) postage costs were run through the University of Dundee mailroom at an estimated cost of £120 GBP. The total direct cost was therefore £192 GBP (€212 EUR; $259 USD).
If the 274 participants who received the card had attended at the same rate as those who did not, a total of 222 participants would have attended the next visit. In fact, 231 attended, meaning an extra nine participants were retained. The cost per additional participant retained was £192/9, or £21.33 (€23.55; $28.77).
Sending a simple card about one month prior to a face-to-face primary outcome measurement visit may result in a slight improvement in attendance. This is GRADE low certainty evidence because there is just this single evaluation and it is imprecise.
We are not aware of other pre-notification interventions that target face-to-face trial visits. The 2021 update of the Cochrane retention review4 found no additional completed pre-notification studies (ST is a co-author of this update). There are at least two pre-notification protocols (SWAT 77 and SWAT 86) registered on the SWAT repository (https://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/SWATSWARInformation/Repositories/SWATStore/) in addition to our own SWAT 76, meaning there may be additional studies underway but not yet complete. Mitchell and colleagues5 added their evaluation to a meta-analysis of pre-notification evidence done outside trials and healthcare. As might be expected, there was substantial heterogeneity but the overall direction of effect was also in favour of pre-notification.
We had a record-keeping error, which means we cannot say with confidence that all participants who should have been sent a pre-notification card were sent one. However, the number of participants affected was relatively small and our overall results and conclusion remain the same regardless of whether we analyse according to the randomisation schedule or the tracker system, which suggests the results are robust in the face of this error. There are currently only two evaluations of pre-notification in trials, our own of cards aimed at face-to-face visits and that of Mitchell and colleagues of a newsletter to increase questionnaire response.5 This does not provide a broad range of contexts for this evidence base. Both evaluations were done in the UK and in women only. Finally, although the text on the card did target factors thought to influence attendance we did not use formal behavioural change theory to identify these factors, or to suggest text that might target these factors.
Given the paucity of evidence to support retention decisions,4 trialists could consider using pre-notification as they may gain a slight increase in retention to face-to-face trial measurement visits at 1-year. Trials are expensive and in that context pre-notification cards are a very cheap intervention that may provide a small increase in the proportion of primary outcome data a trial team obtains. We had no negative reaction to them from participants (i.e. there were no complaints) and our cost of around £21 per additional retained participant is likely to be substantially cheaper that recruiting an additional participant to replace these lost primary outcome data.
Looking at the existing evidence and referring to Trial Forge Guidance 2 as to whether further SWATs evaluating this intervention are required,10 we conclude that more evaluations are needed because the GRADE certainty in the evidence is not high, there is only a single evaluation meaning cumulative meta-analysis cannot converge and few host trial contexts are included.
Further evaluations of pre-notification in trials could target either face-to-face or questionnaires but should aim to add new host trial contexts. Future host trials should involve men. Formal approaches to developing intervention content may increase effect sizes.
Open Science Framework: SWAT 76 data for host trial ActWELL, https://doi.org/10.17605/OSF.IO/N64HU.11
This project contains the following underlying data:
Open Science Framework: CONSORT checklist for ‘SWAT 76 evaluation: randomised evaluation of sending pre-notification cards to trial participants before a face-to-face primary outcome measurement to increase attendance’, https://doi.org/10.17605/OSF.IO/B78JT.12
Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).
The SWAT evaluation and the ActWELL trial were funded by the Scottish Government (BC/Screening/17/01).
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Views | Downloads | |
---|---|---|
F1000Research | - | - |
PubMed Central
Data from PMC are received and updated monthly.
|
- | - |
Competing Interests: No competing interests were disclosed.
Reviewer Expertise: Clinical trial methodology, particularly around trial conduct and retention methods.
Is the work clearly and accurately presented and does it cite the current literature?
Yes
Is the study design appropriate and is the work technically sound?
Yes
Are sufficient details of methods and analysis provided to allow replication by others?
Yes
If applicable, is the statistical analysis and its interpretation appropriate?
Partly
Are all the source data underlying the results available to ensure full reproducibility?
Yes
Are the conclusions drawn adequately supported by the results?
Yes
Competing Interests: No competing interests were disclosed.
Reviewer Expertise: Critical care clinical trials; health research methodology; rehabilitation; physiotherapy
Is the work clearly and accurately presented and does it cite the current literature?
Yes
Is the study design appropriate and is the work technically sound?
Yes
Are sufficient details of methods and analysis provided to allow replication by others?
Yes
If applicable, is the statistical analysis and its interpretation appropriate?
Partly
Are all the source data underlying the results available to ensure full reproducibility?
Yes
Are the conclusions drawn adequately supported by the results?
Yes
Competing Interests: I am a Co-Investigator on the MRC-funded "Routinely embedding recruitment and retention interventions within randomised controlled trials" [PROMETHEUS] project with the lead author, Shaun Treweek.
Reviewer Expertise: Clinical trial methodology, particularly around trial conduct and retention methods.
Alongside their report, reviewers assign a status to the article:
Invited Reviewers | ||
---|---|---|
1 | 2 | |
Version 2 (revision) 15 Jul 21 |
read | |
Version 1 08 Feb 21 |
read | read |
Provide sufficient details of any financial or non-financial competing interests to enable users to assess whether your comments might lead a reasonable person to question your impartiality. Consider the following examples, but note that this is not an exhaustive list:
Sign up for content alerts and receive a weekly or monthly email with all newly published articles
Already registered? Sign in
The email address should be the one you originally registered with F1000.
You registered with F1000 via Google, so we cannot reset your password.
To sign in, please click here.
If you still need help with your Google account password, please click here.
You registered with F1000 via Facebook, so we cannot reset your password.
To sign in, please click here.
If you still need help with your Facebook account password, please click here.
If your email address is registered with us, we will email you instructions to reset your password.
If you think you should have received this email but it has not arrived, please check your spam filters and/or contact for further assistance.
Comments on this article Comments (0)