Keywords
SWAT; clinical study; patient reported outcomes; quality of life; follow-up
-
Views
0
Downloads
How to cite this article
Maishman R, Joyce K, Walker-Smith T et al. The effects of a theory-informed cover letter on follow-up response rates in an adult cardiac surgery study: a randomised study within a trial (SWAT 24). [version 1; peer review: awaiting peer review]. F1000Research 2025, 14:1188 (https://doi.org/10.12688/f1000research.163134.1)
NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.
Export Citation
Sciwheel
EndNote
Ref. Manager
Bibtex
ProCite
Sente
▬
✚
Brief Report
The effects of a theory-informed cover letter on follow-up response rates in an adult cardiac surgery study: a randomised study within a trial (SWAT 24).
[version 1; peer review: awaiting peer review]
PUBLISHED 31 Oct 2025
OPEN PEER REVIEW
REVIEWER STATUS
This article is included in the Studies Within A Trial (SWAT) collection.
Abstract
Corresponding author:
Lucy Culliford
Competing interests:
No competing interests were disclosed.
Grant information:
OMACS was supported by the National Institute for Health Research (NIHR) Bristol Biomedical Research Centre. OMACS was designed and delivered in collaboration with the Bristol Trials Centre, a UKCRC registered clinical trials unit (CTU), which is in receipt of NIHR CTU support funding.
The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Copyright:
© 2025 Maishman R et al.
This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
How to cite: Maishman R, Joyce K, Walker-Smith T et al. The effects of a theory-informed cover letter on follow-up response rates in an adult cardiac surgery study: a randomised study within a trial (SWAT 24). [version 1; peer review: awaiting peer review]. F1000Research 2025, 14:1188 (https://doi.org/10.12688/f1000research.163134.1)
First published: 31 Oct 2025, 14:1188 (https://doi.org/10.12688/f1000research.163134.1)
Latest published: 31 Oct 2025, 14:1188 (https://doi.org/10.12688/f1000research.163134.1)
The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Background
Clinical studies can experience major loss to follow-up and reduced response rates to participant reported outcomes such as questionnaires. A review of NIHR HTA funded studies in 2016 found that the average retention rate was 89%.1 This can lead to a substantial amount of missing data and a lack of statistical power for the analysis of patient reported outcomes (PROs) such as quality of life (QoL) questionnaires. Attrition can be somewhat accounted for in sample size calculations of a clinical study, however, if response rates drop below the assumed rate, statistical power for the PROs will be reduced.
Interventions to improve retention are being investigated but many lack sufficient evidence to prove whether they are effective.2 One such intervention is the use of a theory informed cover letter to improve return rates of patient completed questionnaires. The Study Within a Trial (SWAT) designed to test this intervention (SWAT 24) is registered on the Northern Ireland Network for Trials Methodology Research SWAT repository.3 Four studies that had embedded this SWAT previously were included in the Cochrane review of retention strategies.2 The Cochrane review concluded that the evidence was uncertain as to whether the theory informed cover letter had an impact on retention rates, with an overall risk difference of 0.03 [95% confidence interval -0.02, 0.08].
The Outcome Monitoring After Cardiac Surgery (OMACS) study (REC ref: 14/EM/1222; ISRCTN90204321), a large observational cardiac surgery study, had SWAT 24 embedded into a 12-month follow-up timepoint to determine if the use of a theory informed cover letter increases the proportion of participants returning 12 month follow-up postal questionnaires. Data for the OMACS study were still being collected at the time of the Cochrane review described above,2 and as such were not included in the review.
The OMACS study has now completed recruitment and this report presents the data collected for SWAT 24. We also provide an update to the meta-analysis conducted by Gillies et al.2
Methods
Study design
The OMACS study was a single centre, secondary care, observational study which created a bank of clinical data, QoL data and biological samples for adult patients undergoing cardiac surgery. Participants were recruited to the OMACS study between May 2016 and May 2022. Full details of the study design and characteristics of the OMACS cohort have been published.4,5 The OMACS study was funded by the NIHR Bristol Biomedical Research Centre. Participants were sent a QoL questionnaire at 3- and 12-months post-operatively. Participants who underwent coronary artery bypass grafts were sent a coronary revascularisation outcome questionnaire,6 all others were sent a 12-item short form 12 questionnaire.7 No reminders were sent if the questionnaires were not returned.
The SWAT used a two-arm, parallel-group, randomised controlled trial (RCT) design. At the 12-month follow-up timepoint participants were randomly allocated to receive either a traditional cover letter (referred to as standard cover letter) or a theory-informed cover letter with their QoL questionnaire. The randomisation sequence was generated by the study statistician using variable block sizes, and allocation was assigned within the study database. There were no costs associated with the SWAT; follow-up questionnaires were sent as part of the OMACS study.
Study participants
In the OMACS study the aim was to approach all eligible patients over the age of 18 who were scheduled for surgery at the Bristol Heart Institute (University Hospitals Bristol and Weston NHS Foundation Trust). Patients were ineligible if they were a prisoner, had their main residence outside of the United Kingdom or lacked the capacity to consent. There were no other exclusion criteria for this SWAT. All OMACS participants who consented to 12-month postal questionnaires and were due questionnaires after 26th June 2019 were eligible for the SWAT.
Intervention
Two different 12-month follow up letters were used for the SWAT:
• ‘Standard cover letter’: based on a trials centre template
• ‘Theory-informed cover letter’: based on a template developed by the SWAT 24 authors.8
Masking
Participants were allocated to one of the two SWAT arms at the 12-month follow-up timepoint. The SWAT allocation was generated by the study database and the appropriate cover letter created via the database. Recruiting staff were not aware of the allocation at the point of recruitment through to 12-months post-operatively.
Site staff distributing follow-up questionnaires could not be masked to allocation due to the need for them to print and pack the correct cover letter. However, as the site staff did not send reminders or have any further planned contact with the participants, it is unlikely that this had any impact on the return rates of questionnaires. Participants were not aware that they were randomised into a SWAT at the 12-month timepoint, so were not aware that they were receiving one of two possible cover letters in their follow-up pack.
Outcome measures
The primary outcome of the SWAT was the proportion of participants who returned 12-month QoL questionnaires. Responses to individual questionnaire items are beyond the scope of the SWAT.
Sample size calculation
Participants were randomised to receive one of the two cover letters in a 1:1 ratio. The target sample size was calculated as 2040 (1020 in each group) with an assumed return rate of 84% for the standard cover letter. This would give 90% power to detect a 5% difference in return rates between the two groups with a significance level of 5%.
This sample size was based on the return rate among participants who were approached three months post-operatively for QoL questionnaires and returned their consent form and 3 month questionnaire at the same time. Following a protocol amendment, pre-operative consent became standard procedure for all participants, with participants consenting to passive in-hospital data collection in addition to the post-operative questionnaires. The return rate among participants with pre-operative consent was considerably lower (64%). Using the lower return rate, a revised sample size of 3822 participants would be required to detect a 5% difference in return rates between the groups.
Statistical analyses
The proportion of participants who returned 12-month questionnaires was compared between the two groups using unadjusted logistic regression and presented as an odds ratio (OR) with 95% confidence interval (CI), with the ‘standard cover letter’ as the reference group. Results were also reported as a risk difference (RD) using a generalised linear model (binomial family with identity link). A random-effects meta-analysis (MA) was performed to include the results of this SWAT in the previously published Cochrane review.2 As per the original MA, results were presented as a RD. A comprehensive risk of bias assessment was performed on all studies included in the MA by Gillies et al.. This SWAT was assessed against the Cochrane risk of bias criteria9 and the MA grade assessment was also reviewed to include the additional study.
Ethical approval and consent
The OMACS study was approved by the East Midlands Nottingham 2 Research Ethics Committee on 4th December 2014 (REC number 14/EM/1222). SWAT 24 was added to the protocol as part of a substantial amendment which was approved by the above ethics committee on 27th June 2018. Participants provided written consent for the OMACS study, which included the completion of 12month follow-up questionnaires. Participants were not informed about the embedded SWAT, however all study materials including the theory informed cover letter were approved by the Ethics Committee.
Results
The SWAT was terminated early due to closure of the OMACS study. In total 1494 participants were randomised into the SWAT, 748 to the standard cover letter and 746 to the theory-informed cover letter, between July 2019 and August 2022. Demographic data were similar between the two groups ( Table 1). Overall, 27% of the participants were female and the average age of participants was 63.7 years of age. Coronary artery bypass grafting was performed on 51% of patients, with 53% of patients undergoing valve surgery; patients could undergo more than one procedure during the surgery. The majority of procedures were either elective (57% of patients) or urgent (42% of patients). Clinical characteristics at baseline were balanced across the groups (data not shown).
Table 1. Demographic and operative characteristics.
Follow-up questionnaires were returned by 504/748 participants (67.4%) in the standard cover letter group compared to 511/746 participants (68.5%) in the theory-informed cover letter group. Return rates did not differ significantly between the groups (odds ratio (OR) 1.05, 95% CI 0.85 to 1.31; risk difference (RD) 0.011, 95% CI -0.036 to 0.059). Due to the early termination of the OMACS study and the lower than anticipated retention rate, the SWAT was underpowered to detect the hypothesised increase in return rates with the theory-informed cover letter.
In the updated meta-analysis, there was no significant improvement in study retention with the theory-informed cover letter (RD 3%, 95% CI -1% to 6%) ( Figure 1). Gillies et al.2 reported very low certainty of evidence in their GRADE assessment. The OMACS study was assessed for risk of bias using Cochrane guidelines and was deemed to be at low risk of bias across all domains.
Figure 1. Meta-analysis comparing patient retention when using theory-informed cover letter compared to a standard cover.
Discussion
Gillies et al. published a meta-analysis of a group of SWATs all looking to improve trial retention and participant engagement,2 this included SWAT 243 which tested the use of a theory informed cover letter for follow-up questionnaires to improve retention. SWAT 24 was implemented in the OMACS study and while the SWAT was introduced in the OMACS study prior to the publication of the above Cochrane review, it was ongoing at the time of publication and so could not be included. The study team decided to continue with the SWAT and update the published meta-analysis once the SWAT had been completed.
When interpreting the results of the meta-analysis, it should be noted that OMACS study was a large observational study, whereas all other host studies included in the prior Cochrane review2 were randomised controlled trials (RCTs). The overall retention rates were similar between the OMACS study (68%) and the included RCTs (~70% for all except AMBER, which has a response rate of 81%), supporting the inclusion of both study designs in this MA. One potential difference between the OMACS study and other cohort studies is that participants in the OMACS study were given the option to opt-out of the follow-up questionnaires. The group of participants sent questionnaires were therefore a self-selecting group of participants engaged in the study and who confirmed their participation in questionnaires explicitly on the study consent form.
It should also be noted that each of the studies included in the MA were carried out in different patient populations (cardiac surgery,5 dentistry,10 urinary incontinence2 and multiple sclerosis2). Retention rates were similar across the studies despite the different patient populations, suggesting that the findings of this study could be generalised to patient populations in a range of clinical areas.
The retention rate in the OMACS study was lower than the rate reported by Walters.1 A possible reason for the lower retention rate observed in OMACS is that the questionnaires in this study were not the primary focus of the study, and so additional strategies such as reminder letters to non-responders were not implemented. Retention rates were however similar to those observed in the other studies included in the meta-analysis of SWAT 24.
The results of the SWAT in the OMACS study are consistent with the published meta-analysis. The overall effect size remains unchanged from the meta-analysis, with a slight narrowing of the CI due to the larger sample size (RD from original meta-analysis of 3%, 95% CI -2% to 8%). Given the lack of benefit demonstrated, we would recommend exploring other strategies to improve retention. A number of alternative strategies were included in the published meta-analysis by Gillies et al. and we would recommend exploring these further.2
Download
Export To
metrics
| Views | Downloads | |
|---|---|---|
| F1000Research | - | - |
|
PubMed Central
Data from PMC are received and updated monthly.
|
- | - |
Citations
CITE
how to cite this article
Maishman R, Joyce K, Walker-Smith T et al. The effects of a theory-informed cover letter on follow-up response rates in an adult cardiac surgery study: a randomised study within a trial (SWAT 24). [version 1; peer review: awaiting peer review]. F1000Research 2025, 14:1188 (https://doi.org/10.12688/f1000research.163134.1)
NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.
track
receive updates on this article
Track an article to receive email alerts on any updates to this article.
Open Peer Review
Current Reviewer Status:
AWAITING PEER REVIEW
AWAITING PEER REVIEW
Key to Reviewer Statuses
VIEW
HIDE
ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations
A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.
Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions
Comments on this article Comments (0)