F1000ResearchF1000Research2046-1402F1000 Research LimitedLondon, UK10.12688/f1000research.163134.1Brief ReportArticlesThe effects of a theory-informed cover letter on follow-up response rates in an adult cardiac surgery study: a randomised study within a trial (SWAT 24).
[version 1; peer review: awaiting peer review]
MaishmanRachelConceptualizationData CurationFormal AnalysisInvestigationMethodologyVisualizationWriting – Original Draft PreparationWriting – Review & Editinghttps://orcid.org/0000-0002-5052-59511JoyceKatherineConceptualizationData CurationMethodologyProject AdministrationWriting – Original Draft PreparationWriting – Review & Editinghttps://orcid.org/0000-0002-5539-71781Walker-SmithTerrieConceptualizationData CurationMethodologyProject AdministrationWriting – Original Draft PreparationWriting – Review & Editing1SmartHelenaData CurationWriting – Review & Editing1CullifordLucyConceptualizationData CurationMethodologyWriting – Review & Editinga1
Bristol Trials Centre, University of Bristol Medical School, Bristol, England, UKlucy.culliford@bristol.ac.uk
Clinical studies can experience major loss to follow-up and reduced response rates to participant reported outcomes such as quality of life questionnaires, which can lead to a lack of statistical power for the analysis of these outcomes. A number of studies have been embedded within trials to investigate interventions to improve retention.
Methods
A Study Within a Trial (SWAT) was integrated into the Outcome Monitoring After Cardiac Surgery (OMACS) study, a large observational cardiac surgery study, to investigate whether the use of a theory-informed cover letter with 12-month follow-up questionnaires could improve study retention. OMACS study participants receiving 12-month follow-up questionnaires by post were randomised in a 1:1 ratio to receive either a standard cover letter or a theory informed cover letter with their questionnaires. Study retention was assessed by the proportion of participants who returned 12-month follow-up questionnaires.
Results
Questionnaire return rates were similar between the two groups (67.4% with the standard cover letter compared to 68.5% with the theory-informed cover letter, odds ratio 1.05, 95% confidence interval (CI) [0.85 to 1.31]). The results of the OMACS SWAT were consistent with the previously reported meta-analysis (MA) by Gillies
et al., with a revised risk difference of 3% (95% CI -1% to 6%) when including OMACS in the MA.
Conclusions
There is insufficient evidence to support the use of a theory-informed cover letter to increase study retention. We would recommend study teams explore alternative strategies for improving retention where needed.
Trial registration: ISRCTN90204321, registered on 21/01/2015,
https://doi.org/10.1186/ISRCTN90204321.
SWAT; clinical study; patient reported outcomes; quality of life; follow-upNIHR Bristol Biomedical Research CentreOMACS was supported by the National Institute for Health Research (NIHR) Bristol Biomedical Research Centre. OMACS was designed and delivered in collaboration with the Bristol Trials Centre, a UKCRC registered clinical trials unit (CTU), which is in receipt of NIHR CTU support funding.
The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.Background
Clinical studies can experience major loss to follow-up and reduced response rates to participant reported outcomes such as questionnaires. A review of NIHR HTA funded studies in 2016 found that the average retention rate was 89%.
1 This can lead to a substantial amount of missing data and a lack of statistical power for the analysis of patient reported outcomes (PROs) such as quality of life (QoL) questionnaires. Attrition can be somewhat accounted for in sample size calculations of a clinical study, however, if response rates drop below the assumed rate, statistical power for the PROs will be reduced.
Interventions to improve retention are being investigated but many lack sufficient evidence to prove whether they are effective.
2 One such intervention is the use of a theory informed cover letter to improve return rates of patient completed questionnaires. The Study Within a Trial (SWAT) designed to test this intervention (SWAT 24) is registered on the Northern Ireland Network for Trials Methodology Research SWAT repository.
3 Four studies that had embedded this SWAT previously were included in the Cochrane review of retention strategies.
2 The Cochrane review concluded that the evidence was uncertain as to whether the theory informed cover letter had an impact on retention rates, with an overall risk difference of 0.03 [95% confidence interval -0.02, 0.08].
The Outcome Monitoring After Cardiac Surgery (OMACS) study (REC ref: 14/EM/1222; ISRCTN90204321), a large observational cardiac surgery study, had SWAT 24 embedded into a 12-month follow-up timepoint to determine if the use of a theory informed cover letter increases the proportion of participants returning 12 month follow-up postal questionnaires. Data for the OMACS study were still being collected at the time of the Cochrane review described above,
2 and as such were not included in the review.
The OMACS study has now completed recruitment and this report presents the data collected for SWAT 24. We also provide an update to the meta-analysis conducted by Gillies
et al.2
MethodsStudy design
The OMACS study was a single centre, secondary care, observational study which created a bank of clinical data, QoL data and biological samples for adult patients undergoing cardiac surgery. Participants were recruited to the OMACS study between May 2016 and May 2022. Full details of the study design and characteristics of the OMACS cohort have been published.
4,
5 The OMACS study was funded by the NIHR Bristol Biomedical Research Centre. Participants were sent a QoL questionnaire at 3-
and 12-months post-operatively. Participants who underwent coronary artery bypass grafts were sent a coronary revascularisation outcome questionnaire,
6 all others were sent a 12-item short form 12 questionnaire.
7 No reminders were sent if the questionnaires were not returned.
The SWAT used a two-arm, parallel-group, randomised controlled trial (RCT) design. At the 12-month follow-up timepoint participants were randomly allocated to receive either a traditional cover letter (referred to as standard cover letter) or a theory-informed cover letter with their QoL questionnaire. The randomisation sequence was generated by the study statistician using variable block sizes, and allocation was assigned within the study database. There were no costs associated with the SWAT; follow-up questionnaires were sent as part of the OMACS study.
Study participants
In the OMACS study the aim was to approach all eligible patients over the age of 18 who were scheduled for surgery at the Bristol Heart Institute (University Hospitals Bristol and Weston NHS Foundation Trust). Patients were ineligible if they were a prisoner, had their main residence outside of the United Kingdom or lacked the capacity to consent. There were no other exclusion criteria for this SWAT. All OMACS participants who consented to 12-month postal questionnaires and were due questionnaires after 26
th June 2019 were eligible for the SWAT.
Intervention
Two different 12-month follow up letters were used for the SWAT:
‘Standard cover letter’: based on a trials centre template
‘Theory-informed cover letter’: based on a template developed by the SWAT 24 authors.
8
Masking
Participants were allocated to one of the two SWAT arms at the 12-month follow-up timepoint. The SWAT allocation was generated by the study database and the appropriate cover letter created via the database. Recruiting staff were not aware of the allocation at the point of recruitment through to 12-months post-operatively.
Site staff distributing follow-up questionnaires could not be masked to allocation due to the need for them to print and pack the correct cover letter. However, as the site staff did not send reminders or have any further planned contact with the participants, it is unlikely that this had any impact on the return rates of questionnaires. Participants were not aware that they were randomised into a SWAT at the 12-month timepoint, so were not aware that they were receiving one of two possible cover letters in their follow-up
pack.
Outcome measures
The primary outcome of the SWAT was the proportion of participants who returned 12-month QoL questionnaires. Responses to individual questionnaire items are beyond the scope of the SWAT.
Sample size calculation
Participants were randomised to receive one of the two cover letters in a 1:1 ratio. The target sample size was calculated as 2040 (1020 in each group) with an assumed return rate of 84% for the standard cover letter. This would give 90% power to detect a 5% difference in return rates between the two groups with a significance level of 5%.
This sample size was based on the return rate among participants who were approached three months post-operatively for QoL questionnaires and returned their consent form and 3 month questionnaire at the same time. Following a protocol amendment, pre-operative consent became standard procedure for all participants, with participants consenting to passive in-hospital data collection in addition to the post-operative questionnaires. The return rate among participants with pre-operative consent was considerably lower (64%). Using the lower return rate, a revised sample size of 3822 participants would be required to detect a 5% difference in return rates between the groups.
Statistical analyses
The proportion of participants who returned 12-month questionnaires was compared between the two groups using unadjusted logistic regression and presented as an odds ratio (OR) with 95% confidence interval (CI), with the ‘standard cover letter’ as the reference group. Results were also reported as a risk difference (RD) using a generalised linear model (binomial family with identity link). A random-effects meta-analysis (MA) was performed to include the results of this SWAT in the previously published Cochrane review.
2 As per the original MA, results were presented as a RD. A comprehensive risk of bias assessment was performed on all studies included in the MA by Gillies
et al.. This SWAT was assessed against the Cochrane risk of bias criteria
9 and the MA grade assessment was also reviewed to include the additional study.
Ethical approval and consent
The OMACS study was approved by the East Midlands Nottingham 2 Research Ethics Committee on 4
th December 2014 (REC number 14/EM/1222). SWAT 24 was added to the protocol as part of a substantial amendment which was approved by the above ethics committee on 27
th June 2018. Participants provided written consent for the OMACS study, which included the completion of 12month follow-up questionnaires. Participants were not informed about the embedded SWAT, however all study materials including the theory informed cover letter were approved by the Ethics Committee.
Results
The SWAT was terminated early due to closure of the OMACS study. In total 1494 participants were randomised into the SWAT, 748 to the standard cover letter and 746 to the theory-informed cover letter, between July 2019 and August 2022. Demographic data were similar between the two groups (
Table 1). Overall, 27% of the participants were female and the average age of participants was 63.7 years of age. Coronary artery bypass grafting was performed on 51% of patients, with 53% of patients undergoing valve surgery; patients could undergo more than one procedure during the surgery. The majority of procedures were either elective (57% of patients) or urgent (42% of patients). Clinical characteristics at baseline were balanced across the groups (data not shown).
Follow-up questionnaires were returned by 504/748 participants (67.4%) in the standard cover letter group compared to 511/746 participants (68.5%) in the theory-informed cover letter group. Return rates did not differ significantly between the groups (odds ratio (OR) 1.05, 95% CI 0.85 to 1.31; risk difference (RD) 0.011, 95% CI -0.036 to 0.059). Due to the early termination of the OMACS study and the lower than anticipated retention rate, the SWAT was underpowered to detect the hypothesised increase in return rates with the theory-informed cover letter.
In the updated meta-analysis, there was no significant improvement in study retention with the theory-informed cover letter (RD 3%, 95% CI -1% to 6%) (
Figure 1). Gillies
et al.2 reported very low certainty of evidence in their GRADE assessment. The OMACS study was assessed for risk of bias using Cochrane guidelines and was deemed to be at low risk of bias across all domains.
Meta-analysis comparing patient retention when using theory-informed cover letter compared to a standard cover.
Discussion
Gillies
et al. published a meta-analysis of a group of SWATs all looking to improve trial retention and participant engagement,
2 this included SWAT 24
3 which tested the use of a theory informed cover letter for follow-up questionnaires to improve retention. SWAT 24 was implemented in the OMACS study and while the SWAT was introduced in the OMACS study prior to the publication of the above Cochrane review, it was ongoing at the time of publication and so could not be included. The study team decided to continue with the SWAT and update the published meta-analysis once the SWAT had been completed.
When interpreting the results of the meta-analysis, it should be noted that OMACS study was a large observational study, whereas all other host studies included in the prior Cochrane review
2 were randomised controlled trials (RCTs). The overall retention rates were similar between the OMACS study (68%) and the included RCTs (~70% for all except AMBER, which has a response rate of 81%), supporting the inclusion of both study designs in this MA. One potential difference between the OMACS study and other cohort studies is that participants in the OMACS study were given the option to opt-out of the follow-up questionnaires. The group of participants sent questionnaires were therefore a self-selecting group of participants engaged in the study and who confirmed their participation in questionnaires explicitly on the study consent form.
It should also be noted that each of the studies included in the MA were carried out in different patient populations (cardiac surgery,
5 dentistry,
10 urinary incontinence
2 and multiple sclerosis
2). Retention rates were similar across the studies despite the different patient populations, suggesting that the findings of this study could be generalised to patient populations in a range of clinical areas.
The retention rate in the OMACS study was lower than the rate reported by Walters.
1 A possible reason for the lower retention rate observed in OMACS is that the questionnaires in this study were not the primary focus of the study, and so additional strategies such as reminder letters to non-responders were not implemented. Retention rates were however similar to those observed in the other studies included in the meta-analysis of SWAT 24.
The results of the SWAT in the OMACS study are consistent with the published meta-analysis. The overall effect size remains unchanged from the meta-analysis, with a slight narrowing of the CI due to the larger sample size (RD from original meta-analysis of 3%, 95% CI -2% to 8%). Given the lack of benefit demonstrated, we would recommend exploring other strategies to improve retention. A number of alternative strategies were included in the published meta-analysis by Gillies
et al. and we would recommend exploring these further.
2
Data availabilityUnderlying data
Due to the sensitivity of the data involved, these data are published as a restricted dataset at the University of Bristol Research Data Repository (data.bris.ac.uk/data) and have been assigned the following doi:
10.5523/bris.22qap3ktj6ncr2bd6ubxfeifk9.
11 The metadata record published openly by the repository at this location clearly states how data can be accessed by bona fide researchers. Requests for access will be considered by the University of Bristol Research Data Service, who will assess the motives of potential data re-users before deciding to grant access to the data. No authentic request for access will be refused and re-users will not be charged for any part of this process.
This project contains the following extended data:
OMACS_protocol_v7.0 clean.pdf [Study protocol for the OMACS study, in which the SWAT was embedded)
FUP letter SWAT SAP v1.pdf [Statistical Analysis Plan for SWAT]
OMACS_PIL B_v11.0 clean.pdf [Participant Information Leaflet for the OMACS study]
OMACS_PCF_v8.0 clean.pdf [Participant Consent Form for the OMACS study]
OMACS theory informed cover letter.pdf [Cover letter sent to participants randomly allocated to the theory informed cover letter]
OMACS_CROQ 12 month_v3.0.pdf [12 month questionnaire sent to participants for completion]
OMACS_SF12v2 12 month_v3.0.pdf [12 month questionnaire sent to participants for completion]
CONSORT for SWATS_OMACS Followup_table [CONSORT checklist using CONSORT extension for SWATS]
CONSORT_2025_OMACS_FollowupSWAT [CONSORT checklist using CONSORT extension for SWATS]
Data is available under the terms of the
Creative Commons Attribution 4.0
Acknowledgements
We would like to thank all of the team who designed SWAT 24 and advised on implementation. the members of the database team within the Bristol Trials Centre who developed the database and randomisation system, and the Bristol Heart Institute study administrators and research nurses who facilitated this work by administering the questionnaires and cover letters.
ReferencesWaltersSJBonacho Dos Anjos Henriques-CadbyIBortolamiO:
Recruitment and retention of participants in randomised controlled trials: a review of trials funded and published by the United Kingdom Health Technology Assessment Programme.BMJ Open.Mar 20 2017;7(3):e015276.
2832080010.1136/bmjopen-2016-015276PMC5372123GilliesKKearneyAKeenanC:
Strategies to improve retention in randomised trials.Cochrane Database Syst. Rev.Mar 6 2021;2021(3):MR000032.
3367553610.1002/14651858.MR000032.pub3PMC8092429ForgeT:
SWAT 24: Using a theoretically informed cover letter to improve response rates to annual postal questionnaires.Reference SourceJoyceKMaishmanRWalker-SmithT:
Cohort profile: Outcome Monitoring After Cardiac Surgery (OMACS) – a prospective UK cohort study of cardiac surgery patients at the Bristol Heart Institute.BMJ Open.2025;15(2):e091518.
3990952010.1136/bmjopen-2024-091518PMC11800208Walker-SmithTJoyceKMaishmanR:
Outcome Monitoring After Cardiac Surgery (OMACS): a single-centre prospective cohort study of cardiac surgery patients.BMJ Open.Dec 19 2022;12(12):e063268.
3653571310.1136/bmjopen-2022-063268PMC9764648SchroterSLampingDL:
Coronary revascularisation outcome questionnaire (CROQ): development and validation of a new, patient based measure of outcome in coronary bypass surgery and angioplasty.Heart.Dec 2004;90(12):1460–1466.
1554702910.1136/hrt.2003.021899PMC1768578WareJKosinskiMKellerS:
A 12-Item Short-Form Health Survey: Construction of Scales and Preliminary Tests of Reliability and Validity.Med. Care.1996;34(3):220–233.
10.1097/00005650-199603000-00003MaishmanR:
Supplementary materials for OMACS FUP SWAT publication.2025.
10.5523/bris.5zhb7jf82efi299xlsu7qfztoHigginsJPAltmanDGGotzschePC:
The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials.BMJ.Oct 18 2011;343:d5928.
2200821710.1136/bmj.d5928PMC3196245GoulaoBDuncanAFloateR:
Three behavior change theory-informed randomized studies within a trial to improve response rates to trial postal questionnaires.J. Clin. Epidemiol.Jun 2020;122:35–41.
3202793110.1016/j.jclinepi.2020.01.018MaishmanR:
FUP SWAT dataset.MaishmanR, editor.2025.
10.5523/bris.22qap3ktj6ncr2bd6ubxfeifk9