F1000Research F1000Research 2046-1402 F1000 Research Limited London, UK 10.12688/f1000research.14591.2 Research Article Articles A study update newsletter or Post-it® note did not increase postal questionnaire response rates in a falls prevention trial: an embedded randomised factorial trial

[version 2; peer review: 2 approved]

 Rodgers Sara Conceptualization Data Curation Methodology Project Administration Writing – Original Draft Preparation Writing – Review & Editing https://orcid.org/0000-0003-1757-541X 1 Sbizzera Illary Formal Analysis Methodology Writing – Original Draft Preparation Writing – Review & Editing 1 Cockayne Sarah Conceptualization Investigation Methodology Project Administration Writing – Original Draft Preparation Writing – Review & Editing https://orcid.org/0000-0002-1288-5497 1 Fairhurst Caroline Conceptualization Methodology Writing – Review & Editing https://orcid.org/0000-0003-0547-462X a 1 Lamb Sarah E. Methodology Writing – Review & Editing 2 Vernon Wesley Methodology Writing – Review & Editing 3 Watson Judith Methodology Writing – Review & Editing https://orcid.org/0000-0003-0694-3854 1 Hewitt Catherine Methodology Writing – Review & Editing 1 Torgerson David J. Conceptualization Formal Analysis Methodology Writing – Original Draft Preparation Writing – Review & Editing https://orcid.org/0000-0002-1667-4275 1 York Trials Unit, Department of Health Sciences, University of York, York, YO10 5DD, UK Centre for Rehabilitation Research, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK Division of Podiatry and Clinical Sciences, School of Human and Health Sciences, University of Huddersfield, Huddersfield, HD1 3DH, UK caroline.fairhurst@york.ac.uk

No competing interests were disclosed.

 19 2 2019 2018 7 1083 8 2 2019 Copyright: © 2019 Rodgers S et al. 2019 This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Background: Participants not returning data collection questionnaires is a problem for many randomised controlled trials. The resultant loss of data leads to a reduction in statistical power and can result in bias. The aim of this study was to assess whether the use of a study update newsletter and/or a handwritten or printed Post-it® note sticker increased postal questionnaire response rates for participants of a randomised controlled trial.

Method: This study was a factorial trial embedded within a host trial of a falls-prevention intervention among men and women aged ≥65 years under podiatric care. Participants were randomised into one of six groups: newsletter plus handwritten Post-it®; newsletter plus printed Post-it®; newsletter only; handwritten Post-it® only; printed Post-it® only; or no newsletter or Post-it®. The results were combined with those from previous embedded randomised controlled trials in meta-analyses.

Results: The overall 12-month response rate was 803/826 (97.2%) (newsletter 95.1%, no newsletter 99.3%, printed Post-it® 97.5%, handwritten Post-it® 97.1%, no Post-it® 97.1%). The study update newsletter had a detrimental effect on response rates (adjusted odds ratio 0.14, 95% CI 0.04 to 0.48, p<0.01) and time to return the questionnaire (adjusted hazard ratio 0.86, 95% CI 0.75 to 0.99, p=0.04). No other statistically significant differences were observed between the intervention groups on response rates, time to response, and the need for a reminder.

Conclusions: Post-it® notes have been shown to be ineffective in three embedded trials, whereas the evidence for newsletter reminders is still uncertain.

 Randomised controlled trial; randomisation; embedded trial; newsletter; Post-it® note; response rate National Institute for Health Research 09/77/01 York Trials Unit The REFORM study was funded by the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme (Programme grant number 09/77/01). This SWAT was funded by York Trials Unit. Amendments from Version 1

Following comments from the peer-reviewers, the manuscript was revised as follows. The title was changed to: A study update newsletter or Post-it® note did not increase postal questionnaire response rates in a falls prevention trial: an embedded randomised factorial trial. Clarification of the content and function of the newsletter was added to explain that this provided a study update rather than prenotification of the impending arrival of the 12 month questionnaire. This was suggested as a possible explanation for the detrimental effect of the newsletter on response rates in the Discussion. The term ‘prenotification’ was removed throughout the manuscript as necessary, and the newsletter was included as Supplementary File 2. The meta-analysis of the newsletter trials was changed from a fixed- to a random-effects approach, on account of the substantial heterogeneity among the studies, and Figure 5 was revised. A fixed-effect meta-analysis was conducted as a sensitivity analysis for the Post-it® note meta-analysis since there was no heterogeneity among the studies. We completed GRADE assessments for the two meta-analyses, discussed these in the text, and included the assessment tables as Supplementary Files 4-6. The CONSORT diagram (Figure 1) was revised to include the proportion of returned questionnaires for each trial arm, and unadjusted odds ratios were presented alongside the adjusted odds ratios in the Results section. A sentence was added to confirm that all questionnaire responses were included in the analyses regardless of how long the questionnaire took to be returned.

 Introduction

Postal questionnaires represent a cost-effective and convenient way of collecting participant-reported outcome data in health research, such as in randomised controlled trials (RCTs). However, attrition (i.e. when participants do not return the questionnaires) is a problem for many RCTs. The resultant loss of data leads to a reduction in statistical power and can result in bias 1 . Although a number of strategies have been found to reduce attrition 1, 2 , few of these have been evaluated in the context of healthcare RCTs. A recent systematic review highlighted the need for further research into methods of retaining participants in RCTs 3 .

A Cochrane systematic review 4 evaluating 110 different strategies to improve response rates to postal questionnaires in RCTs identified pre-notification as an effective strategy. The odds of response were increased by nearly half when participants were pre-notified of the impending arrival of the questionnaire (odds ratio (OR) 1.45, 95% CI 1.29 to 1.63); however, there was significant heterogeneity among the results of the 47 included trials (p<0.001). Although there have been several studies evaluating different methods of pre-notification (such as letters, postcards or telephone calls) very few of these have been conducted in a healthcare setting. Only one RCT has evaluated the effectiveness of a pre-notification newsletter to increase response rates 5 . This study found a statistically significant increase in response rates (OR 1.45, 95% CI 1.01 to 2.10) among participants allocated to receive the pre-notification newsletter.

In this trial, we sent participants a study update newsletter shortly before their 12-month questionnaire was due. While this newsletter was not specifically designed to pre-notify participants of the impending arrival of their 12-month questionnaire, it did serve as a reminder about the REFORM study as participants may not otherwise have received any correspondence since the 6 month time point.

The Cochrane review 4 also reported that the appearance of the questionnaire (e.g. making questionnaire materials more personal by using handwritten signatures) can affect response rates. For example, the odds of response increased by a quarter when addresses were handwritten compared to using computer-printed labels (OR 1.25, 95% CI 1.08 to 1.45). We are also aware of six studies that evaluated the effectiveness of attaching a Post-it® note to questionnaires to increase response rate 6– 8 ; four of these were undertaken within an academic setting and reported a statistically significant increase (p<0.05) in responses rates when personalised Post-it® notes were used 3, 6 .

At the York Trials Unit we have a programme of undertaking studies within a trial (SWATs) 9 that aim to evaluate simple interventions to increase response rates to postal questionnaires. Newsletters and Post-it® notes are relatively inexpensive, so even a small benefit is likely to be cost-effective. A single embedded trial will often not have the statistical power to detect a modest difference if there truly was one present; therefore, we have a strategy of repeating our SWATs in order to conduct meta-analyses to strengthen the evidence base. With respect to newsletters sent prior to questionnaires, our previous trial showed a small absolute difference in favour of the intervention, which was borderline statistically significant (p=0.05) 5 , whereas our two previous studies of Post-it® notes 7, 8 produced identical, non-statistically significant ORs (0.97) favouring the control group (no Post-it® note).

We conducted a SWAT to evaluate the effectiveness of a study update newsletter and/or applying a handwritten or printed Post-it® note to the questionnaire as a means of increasing response rates to the 12-month follow-up questionnaire sent to participants in the REFORM trial. This paper presents the results of this sub-study. We also present the results of a meta-analysis of the three ‘Post-it® notes’ and two ‘newsletters sent prior to questionnaires’ studies previously undertaken at York Trials Unit to increase questionnaire response rates in RCTs of health treatments.

 Methods Ethical approval

This trial was embedded within the National Institute for Health Research Health Technology Assessment (NIHR HTA) programme funded REFORM (REducing Falls with ORthoses and a Multifaceted podiatry intervention) study (registration number ISRCTN68240461; registration date, 1 st July 2011; http://www.isrctn.com/ISRCTN68240461) 10 , which aimed to evaluate the clinical and cost effectiveness of a podiatry intervention for the prevention of falls in older people. Ethical approval for the REFORM study and this embedded sub-study was given by National Research Ethics Service East of England – Cambridge East Research Ethics Committee (REC reference 11/EE/0379) and the University of York, Department of Health Sciences Research Governance Committee.

 Participants

Participants in the REFORM study who were due to be sent their 12-month follow-up questionnaire were included in this nested RCT. Participants who had asked to be withdrawn from the REFORM study or who did not wish to receive a questionnaire at this time point were excluded. Supplementary File 1 contains the full trial protocol of the REFORM study.

 Design and randomisation

We undertook a three-by-two SWAT. Participants were allocated to one of six arms using block randomisation with a block size of 18, stratified by REFORM treatment group allocation. An independent data manager who was not involved in the recruitment of participants generated the allocation sequence by computer and allocated participants in a 1:1:1:1:1:1 ratio.

 Interventions

Participants were assigned to one of the following six groups: study update newsletter plus handwritten Post-it® note applied to the questionnaire; newsletter plus printed Post-it®; newsletter only; handwritten Post-it® note only; printed Post-it® note only; or neither newsletter nor Post-it® note. The newsletter contained information regarding trial progress, including the geographical location and number of participants recruited and what happens at the end of the study [ Supplementary File 2]. The newsletter was posted to participants 3 weeks prior to posting the 12-month questionnaire. Those participants randomised to not receive the newsletter were sent this eight weeks after the questionnaire was sent out. The wording on the Post-it® note was “Please take a few minutes to complete this for us. Thank you! Sarah”. (Sarah was the name of the REFORM Trial Manager.) In order to minimise the possibility of heterogeneity, the wording (except for the name), text size and font on the printed Post-it® note was the same as that used for the studies by Tilbrook et al. 7 and Lewis et al. 8 and the Post-it® note was placed in the same location, on the top right hand corner of the questionnaire. Two researchers and three trial secretaries wrote the text of the handwritten Post-it® notes and every effort was made to ensure the format of the message was consistent. All participants also received an unconditional £5 note with their final follow up.

 Management of the postal questionnaires

The date participants were sent and returned their postal questionnaires was recorded. All participants who did not return their follow-up questionnaire within 2 weeks were sent up to two standard reminders, 2 weeks apart, by post, text or email according to the participant’s preference, followed by a telephone reminder 1 week later.

 Primary outcome

The primary outcome was questionnaire response rate defined as the proportion of participants that returned their 12-month postal follow-up questionnaire to York Trials Unit.

 Secondary outcomes

The secondary outcomes were: time to response, defined as number of days between the questionnaire being mailed out to a participant and the questionnaire being recorded as returned to York Trials Unit; and the proportion of participants that needed a reminder.

 Statistical analysis

All statistical analyses were conducted in Stata version 14 (StataCorp. 2015. Stata Statistical Software: Release 14. College Station, TX: StataCorp LP) using two-sided tests at the 5% significance level on an intention-to-treat basis. Age at randomisation into the main REFORM trial, gender and main trial allocation are summarised by randomised sub-study group. This factorial trial is reported as recommended by Montgomery et al. 11 Response rates were calculated for each intervention. All survey responses were included regardless of how long the questionnaire took to be returned. A logistic regression model containing the two interventions (Post-it® note and newsletter), age, gender and REFORM treatment allocation was performed. Adjusted ORs and corresponding 95% confidence intervals (CIs) were obtained from this model. The presence of an interaction between the two interventions was also tested by introducing the interaction term of the interventions into the logistic model.

Time to return the 12-month follow-up questionnaire was calculated as the number of days from the date the questionnaire was sent out, to the date it was returned. Median time to return was calculated for all participants who returned their questionnaire. For the time-to-event analysis, questionnaires that were not returned or returned 6 weeks (42 days) or more after being sent were treated as censored. Time to questionnaire return was plotted for both interventions using Kaplan-Meier survival curves, and the log-rank test was used to compare the randomised groups within each intervention. A Cox proportional hazards regression model containing the two interventions, age, gender and REFORM treatment allocation was performed; adjusted hazard ratios (HR) and corresponding 95% CIs were obtained. The proportion of participants requiring a reminder was analysed using a similarly adjusted logistic model.

An aggregated random effects meta-analysis of this study with the study reported by Mitchell et al. 5 evaluated the effect of sending a newsletter before receiving the questionnaire to improve response rates. A second aggregated random effects meta-analysis was conducted incorporating the results of this study and those by Tilbrook et al. 7 and Lewis et al. 8 in order to evaluate the effect of receiving a questionnaire with an attached Post-it® note on response rates. We also performed a GRADE (Grading of Recommendations Assessment, Development and Evaluation) assessment 12 to assess the certainty of the recommendations we have made.

Supplementary File 3 contains a completed CONSORT checklist for this study.

 Results

A total of 1010 participants were recruited into the REFORM study and randomised to receive a multifaceted podiatry intervention or usual care. In total, 917 (90.8%) reached the 12-month time point and were sent a follow-up questionnaire, of which 826 (90.1%) were randomised into this embedded RCT (due to a delay in the start of the sub-study): 135 to receive the newsletter and the handwritten Post-it® note; 138 to receive the newsletter and the printed Post-it® note; 137 to receive the newsletter only; 137 to receive the handwritten Post-it® note only; 136 to receive the printed Post-it® note only; and 143 to receive neither the newsletter nor the Post-it® note ( Figure 1). Participants had a mean age of 78 years (range 65 to 96 years), and were predominantly female (n=509, 61.6%). Age and main trial allocation were balanced between the six groups, whereas a small chance imbalance for gender can be seen: the presence of women tended to be higher in the groups receiving the newsletter (65.6% vs 57.7%) and higher in the group receiving the hand-written Post-it® note (66.5%) than the printed (60.2%) or no Post-it® note (58.2%) ( Table 1).

Flow diagram for the REFORM sub-study. Baseline characteristics of participants.
Variable Newsletter and handwritten Post-it® note (n=135) Newsletter and printed Post-it® note (n=138) Newsletter only (n=137) Handwritten Post-it® note only (n=137) Printed Post-it® note only (n=136) No newsletter or Post-it® note (n=143)
Age, years
   Mean (SD) 78.0 (7.0) 76.9 (6.9) 79.0 (7.0) 77.6 (7.2) 77.5 (6.9) 76.3 (7.0)
   (Min–Max) (65–95) (65–95) (65–96) (65–96) (65–93) (65–89)
   Median 78 77 80 78 77 77
Gender, n (%)
   Male 39 (28.9) 48 (34.8) 54 (39.0) 52 (38.0) 61 (44.9) 63 (44.1)
   Female 96 (71.1) 90 (65.2) 83 (61.0) 85 (62.0) 75 (55.1) 80 (55.9)
Main trial allocation, n (%)
   Control 71 (52.6) 69 (50.0) 71 (51.8) 72 (52.6) 69 (50.7) 75 (52.4)
   Intervention 64 (47.4) 69 (50.0) 66 (48.2) 65 (47.4) 67 (49.3) 68 (47.6)
Questionnaire response rate

The total number of participants returning the 12-month follow-up questionnaire was 803 of 826 (97.2%), 390 of 410 (95.1%) of those who received the newsletter, and 413 of 416 (99.3%) of those who did not receive it. The difference in response rates between these two groups was statistically significant (crude difference in percentages (CDP) 4.2%, 95% CI 1.9% to 6.4%; crude OR 0.14, 95% CI 0.04 to 0.48, p<0.01; adjusted OR 0.14, 95% CI 0.04 to 0.48, p<0.01) ( Table 2). With respect to the Post-it® note intervention, 272 of 280 (97.1%) participants who received no Post-it® note, 267 of 274 (97.4%) participants who received the printed Post-it® note, and 264 of 272 (97.1%) who received the handwritten Post-it® note returned their questionnaire. The Post-it® note intervention did not show a statistically significant effect on the response rate (printed Post-it® vs no Post-it®: CDP 0.3%, 95% CI -2.4% to 3.0%; crude OR 1.15, 95% CI 0.41 to 3.24, p=0.79; adjusted OR 1.06, 95% CI 0.37 to 3.01, p=0.92; handwritten Post-it® vs no Post-it®: CDP 0.0%, 95% CI -2.9% to 2.7%; crude OR 0.98, 95% CI 0.36 to 2.67, p=0.97; adjusted OR 0.91, 95% CI 0.33 to 2.49, p=0.85). There was no statistically significant interaction between the interventions.

The effect of the newsletter and Post-it® note interventions on trial outcomes.
OR/HR Adjusted statistic (SE) 95% CI p-value
Questionnaire return (Y/N) 1
   Newsletter vs no newsletter OR 0.14 (0.09) (0.04, 0.48) <0.01
   Printed Post-it® vs no Post-it® OR 1.06 (0.56) (0.37, 3.01) 0.92
   Handwritten Post-it® vs no Post-it® OR 0.91 (0.47) (0.33, 2.49) 0.85
Time-to-return (days) 1
   Newsletter vs no newsletter HR 0.86 (0.06) (0.75, 0.99) 0.04
   Printed Post-it® vs no Post-it® HR 0.95 (0.08) (0.80, 1.13) 0.55
   Handwritten Post-it® vs no Post-it® HR 0.90 (0.08) (0.76, 1.07) 0.22
Reminder required (Y/N) 2
   Newsletter vs no newsletter OR 1.30 (0.26) (0.88, 1.91) 0.19
   Printed Post-it® vs no Post-it® OR 1.20 (0.30) (0.74, 1.94) 0.47
   Handwritten Post-it® vs no Post-it® OR 1.47 (0.35) (0.92, 2.36) 0.11

1Logistic regression; 2Cox regression. All models contained both the newsletter and Post-it® note intervention terms and were adjusted for age, gender and main trial allocation. SE, standard error; OR, odds ration; HR, hazard ratio.

 Time to return

Time to return ranged from 3 to 101 days. Among the participants who responded, the median time taken to return the 12-month questionnaire was 11 days, both overall and in each intervention group (i.e. no newsletter sent, newsletter sent, no Post-it® note, printed Post-it® note, and handwritten Post-it® note). In total, 793 (96.0%) participants returned the questionnaire within 6 weeks (no newsletter: n=407, 97.8%; newsletter: n=386, 94.2%; no Post-it® note: n=271, 96.8%; printed Post-it® note: n=263, 96.0%; and handwritten Post-it® note: n=259, 95.2%). There was evidence of a difference in time to return between those who received the newsletter and those who did not (adjusted HR 0.86, 95% CI 0.75 to 0.99, p=0.04) ( Figure 2; Table 2). The Post-it® note intervention did not appear to have any effect on time to return (printed Post-it® vs no Post-it®: adjusted HR 0.95, 95% CI 0.80 to 1.13, p=0.55; handwritten Post-it® vs no Post-it®: adjusted HR 0.90, 95% CI 0.76 to 1.07, p=0.22) ( Figure 3; Table 2). There was no statistically significant interaction between the interventions.

Kaplan-Meier survival curve of time to return for the newsletter intervention. Kaplan-Meier survival curve of time to return for the Post-it® note intervention. Reminders sent

Overall 125 (15.1%) participants required a reminder following 2 weeks of questionnaire non-response (newsletter: n=69, 16.8%; no newsletter: n=56, 13.5%; no Post-it® note: n=36, 12.9%; printed Post-it® note: n=41, 15.0%; handwritten Post-it® note: n=48, 17.7%). There was no evidence of a difference in the proportion of participants requiring a reminder between the groups (newsletter vs no newsletter: adjusted OR 1.30, 95% CI 0.88 to 1.91, p=0.19; printed Post-it® vs no Post-it®: adjusted OR 1.20, 95% CI 0.74 to 1.94, p=0.47; handwritten Post-it® vs no Post-it®: adjusted OR 1.47, 95% CI 0.92 to 2.36, p=0.11) ( Table 2).

 Meta-analysis

We combined the two previous Post-it® note studies conducted at York Trials Unit with the study described in this paper. Because there was no material difference in response rates between the printed and handwritten Post-it® note (i.e. 97.5% vs 97.1%) in this study we combined these two groups in the meta-analysis (Post-it® note vs no Post-it® note: adjusted OR 0.98, 95% CI 0.40 to 2.37). The pooled OR was 0.97 (favouring no Post-it® note) but was not statistically significant (95% CI 0.70 to 1.35, p=0.87) ( Figure 4). No heterogeneity was observed (I 2=0%). Because no heterogeneity was observed, a sensitivity analysis running a fixed effects meta-analysis was conducted on these data, which produced identical results (to 2 decimal places). As part of the GRADE assessment we assessed the risk of bias of the four trials included in the meta-analyses [ Supplementary File 4]. The GRADE assessment indicated high certainty (i.e. further research is very unlikely to change our confidence in the estimate of effect) [ Supplementary File 5].

Meta-analyses of Post-it® note interventions.

For the newsletter, the meta-analysis ( Figure 5) showed significant heterogeneity (I 2=92%) with a non-statistically significant effect estimate favouring no intervention (pooled OR 0.49, 95% CI 0.05 to 4.76, p=0.54). The GRADE assessment indicated very low certainty (i.e. any estimate of effect is very uncertain) [ Supplementary File 6] largely due to inconsistency between the results of the two studies and imprecision of the estimates.

Meta-analysis of newsletters sent prior to questionnaires. Discussion

We undertook a three-by-two factorial randomised SWAT of a study update newsletter and/or attaching Post-it® notes (printed or handwritten) to postal questionnaires to improve response rates. The trial was embedded at the final (12-month) follow-up time point of the NIHR HTA-funded REFORM RCT. There was evidence that sending a study newsletter 3 weeks prior to the 12-month questionnaire had a detrimental effect on the response rate (adjusted OR 0.14, 95% CI 0.04 to 0.48, p<0.01) and time to return the questionnaire (adjusted HR 0.86, 95% CI 0.75 to 0.99, p=0.04); however, the raw difference in response rates was small (95.1% vs 99.3%). It is possible that the language used in the study update newsletter could have contributed to this as it was not specifically pertaining to pre-notification of the 12-month questionnaire. Instead, the newsletter was initially intended to be sent with the 12-month questionnaire as an acknowledgment of the end of the participant’s involvement in the trial. It therefore indicated that the participant did not need to return any further data relating to falls they experienced. When it was decided to implement this SWAT, the same newsletter was used but was sent 3 weeks prior to the due date of the 12-month questionnaire. In hindsight, the wording of the newsletter may have led participants to believe that they did not need to return the 12-month questionnaire; this may account for its detrimental effect in this trial. A small imbalance in gender among the six groups was observed at randomisation, but gender was adjusted for in all analyses. A previous SWAT of a pre-notification newsletter 5 , conducted in an older female population, showed a positive finding, which was in line with the Cochrane review 4 of pre-notification approaches to enhance survey returns. A meta-analysis combining that trial with ours produced a small, non-statistically significant effect favouring use of a newsletter; however, there was significant heterogeneity in the results and the GRADE assessment we conducted indicates that the level of certainty for this estimate of effect is very low.

Response rates across the groups receiving a printed Post-it® note on their questionnaire, a handwritten Post-it® note and no Post-it® note were all very similar (97.5, 97.1 and 97.1%, respectively). There was no statistically significant difference between the groups in terms of response rate, time to return the questionnaire, and requiring a reminder. This lack of effect on response rates has now been demonstrated across three separate trials. The first trial was among patients with neck pain (mean age, 53 years) 7 , the second trial was among older patients (mean age, 74 years) at risk of depression 8 with the current trial among a similar age group (mean age, 76 years), but no risk/diagnosis of depression. The consistent results suggest that it is not worthwhile undertaking further trials of this intervention among a middle-aged or older population. This is supported by the GRADE assessment which indicates the high certainty of this outcome. There may be merit, however, in testing this intervention in a younger population where response rates may be lower.

No statistically significant differences were observed in the proportion of participants requiring a reminder between the groups.

Response rates in the six groups all exceeded 94%, making significant improvement difficult. These simple interventions were relatively inexpensive but not cost-free due to the price of printing the newsletters and the printed Post-it® notes, and staff time to handwrite the Post-it® notes. A cost-effectiveness analysis was not performed since a benefit was not observed.

 Conclusions

In summary, we found no evidence of a benefit of handwritten or printed Post-it® notes on questionnaire response rates. We also found a negative effect of the study update newsletter; however, a meta-analysis suggests the evidence is still uncertain.

 Data availability

The data referenced by this article are under copyright with the following copyright statement: Copyright: � 2019 Rodgers S et al.

Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Dataset 1. Raw data concerning patient demographics, type of reminder received and the returning of the questionnaire. DOI: 10.5256/f1000research.14591.d202910 13

Acknowledgements

This article has been written by the authors on behalf of the REFORM Study Team.

 Supplementary material

Supplementary File 1. Complete trial protocol for the REFORM study.

Click here to access the data

Supplementary File 2. REFORM 12 month newsletter.

Click here to access the data

Supplementary File 3. Completed CONSORT checklist.

Click here to access the data

Supplementary File 4: Assessment of Risk of Bias.

Click here to access the data

Supplementary File 5: GRADE assessment – Post-it Notes.

Click here to access the data

Supplementary File 6: GRADE assessment – Newsletters.

Click here to access the data

McColl E Jacoby A Thomas L : Design and use of questionnaires: a review of best practice applicable to surveys of health service staff and patients. Health Technol Assess. 2001;5(31):1–256. 11809125 10.3310/hta5310 Hewitt CE Kumaravel B Dumville JC : Assessing the impact of attrition in randomized controlled trials. J Clin Epidemiol. 2010;63(11):1264–70. 20573482 10.1016/j.jclinepi.2010.01.010 Brueton VC Tierney J Stenning S : Strategies to improve retention in randomised trials. Cochrane Database Syst Rev. 2013; (12): MR000032,1–126. 24297482 10.1002/14651858.MR000032.pub2 4470347 Edwards PJ Roberts I Clarke MJ : Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev. 2009; (3):MR000008. 19588449 10.1002/14651858.MR000008.pub4 Mitchell N Hewitt CE Lenaghan E : Prior notification of trial participants by newsletter increased response rates: a randomized controlled trial. J Clin Epidemiol. 2012;65(12):1348–52. 22901398 10.1016/j.jclinepi.2012.05.008 Garner R : Post-it® Note Persuasion: A sticky influence. J Consum Psychol. 2005;15(3):230–37. 10.1207/s15327663jcp1503_8 Tilbrook HE Becque T Buckley H : Randomized trial within a trial of yellow ‘post-it notes’ did not improve questionnaire response rates among participants in a trial of treatments for neck pain. J Eval Clin Pract. 2015;21(2):202–4. 25399758 10.1111/jep.12284 Lewis H Keding A Bosanquet K : An randomized controlled trial of Post-it® notes did not increase postal response rates in older depressed participants. J Eval Clin Pract. 2017;23(1):102–107. 27569403 10.1111/jep.12618 Adamson J Hewitt CE Torgerson DJ : Producing better evidence on how to improve randomised controlled trials. BMJ. 2015;351:h4923. 26408289 10.1136/bmj.h4923 Cockayne S Rodgers S Green L : Clinical effectiveness and cost-effectiveness of a multifaceted podiatry intervention for falls prevention in older people: a multicentre cohort randomised controlled trial (the REducing Falls with ORthoses and a Multifaceted podiatry intervention trial). Health Technol Assess. 2017;21(24):1–198. 28621259 10.3310/hta21240 5494515 Montgomery AA Peters TJ Little P : Design, analysis and presentation of factorial randomised controlled trials. BMC Med Res Methodol. 2003;3:26. 14633287 10.1186/1471-2288-3-26 305359 Guyatt GH Oxman AD Kunz R : What is "quality of evidence" and why is it important to clinicians? BMJ. 2008;336(7651):995–8. 18456631 10.1136/bmj.39490.551019.BE 2364804 Rodgers S Sbizzera I Cockayne S : Dataset 1 in: A nested randomised controlled trial of a newsletter and Post-it® note did not increase postal questionnaire response rates in a falls prevention trial. F1000Research. 2018. http://www.doi.org/10.5256/f1000research.14591.d202910 10.5256/f1000research.19852.r44676 Reviewer response for version 2 Treweek Shaun P. 1 Referee https://orcid.org/0000-0002-7239-7241 Health Services Research Unit, Institute of Applied Health Sciences, School of Medicine, Medical Sciences and Nutrition, University of Aberdeen, Aberdeen, UK

Competing interests: No competing interests were disclosed.

 1 3 2019 Copyright: © 2019 Treweek SP 2019 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve

The authors have made a good job of addressing my suggestions and I have no further comments.

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

Partly

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Yes

Are sufficient details of methods and analysis provided to allow replication by others?

Yes

Reviewer Expertise:

Trial methodology

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

 10.5256/f1000research.19852.r44677 Reviewer response for version 2 Edwards Phil J. 1 Referee https://orcid.org/0000-0003-4431-8822 Woolf Benjamin 2 Co-referee https://orcid.org/0000-0002-1505-2570 Department of Population Health, London School of Hygiene & Tropical Medicine, London, UK London School of Hygiene & Tropical Medicine, London, UK

Competing interests: No competing interests were disclosed.

 20 2 2019 Copyright: © 2019 Edwards PJ and Woolf B 2019 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve

The revised manuscript takes account of the concerns we raised in our initial review.

Phil Edwards and Benjamin Woolf, London School of Hygiene & Tropical medicine.

Is the work clearly and accurately presented and does it cite the current literature?

Partly

If applicable, is the statistical analysis and its interpretation appropriate?

No

Are all the source data underlying the results available to ensure full reproducibility?

Partly

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Partly

Are sufficient details of methods and analysis provided to allow replication by others?

Partly

Reviewer Expertise:

Systematic reviews of methods to increase response to postal and electronic questionnaires

We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

 10.5256/f1000research.15878.r40423 Reviewer response for version 1 Edwards Phil J. 1 Referee https://orcid.org/0000-0003-4431-8822 Woolf Benjamin 2 Co-referee https://orcid.org/0000-0002-1505-2570 Department of Population Health, London School of Hygiene & Tropical Medicine, London, UK London School of Hygiene & Tropical Medicine, London, UK

Competing interests: No competing interests were disclosed.

 10 12 2018 Copyright: © 2018 Edwards PJ and Woolf B 2018 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve-with-reservations

Comments on the manuscript: 

Thank you for inviting us to review this manuscript. The authors should be commended for nesting this factorial study within the REFORM study, to provide further evidence on methods to increase response to postal questionnaires. It is vital that researchers conduct this type of nested trial for evidence-based methods research to progress in a cost-effective manner. 

Overall, we found this to be a well-designed, and well-conducted study. We have a few comments regarding the reporting of the results.

Title:  We don’t think this is currently the best wording for the title - the “nested RCT” was not the intervention.

ABSTRACT 

Conclusions:

The authors say that the evidence for “newsletter reminders” is still uncertain, but we think that the authors meant to say “prenotification newsletters”.  

 

METHODS  

Interventions:

The authors could make it clearer to what extent the newsletter was a prenotification intervention; for example, was there a letter with the newsletter explaining that the questionnaire was imminent? Or did text within the Newsletter explain that a questionnaire was imminent? It is currently unclear the extent to which the newsletter warned of the imminent questionnaire, and whether it tried to encourage participants to complete it and return it. Perhaps the Newsletter might be included in the Supplementary material?

RESULTS 

Meta-analyses:

In the meta-analysis of the Post-it note interventions (figure 4), there is no evidence of heterogeneity among the studies (I-squared=0%) so a fixed effect model, rather than a random effect model, is appropriate. We expect that the 95% confidence interval will be narrower around the estimated odds ratio of 0.97, consistent with the conclusion that the study found no evidence of a benefit of the Post-it notes on increasing response.

In the meta-analysis of prenotification by newsletter interventions (figure 5), there is substantial heterogeneity among the studies (I-squared=92%), so in this case, a random effect model, rather than a fixed effect model is appropriate. We expect that the 95% confidence interval will be wider around the estimated odds ratio of 1.19, consistent with the conclusion that the magnitude of the effect on response of prenotification by newsletter remains uncertain, but that a moderate effect (e.g. OR=1.5) is still plausible. 

CONCLUSIONS 

We were not persuaded that a study with 826 participants is necessarily “a reasonably sized trial”.

A study powered to detect an intervention effect equivalent to an odds ratio of OR=1.5 from a baseline response proportion of 97% would require 3,826 participants in each arm (80% power), or 5,121 participants in each arm (90% power). However, a statistically significant reduction in response with the prenotification newsletter was observed with the study sample of 826 participants, and so this may stand as the study result without any need for the authors to comment on the size of the sample.

In our most recent update to the Cochrane Review (cited in the manuscript), Forty-seven trials (79,651 participants) evaluated the effect on response of contacting participants before sending questionnaires. The odds of response were increased by a half when participants were pre-notified (OR 1.45; 95% CI 1.29 to 1.63). However, there was significant heterogeneity among the trial results (P < 0.00001).  We have recently updated this meta-analysis, for an MSc Epidemiology dissertation (Woolf, B. 2018, unpublished data). In this update, 103 trials were included. Overall, pre-notification increased response 1.38 (95%CI: 1.27-1.49) (pooled result from a random effect model). However, when studies at high or unclear risk of bias were excluded the 95% confidence interval, for the pooled effect of the remaining eight studies, crossed the null. The meta-analysis also found several factors which explained some of the heterogeneity (e.g., the method of pre-notification, using a different method of delivering the pre-notification than the questionnaire, and the risk of bias of the included studies). However, heterogeneity was still present after accounting for these factors.

Given that the method of pre-notification appears to explain study differences, and that this study is only the second to explore the use of Newsletters as a type of pre-notification, it provides important evidence for further understanding this method for potentially reducing questionnaire non-response. 

Is the work clearly and accurately presented and does it cite the current literature? 

Yes, however, there are some minor modifications which would make the paper easier to read. For example, the presentation of the experimental conditions, although presented accurately, was not intuitively easy to grasp. We personally find it easier to understand factorial randomisation (especially when more complex than 2x2) if a matrix or decision tree is provided, for example, the one shown below.

             Condition

          ________|_________

         |                                  |

     Newsletter           no Newsletter

  _________|________   __|_________   

  |                 |         |          |            |         |

hand     printed    no    hand      printed   no    

written  postit    Postit  written  Postit    Postit            

Postit                             Postit

Is the study design appropriate and is the work technically sound?

Yes

Are sufficient details of methods and analysis provided to allow replication by others?

Yes. However, it would be useful if the length of time period for which a survey response would be included in the study was stated explicitly. Prima facie, varying the amount of time participants have for their response to be included in the study could change the results of any potential replication.

If applicable, is the statistical analysis and its interpretation appropriate?

No, we believe that a random effect model, rather than a fixed effect model is appropriate in the meta-analysis of pre-notification by newsletter interventions.

In addition, we would prefer to see more detail about the adjustments which were made for the main analysis. The authors present an odds ratio adjusted for gender. The adjustment was made because of a baseline imbalance after randomisation. However, it is unclear if the decision to make this adjustment was post hoc or part of a decision procedure in a pre-specified protocol. With this in mind, it would be useful if the authors presented 95% confidence intervals for the crude proportions of responses in each conditions to aid integration of these results, as well as the crude odds ratio.

Are all the source data underlying the results available to ensure full reproducibility?

Yes. However, we believe that the study authors should be more explicit about the source and selection methods of studies included in the meta-analyses. 

Are the conclusions drawn adequately supported by the results?

See our comments above

Is the work clearly and accurately presented and does it cite the current literature?

Partly

If applicable, is the statistical analysis and its interpretation appropriate?

No

Are all the source data underlying the results available to ensure full reproducibility?

Partly

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Partly

Are sufficient details of methods and analysis provided to allow replication by others?

Partly

Reviewer Expertise:

Systematic reviews of methods to increase response to postal and electronic questionnaires

We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above.

Rodgers Sara University of York, UK

Competing interests: None declared

 7 2 2019

Points addressed in turn:

Thank you for your comments.  We have addressed each comment in turn below.  Additionally, the paper has been revised with minor amendments to improve clarity and general readability.

Thank you, we have suggested the alternative title of “A study update newsletter or Post-it® note did not increase postal questionnaire response rates in a falls prevention trial: an embedded randomised factorial trial”

Apologies, we did not intend to use the word “pre-notification” to suggest that specific text was included to remind participants of the impending arrival of their 12-month questionnaire. More accurately, we sent a ‘study update’ newsletter to participants, which we have now included in supplementary material. We have removed the use of the term “pre-notification” where necessary and clarified throughout the paper.  

We have now clarified the content of the newsletter and have added this as supplementary material.

Thank you, the text and analyses have been amended as appropriate so that both are a random effects meta-analysis. We have also introduced the fixed effects meta-analysis as a sensitivity analysis given the lack of heterogeneity in the Post-it analysis.

Updated forest plot for newsletter random effect meta-analysis (Figure 5).

Thank you, we acknowledge the reviewer’s point and have removed the text: “in this reasonably sized trial of 826 participants,”.

We have revised the CONSORT diagram according to the reviewer’s suggestion.

All survey responses were included regardless of when they were returned.  Text has been added to the manuscript to clarify this.

The text and analyses have been amended as appropriate so that both are a random effects meta-analysis, and the forest plot for the newsletter meta-analysis (Figure 5) has been updated.

  

The decision to adjust for gender was made prior to analysis and was not based on the chance imbalance observed in gender.  The unadjusted odds ratios have been added to the manuscript, and the 95% CI for the proportions have been added to the CONSORT diagram.

We have clarified that we meta-analysed the similar studies undertaken at York Trials Unit previously.

 10.5256/f1000research.15878.r36069 Reviewer response for version 1 Treweek Shaun P. 1 Referee https://orcid.org/0000-0002-7239-7241 Health Services Research Unit, Institute of Applied Health Sciences, School of Medicine, Medical Sciences and Nutrition, University of Aberdeen, Aberdeen, UK

Competing interests: I am involved in an UK MRC-funded project called PROMETHEUS with one of the authors (David Torgerson) that also focuses on SWATs but was not involved in the work described in this paper.

 3 8 2018 Copyright: © 2018 Treweek SP 2018 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve-with-reservations

This is a well-written article that was great to read. I particularly liked the recommendations in Discussion re: whether it was worth doing more SWAT evaluations. It would be good to see that sort of clarity more often.

I only have a few comments, which are listed below.

Interventions

Would it be possible to see the newsletter, or at least the bit relevant to this study? We have the Post-It text but not the newsletters.

Management of postal questionnaires

I’m guessing that the content of the reminders was the same for all participants regardless of which arm they were allocated to but could you confirm this?

Table 1

The gender imbalance does look odd to me. I know that you say that it is due to chance and that is of course plausible but it differs by up to about 15% across the interventions for women and by up to about 15% for men. These percentages amount to 10 - 20 or so individuals in a total sample size of 135-143. I’m not sure that size of difference would come about just by chance although, of course, it could. Are you sure it’s just chance, or a feature of the randomisation/blocking/something else?  

Table 2

Could you consider giving absolute differences for the primary outcome as well as the OR? ORs are always a bit tricky to interpret. 

Figures 4, 5 and linked text

Two points. I think it would be good to do two GRADE assessments of the evidence included in the two forest plots. This isn’t as hard as it sounds. Depending on the design quality of the included studies (and the current 2018 one is good) my guess is that if the two other studies in Fig 4 are good studies, GRADE is high and for Fig 5 it’s moderate because of inconsistency, though you might pull it down for imprecision too (1.19 with a CI of 0.84 - 1.70 seems pretty wide to me). Regardless, I do think it would be good to say something about the certainty of the body of evidence and then link that to your recommendations in the Discussion.

The second point is that I wasn’t sure why you used a fixed effect model for Fig 5 and a random for Fig 4. My guess is that the random effects model is the one to go for (I’d be surprised if the only differences between studies is random error but that interventions, patients, context etc are at play too). Worth thinking about anyway, especially whether the intention was for the two forest plots to use different models.

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

Partly

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Yes

Are sufficient details of methods and analysis provided to allow replication by others?

Yes

Reviewer Expertise:

Trial methodology

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Rodgers Sara University of York, UK

Competing interests: None declared

 7 2 2019

Thank you for your comments, which we address in turn below.

We have now included the study update newsletter in supplementary material.

This is correct, the reminders were standardised. We have clarified this in the text.

We noticed that there was a mistake in the following sentence: ‘...the presence of women tended to be higher in the groups not receiving the newsletter (65.6% vs 57.7%)’ – the word ‘not’ has been removed.

We can think of no other explanation than chance for the imbalance in gender between the groups. 

These are now included in the manuscript.

Thank you for your suggestion. We have completed the GRADE assessments and included them in the text. We have also included the assessment tables as Supplementary materials.

The text and analyses have been amended as appropriate so that both are a random effects meta-analysis, but we have introduced the fixed effects meta-analysis as a sensitivity analysis for the Post-it® notes given the lack of heterogeneity.