An evaluation of a personalised text message reminder compared to a standard text message on postal questionnaire response rates: an embedded randomised controlled trial

Background: Research outcome data is commonly collected using postal questionnaires; however, poor response can introduce bias and reduce statistical power. Text messaging is simple, cost-effective, and can be customised to the individual. Personalised, reminder text messages may improve response rates. Methods: A two-arm, parallel group ‘Study within a Trial’ (SWAT) was embedded within the Occupational Therapist Intervention Study (OTIS), a randomised controlled trial of a home assessment for falls prevention in older people. OTIS participants who provided a mobile phone number were randomly allocated (1:1) to receive either a personalised text message (Title, Surname, plus York Trials Unit (YTU) text) or the standard YTU text alone, prior to receiving their four-month post-randomisation follow-up postal questionnaire. The primary outcome measure was the proportion of participants who returned the questionnaire. Secondary outcomes were: time to response, completeness of response, requirement of a reminder letter, and cost-effectiveness. Binary data were compared using logistic regression and time to response by Cox proportional hazards regression. Results: A total of 403 participants were randomised: 201 to the personalised text and 202 to the standard text. Of the 283 participants included in the final analysis, 278 (98.2%) returned their questionnaire; 136 (97.8%) for the personalised text versus 142 (98.6%) for the standard text (adjusted odds ratio 0.64, 95% CI 0.10 to 3.88, p=0.63). The median time to response was nine days in both groups. In total, 271 (97.5%) participants returned a complete questionnaire; 133 (97.8%) in the personalised text versus 138 (97.2%) for the standard text. In total, 21 reminder letters were sent. The additional cost of personalised text messages was £0.04 per participant retained. Conclusions: Personalised texts were not superior to standard texts in any outcome assessed in our study. Further SWATs are needed to perform a meta-analysis and obtain more evidence. Registration: ISRCTN22202133; SWAT 35.

outcome assessed in our study. Further SWATs are needed to perform a meta-analysis and obtain more evidence. : ; .

Introduction
Evaluating strategies to improve the efficiency of conducting trials is a priority. Achieving high response rates for postal follow-up questionnaires is challenging; non-response threatens study validity through bias and reduced effective sample size 1 . Rigorous evaluation can be achieved by undertaking a Study within a Trial (SWAT) [2][3][4] . A SWAT is a self-contained study embedded within a host trial, which aims to evaluate an intervention 5 .
There are many strategies towards improving response to postal questionnaires including short messaging service (SMS) text prompts; however, uncertainty remains 6,7 as to their effectiveness [8][9][10][11][12][13] . Furthermore, some evidence exists 14 to suggest that personalised texts, in which recipients were addressed by name, increased average payment of delinquent fines compared to non-personalised texts.
Here, we report the results of a SWAT evaluating a personalised text compared to a standard (non-personalised) text on postal questionnaire response rates in an elderly population.

Design
This two-arm, parallel-group, individually randomised controlled trial (RCT) was embedded within OTIS, a UK-based modified cohort RCT of occupational therapist-led home environmental assessment for the prevention of falls in older people 15

Participants
Eligible OTIS participants who agreed to receive text communication during participation, provided a mobile number, and were due to receive their four-month post-randomisation postal questionnaire, were randomised into this SWAT.

Intervention
Participants received a single text four days after their four-month questionnaire was posted (Table 1).

Outcomes
The primary outcome was the proportion of participants who returned their four-month postal questionnaire. Secondary outcomes were: time to response, completeness of response, use of a reminder letter, and cost-effectiveness (Table 2).

Sample size
As is usual for embedded trials, no formal power calculation was undertaken 3-5 as the sample size was constrained by the number of participants available in the host trial.

Randomisation
Eligible participants (n=403) were randomised (1:1) using randomly varying blocks of four and six, stratified by OTIS trial group allocation. Allocations were generated by the OTIS trial statistician using Stata version 13.0, before being shared with the YTU data management staff responsible for the setup of the text messaging system. Eligible participants were then matched against the generated sequence in the order that they were randomised to the main trial.

Blinding
Participants were not aware of their involvement within this SWAT; only to the OTIS trial group allocation. Study team members performing administrative, statistical or health economic roles were also not blinded, but data entry staff were.

Statistical analysis
Analyses were conducted in Stata version 15.0 16 . Baseline characteristics are summarised descriptively (Table 3). Binary outcomes were analysed using logistic regression, and time to questionnaire return was analysed using Cox proportional hazards regression. Time to return was truncated at 120 days allowing for the next follow-up time point (eight months   (Table 5).

Results
Delays setting-up the text messaging system meant no texts were sent prior to 7 th December 2017. In total 120 (29.8%) randomised participants were due texts before this date. These participants are therefore excluded from the analysis. Participants (n=283) due texts on or after this date were analysed as randomised (Figure 1). The median time to return was nine days in both groups. A log-rank test gave a p-value of 0.57; hence, the data provide little evidence to reject the hypothesis that the two groups have the same survival function. The Cox proportional hazards model corroborated this (hazard ratio 1.06, 95% CI 0.84 to 1.35, p=0.60) (Figure 2). Examination of the log-log plots of the estimated survival functions, and a global test of the Schoenfeld residuals suggested the proportional hazards assumption was reasonable (p=0.52).

Discussion
These results provide little support to the hypothesis that personalisation of texts improves postal questionnaire return rate compared to standard texts, in this population. There was also little evidence to suggest that personalisation led to quicker returns of questionnaires, improved questionnaire completion, or reduced the requirement for a reminder letter to elicit a response. The additional cost of personalised texts was £0.04 per participant retained.

Limitations
Eligible participants who provided a mobile phone number at enrolment to the host trial (78.4%) was lower than antipated. Nearly 30% of SWAT participants had to be excluded from analysis due to problems with text automation. Furthermore, the high proportion of returned postal questionnaires in the standard text group meant only very small improvements could ever be observed or that a ceiling effect may have been reached. Thus, a large sample size would be required in order to provide strong evidence against the null hypothesis in favour of personalisation. Together, the small sample size and high baseline event rate mean this SWAT provides limited evidence for (or against) the personalisation of texts as a means to improving retention of participants.

Conclusions
Given the uncertainty regarding the effectiveness of personalising text messages, we feel that further investigation via RCTs is warranted. Meta-analysis could be used to obtain a more precise estimate for the effectiveness of personalising texts and explore variation across different participant characteristics. This project contains the following underlying data:

Data availability
• OTIS_textswat_data (CSV). Underlying data associated with this study.

1.
Is the work clearly and accurately presented and does it cite the current literature? Partly

Is the study design appropriate and is the work technically sound? Yes
Are sufficient details of methods and analysis provided to allow replication by others? Yes

If applicable, is the statistical analysis and its interpretation appropriate? Yes
Are all the source data underlying the results available to ensure full reproducibility? Yes

Are the conclusions drawn adequately supported by the results? Yes
No competing interests were disclosed. 1.
The work is clearly and accurately presented for the most part, and it does cite the relevant literature as well as the current literature. I have added some comments/corrections below to address.
The study design is appropriate and the work is technically sound. Though I understand the purpose of the study was to evaluate personalised text messages versus non-personalised messages, I would have liked to see a comparison with those that received reminder letters only. Would this be possible with the current data set?
Sufficient details of the methods and analysis are provided for the most part but I've asked for a few minor issues to be addressed in the comments below.
The statistical analysis and its interpretation are appropriate.
Yes, the authors have added information on data availability.
The conclusions drawn are supported by the results but they are different in the abstract to the discussion section of the paper. The authors are definitive in the abstract but are less definitive in the discussion section saying the results provide "little" support. Can we really say little support? I think this should be stronger. There is no evidence to support personalised text messages in this study. I understand there were limitations but these are the findings from this study. I'd ask the authors to address the use of the term "little" in the discussion section.

Further Comments/Corrections:
The reminder letter appears for the first time in the results section and in table 4. This needs to be detailed in the methods section. What justified a person being sent a reminder letter/why were only 21 reminder letters sent? Also, you should add the reminder letter to Figure 1. I expect that their inclusion didn't affect the results given the small numbers and equal proportions in both groups, however, a re-run of the analysis excluding them, and a sentence to say that it didn't affect the results.
Rather than just present the actual costs of each type of text message in the results section under "cost-effectiveness", can you present the findings of the analysis from both? I appreciate it's presented in Table 5 but comment on it.
The outcome definition in Table 2 "proportion of questionnaires returned" should be adjusted as its current meaning does not match your intention. It is not the proportion of questionnaires returned to YTU at four months post-randomisation, it is the proportion of questionnaires returned or the proportion of questionnaires distributed at the four-month randomisation period that were returned or as you have in your main text, the proportion of participants who returned their four-month postal questionnaire. It would be helpful if you specified here how long you gave them to return it. I would add…"within x days".
Paragraph two of the results section, "…compared with 142 in the standardised text group…."replace standardised with standard.
In Table 5, the heading cost of texts is misleading as it implies that more than one text was sent to each person. Change to "cost of text".
Is the work clearly and accurately presented and does it cite the current literature?