Pre-notification and personalisation of text messages to increase questionnaire completion in a smoking cessation pregnancy RCT: an embedded&nbsp;randomised&nbsp;factorial trial

Elizabeth Coleman; Rachel Whitemore; Laura Clark; Karen Daykin; Miranda Clark

doi:10.12688/f1000research.51964.2

Home Browse Pre-notification and personalisation of text messages to increase...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Research Article

Revised

Pre-notification and personalisation of text messages to increase questionnaire completion in a smoking cessation pregnancy RCT: an embedded randomised factorial trial

[version 2; peer review: 2 approved]

Elizabeth Coleman ¹, Rachel Whitemore², Laura Clark¹, Karen Daykin², Miranda Clark²

Elizabeth Coleman ¹, Rachel Whitemore², [...] Laura Clark¹, Karen Daykin², Miranda Clark²

PUBLISHED 30 Sep 2021

Author details Author details

¹ York Trials Unit, Department of Health Sciences, University of York, UK, York, YO10 5DD, UK
² Division of Primary Care, Tower Building, University of Nottingham, Nottingham, NG7 2RD, UK

Elizabeth Coleman
Roles: Data Curation, Formal Analysis, Software, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Rachel Whitemore
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Project Administration, Resources, Writing – Review & Editing

Laura Clark
Roles: Writing – Review & Editing

Karen Daykin
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Project Administration, Resources, Writing – Review & Editing

Miranda Clark
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Project Administration, Resources, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

Abstract

Background:
Low completion rates of questionnaires in randomised controlled trials can compromise the reliability of the results, so ways to boost questionnaire completion are often implemented. Although there is evidence to suggest that sending a text message to participants increases completion, there is little evidence around the timing or personalisation of these text messages.

Methods:
A two-by-two factorial SWAT (study within a trial) was embedded within the MiQuit-3 trial, looking at smoking cessation within pregnant smokers. Participants who reached their 36-week gestational follow-up were randomised to receive a personalised or non-personalised text message, either one week or one day prior to their follow-up. Primary outcomes were completion rate of questionnaire via telephone. Secondary outcomes included: completion rate via any method, time to completion, and number of attempts to contact required.

Results
In total 194 participants were randomised into the SWAT to receive a text message that was personalised early(n=50), personalised late (n=47), non-personalised early(n=50), or non-personalised late(n=47). There was no evidence that timing of the text message (early: one week before; or late: one day before) had an effect on any of the outcomes. There was evidence that a personalised text message would result in fewer completions compared with a non-personalised text message when data was collected only via the telephone(adjusted OR 0.44, 95% CI 0.22–0.87, p=0.02). However, these results were not significant when looking at completion via any method (adjusted OR 0.61, 95% CI 0.30-1.24, p=0.17). There was no evidence to show that personalisation or not was better for any of the secondary outcomes.

Conclusion
Timing of the text message does not appear to influence the completion of questionnaires. Personalisation of a text message may be detrimental to questionnaire completion, if data is only collected via the telephone - however, more SWATs should be undertaken in this field.

Keywords

Randomised Controlled Trial, Embedded Trial, SWAT, Retention, text, notification, personalisation, SMS

Corresponding author: Elizabeth Coleman

Competing interests: No competing interests were disclosed.

Grant information: The MiQuit-3 project was co-funded by the National Institute for Health Research (NIHR) under the Programme Grants for Applied Research programme (RP-PG-0109-10020) and Cancer Research UK (CRUK) (C11232/A23434). The factorial SWAT was funded by the PROMETHEUS MRC programme grant.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2021 Coleman E et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Coleman E, Whitemore R, Clark L et al. Pre-notification and personalisation of text messages to increase questionnaire completion in a smoking cessation pregnancy RCT: an embedded randomised factorial trial [version 2; peer review: 2 approved]. F1000Research 2021, 10:637 (https://doi.org/10.12688/f1000research.51964.2) First published: 22 Jul 2021, 10:637 (https://doi.org/10.12688/f1000research.51964.1) Latest published: 30 Sep 2021, 10:637 (https://doi.org/10.12688/f1000research.51964.2)

Revised Amendments from Version 1

- Throughout the manuscript and the title any reference to 'retention rate' or 'response rate' has been replaced with 'questionnaire completion', for consistency.
- Figure 1 has been updated to ensure consistent terminology.
- Additional text has been included around the various data collection methods, and why telephone was the main interest
- Additional details were included around the money incentives.
- Sentence structure was altered to allow for the manuscript to be grammatically correct.
- The term 'text' was replaced with 'text message' throughout the manuscript.
- The abstract was altered to be clearer that the significant results were only related to telephone collection.
- Wording was altered for clarification around the sample size, and eligibility for the SWAT.

See the authors' detailed response to the review by Frances Shiely
See the authors' detailed response to the review by Phil J. Edwards

Introduction

Randomised controlled trials (RCTs) are the ‘gold standard’ for evaluating healthcare treatments. However, it is well documented that retaining participants can be difficult and low response rates to questionnaires can compromise the reliability and generalisability of the results^1,2. A study within a trial (SWAT) can be used to test interventions to improve retention of participants, via increasing questionnaire completion³.

There is research to support the concept that text messages are effective at improving questionnaire completion rates in trials^4–7. There is insufficient evidence to determine if the timing of text messages improves questionnaire completion rates, and limited papers exploring if personalisation (inclusion of the participants name) impacts questionnaire completion rates^8–11. This factorial SWAT aims to evaluate the effectiveness of the timing and personalisation of text messages within an RCT to add to the evidence base for both of these interventions.

Methods

Design

This two-by-two factorial study was embedded within the MiQuit-3 RCT. MiQuit-3 (ClinicalTrials.gov NCT03231553) is an RCT evaluating the effectiveness of a text-message, smoking cessation self-help support programme for pregnant smokers (MiQuit), and the protocol has been published previously¹². This factorial SWAT was embedded at the 36-week gestational time point. The approval for this factorial SWAT and the MiQuit-3 trial was granted by East Midlands–Nottingham 1 Research Ethics Committee (NRES reference 13/EM/0427 and 17/EM/0327). As the SWAT was considered low risk, informed consent was not obtained from participants, and they were unaware of the SWAT. However, as part of the MiQuit-3 trial all participants consented to their anonymised data being used for further research and being published. The SWATs that form the factorial design are also registered with the Northern Ireland Hub for Trial Methodology Research SWAT Repository (SWATs 35 and 44; both registered December 2015).

Participants and randomisation

As with all SWATs, the sample size is limited by that of the host trial, and a formal power calculation has not been conducted. In total 1002 participants were randomised to the MiQuit-3 trial. As this SWAT was implemented mid-way through follow up for the host trial, all participants that had not yet had their 36-week gestational follow-up, approximately 200, were eligible to participate in the SWAT, and any that had already passed this follow-up time point were unable to be included in this SWAT.

Participants in MiQuit-3 were unaware of their participation in this SWAT, however, they could not be blinded to the contents or timing of the text message. Participants were randomised 1:1:1:1 to each of the four groups (see Table 1). The randomisation was undertaken by a statistician independent of the host trial, and of the staff involved in sending the text messages. Block randomisation was used with varying block sizes of 4, 8, 12 and 16, which was stratified by host trial allocation, and whether they had completed the previous follow-up or not. The randomisation sequence was generated in Stata v.15 (RRID:SCR_012763) and implemented using a remote computer system, independent of the researchers.

Table 1. Details of the SWAT interventions and combinations.

		SWAT 1 – Personalisation
		Intervention 1: Personalised	Control 1: Non-personalised
SWAT 2 – Timing	Intervention 2: Early notification	MiQuit Trial: Hi [name], Thank you for taking part in the MiQuit3 trial. A member of the MiQuit3 team will call next week to complete the final questionnaire. Once completed we will send you a £ 5 or £35 voucher. Whether you have quit smoking or not we would love to speak to you. Thanks, [Researchers name].	MiQuit Trial: Thank you for taking part in the MiQuit3 trial. A member of the MiQuit3 team will call next week to complete the final questionnaire. Once completed we will send you a £ 5 or £35 voucher. Whether you have quit smoking or not we would love to speak to you. Thanks, [Researchers name].
SWAT 2 – Timing	Control 2: Late notification	MiQuit Trial: Hi [name], Thank you for taking part in the MiQuit3 trial. A member of the MiQuit3 team will call tomorrow to complete the final questionnaire. Once completed we will send you a £ 5 or £35 voucher. Whether you have quit smoking or not we would love to speak to you. Thanks, [Researchers name].	MiQuit Trial: Thank you for taking part in the MiQuit3 trial. A member of the MiQuit3 team will call tomorrow to complete the final questionnaire. Once completed we will send you a £ 5 or £35 voucher. Whether you have quit smoking or not we would love to speak to you. Thanks, [Researchers name].

Interventions

This SWAT explored two different interventions; personalisation and timing of text messages (early; one week before follow-up, or late; one day before follow-up). Details of the text message sent to participants can be found in Table one. As detailed in the MiQuit-3 protocol¹², all participants were given a £5 voucher if they completed the 36-week follow-up. Those who provided a saliva sample to validate their smoking status were given an additional £30 voucher. This amount is more than is stated in the published protocol, but this change was made prior to the implementation of this SWAT, and as such all participants involved in this SWAT would have been eligible for this amount. These monetary incentives formed part of the host trial, and where not related to the factorial SWAT being undertaken.

Outcomes

The primary outcome was completion rate; defined as the proportion of the questionnaires completed over the telephone within the follow-up window (14 days).

Secondary outcome measures

The secondary outcome measures included:

Completion rate where the questionnaire was completed by any method (postal, telephone, email/web, or text message) within the follow-up window (14 days)
Time to completion, defined as the number of days between the due date of the 36-week gestation follow-up and the date the questionnaire was recorded as complete
Number of attempts to contact required before the questionnaire was complete, or the maximum number of attempts, six, is reached.

Both time to completion and number of attempts to contact were not restricted by method of data collection, and thus included participants who completed (or were being contacted) via any method.

Statistical analysis

The data were analysed in Stata v.15 (RRID:SCR_012763) on an intention-to-treat (ITT) basis, using two-sided tests at the 2.5% level. As this is a factorial design the Bonferroni correction was applied to allow for multiple testing^13,14. Participants were excluded from the analysis if they had withdrawn prior to the time point.

The primary outcome and completion for all methods were compared using a logistic regression model. Time to completion (days between questionnaire due and complete) was analysed using a Cox Proportional Hazards regression. Participants who completed the questionnaire early had their time set to 0.1, those who did not complete it were censored at either last contact date or 120 days if not contacted, and those who withdrew in the course of the SWAT were set to their withdrawal date. The assumptions for this model were assessed using Schoenfeld residuals¹⁵. The number of attempts to contact was analysed using a negative binomial regression model, due to evidence of overdispersion. All models were adjusted for host trial allocation, whether the participant had completed the previous follow-up, age, and both SWAT intervention allocations separately. All models were repeated with the inclusion of an interaction term to explore any possible interactions between the two SWAT interventions. This was done using two-sided tests at a significance level of 5%.

Stata is proprietary software: a freely available alternative software that could be used to undertake this analysis is RStudio (RRID:SCR_000432)¹⁶.

Results

In total, 194 participants were randomised into the SWAT; 50 received the personalised text message and early notification, 47 received the personalised text message and late notification, 50 received the non-personalised text message and early notification, and 47 received the non-personalised text message and late notification¹⁷. Five participants withdrew prior to the implementation of the SWAT and are not included in the analysis. Participants were only included in a model if all relevant covariates for that model were present. The number included in each of the analysis, by arm, is shown in the flow diagram – Figure 1. Three participants were not contacted due to difficulties/adverse events associated with their pregnancy, but are still included in the analysis under ITT principles. The flow of participants can be seen in Figure 1. Baseline characteristics by SWAT arm and overall, can be found in Table 2.

Figure 1. Flow of participants through the SWAT.

Table 2. Baseline characteristics for participants by SWAT allocation.

	Personalised & Early (n=48)	Personalised & Late (n=44)	Non-personalised & Early (n=50)	Non-personalised & Late (n=47)	Overall (n=189)
Age	N=48	N=44	N=46	N=44	N=182
Mean (SD)	25.4 (5.9)	27.9 (5.9)	27.1 (5.3)	27.2 (6.7)	26.9 (6.0)
Median (min., max.)	24 (17, 41)	27 (17, 41)	26 (16, 39)	28 (17, 41)	26 (16, 41)
Ethnicity: n(%)
Caucasian	43 (89.6)	42 (95.5)	43 (86.0)	40 (85.1)	168 (88.9)
Non-Caucasian	3 (6.3)	1 (2.3)	2 (4.0)	4 (8.5)	10 (5.3)
Missing	2 (4.2)	1 (2.3)	5 (10.0)	3 (6.4)	11 (5.8)
Host trial allocation: n(%)
Intervention	23 (47.9)	19 (43.2)	24 (48.0)	22 (46.8)	88 (46.6)
Usual Care	23 (47.9)	24 (54.6)	22 (44.0)	22 (46.8)	91 (48.2)
Missing	2 (4.2)	1 (2.3)	4 (8.0)	3 (6.4)	10 (5.3)
Completed Previous Follow-up: n(%)
Yes	38 (79.2)	37 (84.1)	36 (72.0)	35 (74.5)	146 (77.3)
No	8 (16.7)	7 (15.9)	10 (20.0)	9 (19.2)	34 (18.0)
Missing	2 (4.2)	0 (0.0)	4 (8.0)	3 (6.4)	9 (4.8)

Primary outcome

The main method of data collection for the MiQuit-3 trial was telephone collection. As such the primary outcome explores the completion rates where the data was collected via telephone calls only. The overall completion rate by telephone was 66.1% (125/189) within the follow-up window (14 days). There were similar completion rates of the questionnaire via telephone within three groups; 50.0% for personalised early (24/48), 52.3% (23/44) for personalised late, and 58.0% (29/50) of non-personalised early, and was slightly higher in the non-personalised late group, 66.0% (31/47).

There was no evidence for a difference in completion rate for the timing of the text message where data was collected via telephone calls; adjusted odds ratio (OR) 0.86 (95% CI 0.44–1.67, p=0.65). There was evidence to suggest a difference in completion rate (adjusted OR 0.44, 95%CI 0.22–0.87, p=0.02) which implies those who received the non-personalised text message were more likely to complete the questionnaire than those who received a personalised text message, when data was collected via the telephone. Full details can be found in Table 3.

Completion rates for all methods. Additional methods of data collection were used alongside telephone calls. For completion via any method the data could have been collected via post, telephone, email/web or text message. When looking at any method, there were similar completion rates of the questionnaire within each of the four groups; 64.6% for personalised early (31/48), 63.6% (28/44) for personalised late, 66.0% for early (33/50) and 70.2% (33/47) of non-personalised.

Table 3. Primary analysis results.

Primary Outcome	Group	Statistic*	95% Confidence Interval	p-value
Completion rate for telephone only	Personalised versus non-personalised	OR = 0.44	0.22 to 0.87	0.02
	Early versus Late	OR = 0.86	0.44 to 1.67	0.65
	Host trial allocation (Intervention versus Control)	OR = 0.63	0.32 to 1.22	0.17
	Completed previous follow-up (Yes versus No)	OR = 9.90	3.87 to 25.35	>0.001
	Age (years)	OR = 1.02	0.96 to 1.07	0.60

* OR = Odds Ratio

Secondary outcomes:

Full details for all secondary outcomes can be found in Table 4.

Table 4. Results for the secondary analyses.

Secondary Outcome	Group	Statistic*	95% Confidence Interval	p-value
Completion rate for all methods	Personalised versus non-personalised	OR = 0.61	0.30 to 1.24	0.17
	Early versus Late	OR = 1.06	0.52 to 2.15	0.87
	Host trial allocation (Intervention versus Control)	OR = 0.79	0.39 to 1.60	0.51
	Completed previous follow-up (Yes versus No)	OR = 8.45	3.60 to 19.86	>0.001
	Age (years)	OR = 1.05	0.99 to 1.11	0.12
Number of attempts to contact required	Personalised versus non-personalised	IRR = 1.14	0.92 to 1.41	0.23
	Early versus Late	IRR = 1.08	0.88 to 1.33	0.45
	Host trial allocation (Intervention versus Control)	IRR = 1.11	0.90 to 1.37	0.33
	Completed previous follow-up (Yes versus No)	IRR = 0.64	0.50 to 0.82	>0.001
	Age (years)	IRR = 1.00	0.98 to 1.02	0.79
Time to completion	Personalised versus non-personalised	HR = 0.76	0.54 to 1.07	0.12
	Early versus Late	HR = 1.00	0.71 to 1.40	0.99
	Host trial allocation (Intervention versus Control)	HR = 0.87	0.62 to 1.21	0.40
	Completed previous follow-up (Yes versus No)	HR = 3.42	1.95 to 5.99	>0.001
	Age (years)	HR = 1.01	0.98 to 1.04	0.51

* OR = Odds Ratio, IRR = Incidence Rate Ratio, HR = Hazards Ratio

There is no evidence to suggest that there is a difference in completion rate for personalised versus non-personalised text messages; adjusted OR 0.61 (95% CI 0.30–1.24, p=0.17). Additionally, there was no evidence to suggest there was a difference in completion rates in participants who received an early or late text message; adjusted OR 1.06 (95% CI 0.52–2.15, p=0.87).

Number of attempts to contact required. The average number of attempts to contact required was 3.0 for all participants, with the average similar for each group (3.3 for both personalised early, 3.2 for personalised late, 3.1 for non-personalised early and 2.7 for non-personalised late). Researchers attempted to contact a participant a maximum of six times. The maximum number of attempts to contact was reached for 55 of the 174 participants (31.3%) and was similar across three groups (38.6% for personalised and early, 31.7% for personalised and late, 31.1% for non-personalised early) and slightly lower in the non-personalised late group, 25%.

There was no evidence of a difference in number attempts to contacts required between those who received an early text message or a late text message (p=0.45). There is also no evidence to suggest a difference between those who received a personalised or non-personalised text message (p=0.23); adjusted incidence rate ratio (IRR) 1.14.

Time to completion. The average time to completion of the questionnaire was 6.2 days (ranging from 5 days early to 103 days late). The time to completion was similar between those who received an early or late personalised text message text message (8.2 days for early versus 7.1 days for late) and was similar for those who received an early or late non-personalised text message (4.9 days for early versus 4.7 days for late). However, there was a slight difference in time to completion between those who received personalised or non-personalised text message.

There was no evidence of a difference in time to completion between those who received the text message early or late (p=0.99) or those who received a personalised or non-personalised text message (p=0.12). This suggest that neither timing nor personalisation of the text message reminder affect the time taken to complete the questionnaire. The assumptions for the model held when examined using Schoenfeld residuals (p=0.66).

Interaction terms. All of the models were re-run with the inclusion of any interaction term between the two SWAT allocations. There was no evidence of an interaction for the completion rate, both by phone only (p=0.57) and all methods (p=0.54). There was also no evidence of an interaction for the number of contacts required (p=0.69), or the time to completion (p=0.88).

Comparison with the whole RCT. There were 1002 participants who were randomised into the MiQuit-3 trial. Of the 777 who were not included in the SWAT, and were due a 36-week follow-up, 499 completed the questionnaire (64.2%). This is similar to the completion rate for the participants in the SWAT (overall 66.1%).

Discussion

This factorial SWAT showed that the timing of the text message reminder had no effect on the questionnaire completion rate, the time to complete, or the number of attempts to contact required; these results mirror what Partha et al. reported in their work⁸. It also showed that personalised text messages have no effect on completion time, or number of attempts to contact required. However, it did show that there was some evidence that sending a non-personalised text message reminder would have a larger increase in response than sending personalised text messages did, but these finding were only significant when exploring telephone data collection. Cochrane et al. found no statistically significant difference in their study, but results favoured the non-personalised text messages¹¹. As our work was conducted in a female-only population, who were between 17 and 41 years of age, the results here are only directly related to this population. Equally, as the SWAT was not powered to detect a difference, more SWATs should be undertaken in this area to allow the results to be combined in a pooled analysis to determine the true effect of the interventions and consider the effects on a wider population.

Data availability

Underlying data

Figshare: Underlying data for ‘Pre-notification and personalisation of text-messages to retain participants in a smoking cessation pregnancy RCT: an embedded randomised factorial trial’. https://doi.org/10.6084/m9.figshare.14224319.v1¹⁷

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Reporting guidelines

Figshare: CONSORT checklist for ‘Pre-notification and personalisation of text-messages to retain participants in a smoking cessation pregnancy RCT: an embedded randomised factorial trial’. https://doi.org/10.6084/m9.figshare.14229647.v1¹⁸

Acknowledgements

The authors would like to acknowledge Professor Tim Coleman, the Chief Investigator for the MiQuit-3 trial, and members of the MiQuit-3 trial involved in implementing the SWAT into the trial.

Faculty Opinions recommended

References

1. Bower P, Brueton V, Gamble C, et al.: Interventions to improve recruitment and retention in clinical trials: a survey and workshop to assess current practice and future priorities. Trials. 2014; 15(1)1: 399. PubMed Abstract | Publisher Full Text | Free Full Text
2. Adamson J, Hewitt CE, Torgerson DJ: Producing better evidence on how to improve randomised controlled trials. BMJ. 2015; 351: h4923. PubMed Abstract | Publisher Full Text
3. Treweek S, Bevan S, Bower P, et al.: Trial Forge Guidance 1: what is a Study Within A Trial (SWAT)? Trials. 2018; 19(1): 139. PubMed Abstract | Publisher Full Text | Free Full Text
4. Man MS, Tilbrook HE, Jayakody S, et al.: Electronic reminders did not improve postal questionnaire response rates or response times: a randomized controlled trial. J Clin Epidemiol. 2011; 64(9): 1001–1004. PubMed Abstract | Publisher Full Text
5. Ashby R, Turner G, Cross B, et al.: A randomized trial of electronic reminders showed a reduction in the time to respond to postal questionnaires. J Clin Epidemiol. 2011; 64(2): 208–212. PubMed Abstract | Publisher Full Text
6. Clark L, Ronaldson S, Dyson L, et al.: Electronic prompts significantly increase response rates to postal questionnaires: a randomized trial within a randomized trial and meta-analysis. J Clin Epidemiol. 2015; 68(12): 1446–1450. PubMed Abstract | Publisher Full Text
7. Naughton F, Riaz M, Sutton S: Response Parameters for SMS Text Message Assessments Among Pregnant and General Smokers Participating in SMS Cessation Trials. Nicotine Tob Res. 2016; 18(5): 1210–1214. PubMed Abstract | Publisher Full Text | Free Full Text
8. Sarathy PP, Kottam L, Parker A, et al.: Timing of electronic reminders did not improve trial participant questionnaire response: a randomized trial and meta-analyses. J Clin Epidemiol. 2020; 122: 70–77. PubMed Abstract | Publisher Full Text
9. Keding A, Brabyn S, MacPherson H, et al.: Text message reminders to improve questionnaire response rates in RCTs: findings from three randomised sub-studies. Trials. 2015; 16(Suppl. 2): 103. PubMed Abstract | Publisher Full Text
10. Mitchell AS, Cook L, Dean A, et al.: An embedded randomised controlled retention trial of personalised text messages compared to non-personalised text messages in an orthopaedic setting [version 1; peer review: 1 approved]. F1000Res. 2020; 9: 591. Publisher Full Text
11. Cochrane A, Welch C, Fairhurst C, et al.: An evaluation of a personalised text message reminder compared to a standard text message on postal questionnaire response rates: an embedded randomised controlled trial [version 1; peer review: 2 approved]. F1000Res. 2020; 9: 154. PubMed Abstract | Publisher Full Text | Free Full Text
12. Whitemore R, Leonardi-Bee J, Naughton F, et al.: Effectiveness and cost-effectiveness of a tailored text-message programme (MiQuit) for smoking cessation in pregnancy: study protocol for a randomised controlled trial (RCT) and meta-analysis. Trials. 2019; 20(1): 280. PubMed Abstract | Publisher Full Text | Free Full Text
13. StataCorp: Stata Statistical Software: Release 15. College Station, TX: StataCorp LLC, 2017. Reference Source
14. Bland JM, Altman DG: Multiple significance tests: the Bonferroni method. BMJ. 1995; 310(6973): 170. PubMed Abstract | Publisher Full Text | Free Full Text
15. Schoenfeld D: Partial residuals for the proportional hazards regression model. Biometrika. 1982; 69(1): 239–241. Publisher Full Text
16. RStudio Team: RStudio: Integrated Development for R. RStudio, PBC, Boston, MA, 2020. Reference Source
17. Coleman E: MiQuit_SWAT_Data.csv. figshare. Dataset. 2021. http://www.doi.org/10.6084/m9.figshare.14224319.v1
18. Coleman E: MiQuit SWAT CONSORT Checklist. figshare. Journal contribution. 2021. http://www.doi.org/10.6084/m9.figshare.14229647.v1

Comments on this article Comments (0)

Version 2

VERSION 2 PUBLISHED 22 Jul 2021

Author details Author details

¹ York Trials Unit, Department of Health Sciences, University of York, UK, York, YO10 5DD, UK
² Division of Primary Care, Tower Building, University of Nottingham, Nottingham, NG7 2RD, UK

Elizabeth Coleman
Roles: Data Curation, Formal Analysis, Software, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Rachel Whitemore
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Project Administration, Resources, Writing – Review & Editing

Laura Clark
Roles: Writing – Review & Editing

Karen Daykin
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Project Administration, Resources, Writing – Review & Editing

Miranda Clark
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Project Administration, Resources, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

The MiQuit-3 project was co-funded by the National Institute for Health Research (NIHR) under the Programme Grants for Applied Research programme (RP-PG-0109-10020) and Cancer Research UK (CRUK) (C11232/A23434). The factorial SWAT was funded by the PROMETHEUS MRC programme grant.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (2)

version 2

Revised

Published: 30 Sep 2021, 10:637

https://doi.org/10.12688/f1000research.51964.2

version 1

Published: 22 Jul 2021, 10:637

https://doi.org/10.12688/f1000research.51964.1

© 2021 Coleman E et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Coleman E, Whitemore R, Clark L et al. Pre-notification and personalisation of text messages to increase questionnaire completion in a smoking cessation pregnancy RCT: an embedded randomised factorial trial [version 2; peer review: 2 approved]. F1000Research 2021, 10:637 (https://doi.org/10.12688/f1000research.51964.2)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Version 2

VERSION 2

PUBLISHED 30 Sep 2021

Revised

Views

Reviewer Report 04 Oct 2021

Frances Shiely, Trials Research and Methodologies Unit, HRB Clinical Research Facility, Mercy University Hospital, University College Cork, Cork, Ireland; School of Public Health, University College cork, Cork, Ireland

Approved

https://doi.org/10.5256/f1000research.77567.r95844

Thank you to authors for making the suggested changes. I believe these amendments ... Continue reading

CITE

Report a concern

Respond or Comment

Version 1

VERSION 1

PUBLISHED 22 Jul 2021

Views

Reviewer Report 04 Aug 2021

Approved with Reservations

https://doi.org/10.5256/f1000research.55180.r90184

Summary
This was a SWAT within the MiQuit-3 trial. The purpose of the SWAT was to establish if the timing or personalisation of text messages increases completion of a questionnaire. Study design was a 2 x 2 factorial SWAT. Participants who reached their 36-week gestational follow-up were randomised to receive a personalised or non-personalised text message, either one week or one day prior to the telephone follow-up. Primary outcome was completion rate of questionnaire via telephone. Secondary outcomes included: completion rate via any method, time to completion, and number of reminders required. The authors concluded that timing of the text message did not appear to influence the retention of participants. The authors concluded that personalisation of a text message may be detrimental to retention; however, more SWATs should be undertaken in this field.

Major Comments
My first comment is on the title, the purpose of the study and the conclusions drawn. The title, correctly refers to the retention of participants in the smoking cessation pregnancy RCT. The purpose of the SWAT though is to evaluate completion of a questionnaire, not retention in the host trial, as claimed in the conclusion in the abstract. I think this is conflated throughout and the authors need to consider this carefully and amend their paper. In fact, the registered SWATs in the NI SWAT repository give the outcomes in both as questionnaire completion. So therefore the conclusions drawn in the abstract and in the discussion are not supported by the data.

Abstract

"There was evidence that a personalised text would result in fewer completions via telephone compared with a non-personalised text (adjusted OR 0.44, 95% CI 0.22–0.87, p=0.02)". This statement is confusing. The research question is personalised text message versus non-personalised text message. If using via telephone, then it should be also say non-personalised text via telephone.
Also, use text message, rather than text, throughout.
"Personalisation of a text message may be detrimental to retention". I don't think your results support a statement this strong. Firstly, this is not a trial with an adequate sample size to make this claim. Secondly, when you included all methods of receiving the questionnaire (I think this is correct interpretation but it is challenging to establish in the current version of the paper as the detail on all methods is unclear) you did not find that personalised or non-personalised texts mattered. At best you can say, personalisation of a text message appears to affect questionnaire completion via telephone.

Introduction

I like the succinct introduction.

Methods

The methods says "The SWATs are also registered with the Northern Ireland Hub for Trial Methodology Research SWAT Repository (SWATs 35 and 44; both registered December 2015). It's a little confusing to the reader. It looks like you're conducting two SWATs. I understand what you have taken both of these SWATS and conducted the two in a 2 x 2 factorial design. However, make this clear to the reader.
Replace the phrase "carried out" with conducted throughout. I know the phrase carried out is used all the time in literature but it is not correct unless you are describing physically carrying an object.
I would use the word women rather than participants, given that it is a trial of pregnant women.
The phrase "all participants that had not yet had their 36-week gestational follow-up were eligible to participate" suggests that some people were more than 36 weeks pregnant. Is this the case? Otherwise you could simplify it and say women who were 36 weeks pregnant. It's important to be clear on the timing of the intervention given that this is one of your outcomes.
"Participants in MiQuit-3 were blind to their participation in this SWAT". Blinding is different to being unaware. Were they both blinded and unaware?
"The randomisation was undertaken by a statistician independent of the host trial, and of the staff involved in sending the texts". Explain how the statistician did the randomisation, e.g. computer generated. Also, explain how he communicated that randomisation to the researcher assigning the women to each group.
"Block randomisation, stratified by host trial allocation, and whether they had completed the previous follow-up; with varying block sizes of 4, 8, 12 and 16". This is not a sentence.
"A £5 voucher was given to all participants who completed a follow-up…". Was this part of the host trial or the SWAT?
"…additionally those who provided a saliva sample were given another £30 (£35 total)." Where does the saliva sample come into it? Is this part of the host trial? Explain in the paper.
How did you decide on how many to include for the SWAT. I accept that a sample size calculation is not required but a line in the paper on why you decided on (it appears to be 200) would be useful.
In the secondary outcome, explain what you mean by completed by any method.
"Time to response, defined as the number of days between the due date of the 36-week gestation follow-up and the date the questionnaire was recorded as complete". Are you certain all follow-up calls were made in 24 hours?

Statistical Analysis

"The primary outcome was completion rate; defined as the proportion of the questionnaires completed over the telephone within the follow-up window (14 days)". However, you then go on to say that you used logistic regression to analyse this. Logistic regression is not suitable for four categories. I suspect what you mean is that you compared the completion rates across the two personalised/not personalised and again early/late. I can see you did this from the table. However, you need to articulate that in the text because it is currently confusing.
In statistical analysis, a full stop after level and a new sentence for "As this is a factorial..."
Full stop required in this sentence too. Also, I suspect the word "compared" in this sentence should read completed. - "Time to response (days between questionnaire due and complete) was analysed using a Cox Proportional Hazards regression, those who compared the questionnaire early had their time set to 0.1, those did not complete were censored at either last contact date or 120 days if not contacted, and those who withdrew in the course of the SWAT were set to their withdrawal date".
Again the following is not a sentence, "All models were repeated with the inclusion of an interaction term to explore any possible interactions between the two SWAT interventions; with a significance level of 5%."

Results

"Additional participants were excluded from the analysis, where the covariates required for the model were not provided". What additional participants? Quantify and explain please. Why were the covariates not 'provided'? Explain please.
"Three participants were not contacted due to difficulties/adverse events associated with their pregnancy but are still included in the analysis under ITT principles". Commas are required to make sense of the sentence.
In your flow chart, what does primary refer to, and proximity? Add an explanation or use a term that explains a little better. Also in the flow chart, you say response rate but provide the number of participants. This is not a rate.
In the primary outcome, continue with your phraseology - the 14-day follow-up window rather than "within 14 days of the due date".
I find the writing of the results very confusing. This sentence below suggests the outcome was completion rate via telephone versus completion rate via something else. "There was evidence to suggest a difference in completion rate via telephone adjusted OR 0.44 (0.22–0.87, p=0.02) which implies those who received the non-personalised text were more likely to complete the questionnaire when completing via the telephone". I'm wondering why you keep saying via telephone. It is particularly confusing when explaining the results. The last part above again says … those who received the non-personalised text were more likely to complete the questionnaire when completing via the telephone". It looks like the method of completion is the purpose of the study.
When you use the phrase "were more likely to", you must give the details of the comparison, i.e., more likely than who?
It is implied, but not adequately explained, that some women completed the questionnaire by some other means. It is not clear how this was handled in terms of the numbers analysed throughout the study and this needs to be explained.
For your tables 3 and 4, add the number of women. Why does the left column say "response rate for all methods" when the primary outcome is defined as the proportion of the questionnaires completed over the telephone…"
The heading in the results section, "Response rates for all method", do you mean completion rates for all methods? A response rate is different.
If you hang your hat on statistically significant evidence, by quoting CIs and p-values, to establish if your SWAT was effective, or not, then the following has no place in your paper. "There is some, non statistically significant, evidence to suggest that there may be a difference in response rate for personalised versus non-personalised text reminders; adjusted OR 0.61 (95% CI 0.30–1.24, p=0.17), in favour of the non-personalised text messages". You cannot say there is non-statistically significant evidence and then support that statement with statistics! Remove this please.
The heading, "Number of attempts to contact required". Replace with, Number of attempts required to contact the women.
What do you mean by the maximum number of calls, as stated here "The maximum number of calls was reached for 55 of the 174 participants…"?
Contacts required is a new term introduced here " There was no evidence of a difference in number of contacts required". What do you mean by it?
"The average time to respond was 6.2 days (ranging from 5 days early to 103 days late)". Respond to what, the phone call or the text, or the questionnaire?
"This was similar between those who received a personalised text (8.2 days for early versus 7.1 days for late) and those who received the non-personalised text (4.9 days for early versus 4.7 days for late), but there is a slight difference between those who received personalised or non-personalised texts". If it is similar, how can there be a slight difference? What point are you making here?
Include the number of participants in the MiQuit Trial earlier in the paper when discussing the 200 randomised for the SWAT.

Discussion

"It did show that there was some evidence that sending a non-personalised text message reminder would have a larger increase in response than sending personalised text messages did". This is misleading because it was not the case when all methods of questionnaire were included. Please amend the statement.
The final sentence of the discussion needs to be reviewed. What do you mean by overall effectiveness?

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

No

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Epidemiology; Trial Methodology; SWATs

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Author Response 30 Sep 2021

Elizabeth Coleman, York Trials Unit, Department of Health Sciences, University of York, UK, York, YO10 5DD, UK

30 Sep 2021

Author Response

The authors would like to thank you for a very thorough review of this article - please see responds to your comments and suggestions below. We have amended the article ... Continue reading The authors would like to thank you for a very thorough review of this article - please see responds to your comments and suggestions below. We have amended the article to incorporate your corrections and suggestions were we pleased it was suitable to do so.

Summary
This was a SWAT within the MiQuit-3 trial. The purpose of the SWAT was to establish if the timing or personalisation of text messages increases completion of a questionnaire. Study design was a 2 x 2 factorial SWAT. Participants who reached their 36-week gestational follow-up were randomised to receive a personalised or non-personalised text message, either one week or one day prior to the telephone follow-up. Primary outcome was completion rate of questionnaire via telephone. Secondary outcomes included: completion rate via any method, time to completion, and number of reminders required. The authors concluded that timing of the text message did not appear to influence the retention of participants. The authors concluded that personalisation of a text message may be detrimental to retention; however, more SWATs should be undertaken in this field.
Thank you for this well written summary of the paper.

Major Comments
My first comment is on the title, the purpose of the study and the conclusions drawn. The title, correctly refers to the retention of participants in the smoking cessation pregnancy RCT. The purpose of the SWAT though is to evaluate completion of a questionnaire, not retention in the host trial, as claimed in the conclusion in the abstract. I think this is conflated throughout and the authors need to consider this carefully and amend their paper. In fact, the registered SWATs in the NI SWAT repository give the outcomes in both as questionnaire completion. So therefore the conclusions drawn in the abstract and in the discussion are not supported by the data.
The authors have amended the title of the paper, and changed references to retention within the paper to completion of questionnaires throughout. Alongside making the other suggested changes by the reviewer, we hope to have addressed any concerns in relation to the conclusions drawn.

Abstract
"There was evidence that a personalised text would result in fewer completions via telephone compared with a non-personalised text (adjusted OR 0.44, 95% CI 0.22–0.87, p=0.02)". This statement is confusing. The research question is personalised text message versus non-personalised text message. If using via telephone, then it should be also say non-personalised text via telephone.
The wording has been amended to attempt to make this statement clearer, and also reference the results for any data collection method.

Also, use text message, rather than text, throughout.
The term ‘text’ has been changed to ‘text message’ throughout the paper.

"Personalisation of a text message may be detrimental to retention". I don't think your results support a statement this strong. Firstly, this is not a trial with an adequate sample size to make this claim. Secondly, when you included all methods of receiving the questionnaire (I think this is correct interpretation but it is challenging to establish in the current version of the paper as the detail on all methods is unclear) you did not find that personalised or non-personalised texts mattered. At best you can say, personalisation of a text message appears to affect questionnaire completion via telephone.
This sentence has been altered to include ‘data was collected only via the telephone’, and now also references that the results do not hold for data collection via all methods. The results are correct as finding of this work, and the conclusion of the abstract references that more work is needed in this area so no other alterations have been made in regards to this comment.

Introduction
I like the succinct introduction.
Thank you for this comment.

Methods
The methods says "The SWATs are also registered with the Northern Ireland Hub for Trial Methodology Research SWAT Repository (SWATs 35 and 44; both registered December 2015). It's a little confusing to the reader. It looks like you're conducting two SWATs. I understand what you have taken both of these SWATS and conducted the two in a 2 x 2 factorial design. However, make this clear to the reader.
Additional words have been added to this paragraph to make it clear that this was a factorial SWAT involving these two SWATs.

Replace the phrase "carried out" with conducted throughout. I know the phrase carried out is used all the time in literature but it is not correct unless you are describing physically carrying an object.
The phase carried out has been replaced with conducted.

I would use the word women rather than participants, given that it is a trial of pregnant women.
We thank you for the suggestion – however, the authors have decided not to make this change. The protocol for the MiQuit-3 trial refers to those involved both as participants, and as women. The term participants is not incorrect, and as it is made clear that this work is being done in pregnant women in both the introduction and conclusions of this paper, we feel it does not need to be re-emphasised throughout the whole paper.

The phrase "all participants that had not yet had their 36-week gestational follow-up were eligible to participate" suggests that some people were more than 36 weeks pregnant. Is this the case? Otherwise you could simplify it and say women who were 36 weeks pregnant. It's important to be clear on the timing of the intervention given that this is one of your outcomes.
This sentence references those who were eligible to be included in the SWAT. Some participants in the host trial would have been more than 36 weeks pregnant (or have had their baby/babies) when the SWAT was implement, and thus they were not eligible to be included in the SWAT. The wording of this paragraph has been altered in relation to this comment, and the reviewer comment regarding the number of participants involved in the SWAT, hopefully this has made it clearer.

"Participants in MiQuit-3 were blind to their participation in this SWAT". Blinding is different to being unaware. Were they both blinded and unaware?
A correction has been made in regards to the unawareness and blinding of participants.

"The randomisation was undertaken by a statistician independent of the host trial, and of the staff involved in sending the texts". Explain how the statistician did the randomisation, e.g. computer generated. Also, explain how he communicated that randomisation to the researcher assigning the women to each group.
Details have been added to explain how the randomisation was generated (Stata) and how the randomisation sequence was used.

"Block randomisation, stratified by host trial allocation, and whether they had completed the previous follow-up; with varying block sizes of 4, 8, 12 and 16". This is not a sentence.
Corrections have been made to this sentence.

"A £5 voucher was given to all participants who completed a follow-up…". Was this part of the host trial or the SWAT?
Detail has been added to clarify that this is part of the host trial

"…additionally those who provided a saliva sample were given another £30 (£35 total)." Where does the saliva sample come into it? Is this part of the host trial? Explain in the paper.
Detail has been added to explain why a saliva sample was given, and that it, and associated voucher were part of the host trial.

How did you decide on how many to include for the SWAT. I accept that a sample size calculation is not required but a line in the paper on why you decided on (it appears to be 200) would be useful.
Further details have been added to explain why only 200 were included.

In the secondary outcome, explain what you mean by completed by any method.
Details have been added to explain what other methods of completion could be used.

"Time to response, defined as the number of days between the due date of the 36-week gestation follow-up and the date the questionnaire was recorded as complete". Are you certain all follow-up calls were made in 24 hours?
The date of each attempt to contact was recorded. However this outcome captures completion by any follow-up method.

Statistical Analysis
"The primary outcome was completion rate; defined as the proportion of the questionnaires completed over the telephone within the follow-up window (14 days)". However, you then go on to say that you used logistic regression to analyse this. Logistic regression is not suitable for four categories. I suspect what you mean is that you compared the completion rates across the two personalised/not personalised and again early/late. I can see you did this from the table. However, you need to articulate that in the text because it is currently confusing.
A logistic model can only deal with two levels of an outcome; it can handle multiple levels of a covariate. The outcome was completion (yes/no), and the swat allocations are included as covariates in the model. Each swat allocation is include as a separate covariate – as stated in the analysis section, providing two, two-level covariates. Thus the model does not include a four level variable, as you have suggested. Had swat allocation been included as a four level covariate, the model would have been able to do this – but it would not have provided an OR for each of the two SWAT interventions separately. This method is the appropriate method for analysing a binary outcome for a factorial SWAT, as such no changes have been made to the text.

In statistical analysis, a full stop after level and a new sentence for "As this is a factorial..."
Thank you for suggesting this correction – it has been made.

Full stop required in this sentence too. Also, I suspect the word "compared" in this sentence should read completed. - "Time to response (days between questionnaire due and complete) was analysed using a Cox Proportional Hazards regression, those who compared the questionnaire early had their time set to 0.1, those did not complete were censored at either last contact date or 120 days if not contacted, and those who withdrew in the course of the SWAT were set to their withdrawal date".
Corrections to this sentence have been made.

Again the following is not a sentence, "All models were repeated with the inclusion of an interaction term to explore any possible interactions between the two SWAT interventions; with a significance level of 5%."
Additional words have been added.

Results
"Additional participants were excluded from the analysis, where the covariates required for the model were not provided". What additional participants? Quantify and explain please. Why were the covariates not 'provided'? Explain please.
The text in this sentence has been altered to make this clearer.

"Three participants were not contacted due to difficulties/adverse events associated with their pregnancy but are still included in the analysis under ITT principles". Commas are required to make sense of the sentence.
Changes have been made to this sentence.

In your flow chart, what does primary refer to, and proximity? Add an explanation or use a term that explains a little better. Also in the flow chart, you say response rate but provide the number of participants. This is not a rate.
The flow diagram has been updated to include consistent terminology. The numbers in the flow diagram are referencing the numbers within each analysis.

In the primary outcome, continue with your phraseology - the 14-day follow-up window rather than "within 14 days of the due date".
This term has been made consistent throughout the paper.

I find the writing of the results very confusing. This sentence below suggests the outcome was completion rate via telephone versus completion rate via something else. "There was evidence to suggest a difference in completion rate via telephone adjusted OR 0.44 (0.22–0.87, p=0.02) which implies those who received the non-personalised text were more likely to complete the questionnaire when completing via the telephone". I'm wondering why you keep saying via telephone. It is particularly confusing when explaining the results. The last part above again says … those who received the non-personalised text were more likely to complete the questionnaire when completing via the telephone". It looks like the method of completion is the purpose of the study.
The main data collection method for MiQuit-3 was via the telephone. As such, within the SWAT protocol for this factorial SWAT, it was decided that the primary outcome would focus specifically on data collected via this method, and as a secondary explore data collected via any method. The results for the primary and associated secondary outcome have been rewritten slightly to clarify that the results are only including those who completed it via the telephone, as opposed to comparing those who did it via the telephone with something else.

When you use the phrase "were more likely to", you must give the details of the comparison, i.e., more likely than who?
Additional words have been added for clarification.

It is implied, but not adequately explained, that some women completed the questionnaire by some other means. It is not clear how this was handled in terms of the numbers analysed throughout the study and this needs to be explained.
For the secondary outcomes (time to completion, and number of attempts to contact) there have been no additional adjustments or changes to the analysis due to the method of data collection. The completion method is only relevant to the primary outcome, where the trial team detailed that their main method of data collection was via the telephone, and thus the primary outcome explores this. However, to explore the effectiveness of the interventions on any data collection method (as is more typically done in RCTs where a number of methods are utilised) a secondary outcome was to explore this via any method.

For your tables 3 and 4, add the number of women. Why does the left column say "response rate for all methods" when the primary outcome is defined as the proportion of the questionnaires completed over the telephone…"
Response rate, completion rate and retention rate are all frequently used synonymously within the SWAT world. As such the terminology within this paper has switched between the terms. In response to this comment the terminology has been made consistent.

The heading in the results section, "Response rates for all method", do you mean completion rates for all methods? A response rate is different.
Response rate has been changed to completion rate throughout.

If you hang your hat on statistically significant evidence, by quoting CIs and p-values, to establish if your SWAT was effective, or not, then the following has no place in your paper. "There is some, non statistically significant, evidence to suggest that there may be a difference in response rate for personalised versus non-personalised text reminders; adjusted OR 0.61 (95% CI 0.30–1.24, p=0.17), in favour of the non-personalised text messages". You cannot say there is non-statistically significant evidence and then support that statement with statistics! Remove this please.
This section has been reworded to reflect your suggestion.

The heading, "Number of attempts to contact required". Replace with, Number of attempts required to contact the women.
The authors have declined to make this change. In some instances the participant was not contacted, therefore it would be incorrect for the heading to referring the number required to contact them, when some were not contactable. Additionally, as explained in a previous comment we will not be changing participant to women.

What do you mean by the maximum number of calls, as stated here "The maximum number of calls was reached for 55 of the 174 participants…"?
This has been clarified both when describing the outcome, and in the results. The researcher tried to contact the participant a maximum of 6 times.

Contacts required is a new term introduced here " There was no evidence of a difference in number of contacts required". What do you mean by it?
This is referring the number of attempts to contact – a word had been omitted, and has been corrected.

"The average time to respond was 6.2 days (ranging from 5 days early to 103 days late)". Respond to what, the phone call or the text, or the questionnaire?
Terminology has been changed.

"This was similar between those who received a personalised text (8.2 days for early versus 7.1 days for late) and those who received the non-personalised text (4.9 days for early versus 4.7 days for late), but there is a slight difference between those who received personalised or non-personalised texts". If it is similar, how can there be a slight difference? What point are you making here?
The wording of the sentence has been changed to clarify this point. The sentence was meant to emphasis the difference in time between the personalised/non-personalised, whilst highlighting the similarities between early/late.

Include the number of participants in the MiQuit Trial earlier in the paper when discussing the 200 randomised for the SWAT.
The authors have included this figure (n=1002) earlier on in the paper.

Discussion
"It did show that there was some evidence that sending a non-personalised text message reminder would have a larger increase in response than sending personalised text messages did". This is misleading because it was not the case when all methods of questionnaire were included. Please amend the statement.
Additional words have been added to explain that these results are only valid for telephone completion

The final sentence of the discussion needs to be reviewed. What do you mean by overall effectiveness?
These words have been removed.
The authors would like to thank you for a very thorough review of this article - please see responds to your comments and suggestions below. We have amended the article to incorporate your corrections and suggestions were we pleased it was suitable to do so.

Summary
This was a SWAT within the MiQuit-3 trial. The purpose of the SWAT was to establish if the timing or personalisation of text messages increases completion of a questionnaire. Study design was a 2 x 2 factorial SWAT. Participants who reached their 36-week gestational follow-up were randomised to receive a personalised or non-personalised text message, either one week or one day prior to the telephone follow-up. Primary outcome was completion rate of questionnaire via telephone. Secondary outcomes included: completion rate via any method, time to completion, and number of reminders required. The authors concluded that timing of the text message did not appear to influence the retention of participants. The authors concluded that personalisation of a text message may be detrimental to retention; however, more SWATs should be undertaken in this field.
Thank you for this well written summary of the paper.

Major Comments
My first comment is on the title, the purpose of the study and the conclusions drawn. The title, correctly refers to the retention of participants in the smoking cessation pregnancy RCT. The purpose of the SWAT though is to evaluate completion of a questionnaire, not retention in the host trial, as claimed in the conclusion in the abstract. I think this is conflated throughout and the authors need to consider this carefully and amend their paper. In fact, the registered SWATs in the NI SWAT repository give the outcomes in both as questionnaire completion. So therefore the conclusions drawn in the abstract and in the discussion are not supported by the data.
The authors have amended the title of the paper, and changed references to retention within the paper to completion of questionnaires throughout. Alongside making the other suggested changes by the reviewer, we hope to have addressed any concerns in relation to the conclusions drawn.

Abstract
"There was evidence that a personalised text would result in fewer completions via telephone compared with a non-personalised text (adjusted OR 0.44, 95% CI 0.22–0.87, p=0.02)". This statement is confusing. The research question is personalised text message versus non-personalised text message. If using via telephone, then it should be also say non-personalised text via telephone.
The wording has been amended to attempt to make this statement clearer, and also reference the results for any data collection method.

Also, use text message, rather than text, throughout.
The term ‘text’ has been changed to ‘text message’ throughout the paper.

"Personalisation of a text message may be detrimental to retention". I don't think your results support a statement this strong. Firstly, this is not a trial with an adequate sample size to make this claim. Secondly, when you included all methods of receiving the questionnaire (I think this is correct interpretation but it is challenging to establish in the current version of the paper as the detail on all methods is unclear) you did not find that personalised or non-personalised texts mattered. At best you can say, personalisation of a text message appears to affect questionnaire completion via telephone.
This sentence has been altered to include ‘data was collected only via the telephone’, and now also references that the results do not hold for data collection via all methods. The results are correct as finding of this work, and the conclusion of the abstract references that more work is needed in this area so no other alterations have been made in regards to this comment.

Introduction
I like the succinct introduction.
Thank you for this comment.

Methods
The methods says "The SWATs are also registered with the Northern Ireland Hub for Trial Methodology Research SWAT Repository (SWATs 35 and 44; both registered December 2015). It's a little confusing to the reader. It looks like you're conducting two SWATs. I understand what you have taken both of these SWATS and conducted the two in a 2 x 2 factorial design. However, make this clear to the reader.
Additional words have been added to this paragraph to make it clear that this was a factorial SWAT involving these two SWATs.

Replace the phrase "carried out" with conducted throughout. I know the phrase carried out is used all the time in literature but it is not correct unless you are describing physically carrying an object.
The phase carried out has been replaced with conducted.

I would use the word women rather than participants, given that it is a trial of pregnant women.
We thank you for the suggestion – however, the authors have decided not to make this change. The protocol for the MiQuit-3 trial refers to those involved both as participants, and as women. The term participants is not incorrect, and as it is made clear that this work is being done in pregnant women in both the introduction and conclusions of this paper, we feel it does not need to be re-emphasised throughout the whole paper.

The phrase "all participants that had not yet had their 36-week gestational follow-up were eligible to participate" suggests that some people were more than 36 weeks pregnant. Is this the case? Otherwise you could simplify it and say women who were 36 weeks pregnant. It's important to be clear on the timing of the intervention given that this is one of your outcomes.
This sentence references those who were eligible to be included in the SWAT. Some participants in the host trial would have been more than 36 weeks pregnant (or have had their baby/babies) when the SWAT was implement, and thus they were not eligible to be included in the SWAT. The wording of this paragraph has been altered in relation to this comment, and the reviewer comment regarding the number of participants involved in the SWAT, hopefully this has made it clearer.

"Participants in MiQuit-3 were blind to their participation in this SWAT". Blinding is different to being unaware. Were they both blinded and unaware?
A correction has been made in regards to the unawareness and blinding of participants.

"The randomisation was undertaken by a statistician independent of the host trial, and of the staff involved in sending the texts". Explain how the statistician did the randomisation, e.g. computer generated. Also, explain how he communicated that randomisation to the researcher assigning the women to each group.
Details have been added to explain how the randomisation was generated (Stata) and how the randomisation sequence was used.

"Block randomisation, stratified by host trial allocation, and whether they had completed the previous follow-up; with varying block sizes of 4, 8, 12 and 16". This is not a sentence.
Corrections have been made to this sentence.

"A £5 voucher was given to all participants who completed a follow-up…". Was this part of the host trial or the SWAT?
Detail has been added to clarify that this is part of the host trial

"…additionally those who provided a saliva sample were given another £30 (£35 total)." Where does the saliva sample come into it? Is this part of the host trial? Explain in the paper.
Detail has been added to explain why a saliva sample was given, and that it, and associated voucher were part of the host trial.

How did you decide on how many to include for the SWAT. I accept that a sample size calculation is not required but a line in the paper on why you decided on (it appears to be 200) would be useful.
Further details have been added to explain why only 200 were included.

In the secondary outcome, explain what you mean by completed by any method.
Details have been added to explain what other methods of completion could be used.

"Time to response, defined as the number of days between the due date of the 36-week gestation follow-up and the date the questionnaire was recorded as complete". Are you certain all follow-up calls were made in 24 hours?
The date of each attempt to contact was recorded. However this outcome captures completion by any follow-up method.

Statistical Analysis
"The primary outcome was completion rate; defined as the proportion of the questionnaires completed over the telephone within the follow-up window (14 days)". However, you then go on to say that you used logistic regression to analyse this. Logistic regression is not suitable for four categories. I suspect what you mean is that you compared the completion rates across the two personalised/not personalised and again early/late. I can see you did this from the table. However, you need to articulate that in the text because it is currently confusing.
A logistic model can only deal with two levels of an outcome; it can handle multiple levels of a covariate. The outcome was completion (yes/no), and the swat allocations are included as covariates in the model. Each swat allocation is include as a separate covariate – as stated in the analysis section, providing two, two-level covariates. Thus the model does not include a four level variable, as you have suggested. Had swat allocation been included as a four level covariate, the model would have been able to do this – but it would not have provided an OR for each of the two SWAT interventions separately. This method is the appropriate method for analysing a binary outcome for a factorial SWAT, as such no changes have been made to the text.

In statistical analysis, a full stop after level and a new sentence for "As this is a factorial..."
Thank you for suggesting this correction – it has been made.

Full stop required in this sentence too. Also, I suspect the word "compared" in this sentence should read completed. - "Time to response (days between questionnaire due and complete) was analysed using a Cox Proportional Hazards regression, those who compared the questionnaire early had their time set to 0.1, those did not complete were censored at either last contact date or 120 days if not contacted, and those who withdrew in the course of the SWAT were set to their withdrawal date".
Corrections to this sentence have been made.

Again the following is not a sentence, "All models were repeated with the inclusion of an interaction term to explore any possible interactions between the two SWAT interventions; with a significance level of 5%."
Additional words have been added.

Results
"Additional participants were excluded from the analysis, where the covariates required for the model were not provided". What additional participants? Quantify and explain please. Why were the covariates not 'provided'? Explain please.
The text in this sentence has been altered to make this clearer.

"Three participants were not contacted due to difficulties/adverse events associated with their pregnancy but are still included in the analysis under ITT principles". Commas are required to make sense of the sentence.
Changes have been made to this sentence.

In your flow chart, what does primary refer to, and proximity? Add an explanation or use a term that explains a little better. Also in the flow chart, you say response rate but provide the number of participants. This is not a rate.
The flow diagram has been updated to include consistent terminology. The numbers in the flow diagram are referencing the numbers within each analysis.

In the primary outcome, continue with your phraseology - the 14-day follow-up window rather than "within 14 days of the due date".
This term has been made consistent throughout the paper.

I find the writing of the results very confusing. This sentence below suggests the outcome was completion rate via telephone versus completion rate via something else. "There was evidence to suggest a difference in completion rate via telephone adjusted OR 0.44 (0.22–0.87, p=0.02) which implies those who received the non-personalised text were more likely to complete the questionnaire when completing via the telephone". I'm wondering why you keep saying via telephone. It is particularly confusing when explaining the results. The last part above again says … those who received the non-personalised text were more likely to complete the questionnaire when completing via the telephone". It looks like the method of completion is the purpose of the study.
The main data collection method for MiQuit-3 was via the telephone. As such, within the SWAT protocol for this factorial SWAT, it was decided that the primary outcome would focus specifically on data collected via this method, and as a secondary explore data collected via any method. The results for the primary and associated secondary outcome have been rewritten slightly to clarify that the results are only including those who completed it via the telephone, as opposed to comparing those who did it via the telephone with something else.

When you use the phrase "were more likely to", you must give the details of the comparison, i.e., more likely than who?
Additional words have been added for clarification.

It is implied, but not adequately explained, that some women completed the questionnaire by some other means. It is not clear how this was handled in terms of the numbers analysed throughout the study and this needs to be explained.
For the secondary outcomes (time to completion, and number of attempts to contact) there have been no additional adjustments or changes to the analysis due to the method of data collection. The completion method is only relevant to the primary outcome, where the trial team detailed that their main method of data collection was via the telephone, and thus the primary outcome explores this. However, to explore the effectiveness of the interventions on any data collection method (as is more typically done in RCTs where a number of methods are utilised) a secondary outcome was to explore this via any method.

For your tables 3 and 4, add the number of women. Why does the left column say "response rate for all methods" when the primary outcome is defined as the proportion of the questionnaires completed over the telephone…"
Response rate, completion rate and retention rate are all frequently used synonymously within the SWAT world. As such the terminology within this paper has switched between the terms. In response to this comment the terminology has been made consistent.

The heading in the results section, "Response rates for all method", do you mean completion rates for all methods? A response rate is different.
Response rate has been changed to completion rate throughout.

If you hang your hat on statistically significant evidence, by quoting CIs and p-values, to establish if your SWAT was effective, or not, then the following has no place in your paper. "There is some, non statistically significant, evidence to suggest that there may be a difference in response rate for personalised versus non-personalised text reminders; adjusted OR 0.61 (95% CI 0.30–1.24, p=0.17), in favour of the non-personalised text messages". You cannot say there is non-statistically significant evidence and then support that statement with statistics! Remove this please.
This section has been reworded to reflect your suggestion.

The heading, "Number of attempts to contact required". Replace with, Number of attempts required to contact the women.
The authors have declined to make this change. In some instances the participant was not contacted, therefore it would be incorrect for the heading to referring the number required to contact them, when some were not contactable. Additionally, as explained in a previous comment we will not be changing participant to women.

What do you mean by the maximum number of calls, as stated here "The maximum number of calls was reached for 55 of the 174 participants…"?
This has been clarified both when describing the outcome, and in the results. The researcher tried to contact the participant a maximum of 6 times.

Contacts required is a new term introduced here " There was no evidence of a difference in number of contacts required". What do you mean by it?
This is referring the number of attempts to contact – a word had been omitted, and has been corrected.

"The average time to respond was 6.2 days (ranging from 5 days early to 103 days late)". Respond to what, the phone call or the text, or the questionnaire?
Terminology has been changed.

"This was similar between those who received a personalised text (8.2 days for early versus 7.1 days for late) and those who received the non-personalised text (4.9 days for early versus 4.7 days for late), but there is a slight difference between those who received personalised or non-personalised texts". If it is similar, how can there be a slight difference? What point are you making here?
The wording of the sentence has been changed to clarify this point. The sentence was meant to emphasis the difference in time between the personalised/non-personalised, whilst highlighting the similarities between early/late.

Include the number of participants in the MiQuit Trial earlier in the paper when discussing the 200 randomised for the SWAT.
The authors have included this figure (n=1002) earlier on in the paper.

Discussion
"It did show that there was some evidence that sending a non-personalised text message reminder would have a larger increase in response than sending personalised text messages did". This is misleading because it was not the case when all methods of questionnaire were included. Please amend the statement.
Additional words have been added to explain that these results are only valid for telephone completion

The final sentence of the discussion needs to be reviewed. What do you mean by overall effectiveness?
These words have been removed.
Competing Interests: I have no competing interests to declare. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 30 Sep 2021

Elizabeth Coleman, York Trials Unit, Department of Health Sciences, University of York, UK, York, YO10 5DD, UK

30 Sep 2021

Author Response

The authors would like to thank you for a very thorough review of this article - please see responds to your comments and suggestions below. We have amended the article ... Continue reading The authors would like to thank you for a very thorough review of this article - please see responds to your comments and suggestions below. We have amended the article to incorporate your corrections and suggestions were we pleased it was suitable to do so.

Summary
This was a SWAT within the MiQuit-3 trial. The purpose of the SWAT was to establish if the timing or personalisation of text messages increases completion of a questionnaire. Study design was a 2 x 2 factorial SWAT. Participants who reached their 36-week gestational follow-up were randomised to receive a personalised or non-personalised text message, either one week or one day prior to the telephone follow-up. Primary outcome was completion rate of questionnaire via telephone. Secondary outcomes included: completion rate via any method, time to completion, and number of reminders required. The authors concluded that timing of the text message did not appear to influence the retention of participants. The authors concluded that personalisation of a text message may be detrimental to retention; however, more SWATs should be undertaken in this field.
Thank you for this well written summary of the paper.

Major Comments
My first comment is on the title, the purpose of the study and the conclusions drawn. The title, correctly refers to the retention of participants in the smoking cessation pregnancy RCT. The purpose of the SWAT though is to evaluate completion of a questionnaire, not retention in the host trial, as claimed in the conclusion in the abstract. I think this is conflated throughout and the authors need to consider this carefully and amend their paper. In fact, the registered SWATs in the NI SWAT repository give the outcomes in both as questionnaire completion. So therefore the conclusions drawn in the abstract and in the discussion are not supported by the data.
The authors have amended the title of the paper, and changed references to retention within the paper to completion of questionnaires throughout. Alongside making the other suggested changes by the reviewer, we hope to have addressed any concerns in relation to the conclusions drawn.

Abstract
"There was evidence that a personalised text would result in fewer completions via telephone compared with a non-personalised text (adjusted OR 0.44, 95% CI 0.22–0.87, p=0.02)". This statement is confusing. The research question is personalised text message versus non-personalised text message. If using via telephone, then it should be also say non-personalised text via telephone.
The wording has been amended to attempt to make this statement clearer, and also reference the results for any data collection method.

Also, use text message, rather than text, throughout.
The term ‘text’ has been changed to ‘text message’ throughout the paper.

"Personalisation of a text message may be detrimental to retention". I don't think your results support a statement this strong. Firstly, this is not a trial with an adequate sample size to make this claim. Secondly, when you included all methods of receiving the questionnaire (I think this is correct interpretation but it is challenging to establish in the current version of the paper as the detail on all methods is unclear) you did not find that personalised or non-personalised texts mattered. At best you can say, personalisation of a text message appears to affect questionnaire completion via telephone.
This sentence has been altered to include ‘data was collected only via the telephone’, and now also references that the results do not hold for data collection via all methods. The results are correct as finding of this work, and the conclusion of the abstract references that more work is needed in this area so no other alterations have been made in regards to this comment.

Introduction
I like the succinct introduction.
Thank you for this comment.

Methods
The methods says "The SWATs are also registered with the Northern Ireland Hub for Trial Methodology Research SWAT Repository (SWATs 35 and 44; both registered December 2015). It's a little confusing to the reader. It looks like you're conducting two SWATs. I understand what you have taken both of these SWATS and conducted the two in a 2 x 2 factorial design. However, make this clear to the reader.
Additional words have been added to this paragraph to make it clear that this was a factorial SWAT involving these two SWATs.

Replace the phrase "carried out" with conducted throughout. I know the phrase carried out is used all the time in literature but it is not correct unless you are describing physically carrying an object.
The phase carried out has been replaced with conducted.

I would use the word women rather than participants, given that it is a trial of pregnant women.
We thank you for the suggestion – however, the authors have decided not to make this change. The protocol for the MiQuit-3 trial refers to those involved both as participants, and as women. The term participants is not incorrect, and as it is made clear that this work is being done in pregnant women in both the introduction and conclusions of this paper, we feel it does not need to be re-emphasised throughout the whole paper.

The phrase "all participants that had not yet had their 36-week gestational follow-up were eligible to participate" suggests that some people were more than 36 weeks pregnant. Is this the case? Otherwise you could simplify it and say women who were 36 weeks pregnant. It's important to be clear on the timing of the intervention given that this is one of your outcomes.
This sentence references those who were eligible to be included in the SWAT. Some participants in the host trial would have been more than 36 weeks pregnant (or have had their baby/babies) when the SWAT was implement, and thus they were not eligible to be included in the SWAT. The wording of this paragraph has been altered in relation to this comment, and the reviewer comment regarding the number of participants involved in the SWAT, hopefully this has made it clearer.

"Participants in MiQuit-3 were blind to their participation in this SWAT". Blinding is different to being unaware. Were they both blinded and unaware?
A correction has been made in regards to the unawareness and blinding of participants.

"The randomisation was undertaken by a statistician independent of the host trial, and of the staff involved in sending the texts". Explain how the statistician did the randomisation, e.g. computer generated. Also, explain how he communicated that randomisation to the researcher assigning the women to each group.
Details have been added to explain how the randomisation was generated (Stata) and how the randomisation sequence was used.

"Block randomisation, stratified by host trial allocation, and whether they had completed the previous follow-up; with varying block sizes of 4, 8, 12 and 16". This is not a sentence.
Corrections have been made to this sentence.

"A £5 voucher was given to all participants who completed a follow-up…". Was this part of the host trial or the SWAT?
Detail has been added to clarify that this is part of the host trial

"…additionally those who provided a saliva sample were given another £30 (£35 total)." Where does the saliva sample come into it? Is this part of the host trial? Explain in the paper.
Detail has been added to explain why a saliva sample was given, and that it, and associated voucher were part of the host trial.

How did you decide on how many to include for the SWAT. I accept that a sample size calculation is not required but a line in the paper on why you decided on (it appears to be 200) would be useful.
Further details have been added to explain why only 200 were included.

In the secondary outcome, explain what you mean by completed by any method.
Details have been added to explain what other methods of completion could be used.

"Time to response, defined as the number of days between the due date of the 36-week gestation follow-up and the date the questionnaire was recorded as complete". Are you certain all follow-up calls were made in 24 hours?
The date of each attempt to contact was recorded. However this outcome captures completion by any follow-up method.

Statistical Analysis
"The primary outcome was completion rate; defined as the proportion of the questionnaires completed over the telephone within the follow-up window (14 days)". However, you then go on to say that you used logistic regression to analyse this. Logistic regression is not suitable for four categories. I suspect what you mean is that you compared the completion rates across the two personalised/not personalised and again early/late. I can see you did this from the table. However, you need to articulate that in the text because it is currently confusing.
A logistic model can only deal with two levels of an outcome; it can handle multiple levels of a covariate. The outcome was completion (yes/no), and the swat allocations are included as covariates in the model. Each swat allocation is include as a separate covariate – as stated in the analysis section, providing two, two-level covariates. Thus the model does not include a four level variable, as you have suggested. Had swat allocation been included as a four level covariate, the model would have been able to do this – but it would not have provided an OR for each of the two SWAT interventions separately. This method is the appropriate method for analysing a binary outcome for a factorial SWAT, as such no changes have been made to the text.

In statistical analysis, a full stop after level and a new sentence for "As this is a factorial..."
Thank you for suggesting this correction – it has been made.

Full stop required in this sentence too. Also, I suspect the word "compared" in this sentence should read completed. - "Time to response (days between questionnaire due and complete) was analysed using a Cox Proportional Hazards regression, those who compared the questionnaire early had their time set to 0.1, those did not complete were censored at either last contact date or 120 days if not contacted, and those who withdrew in the course of the SWAT were set to their withdrawal date".
Corrections to this sentence have been made.

Again the following is not a sentence, "All models were repeated with the inclusion of an interaction term to explore any possible interactions between the two SWAT interventions; with a significance level of 5%."
Additional words have been added.

Results
"Additional participants were excluded from the analysis, where the covariates required for the model were not provided". What additional participants? Quantify and explain please. Why were the covariates not 'provided'? Explain please.
The text in this sentence has been altered to make this clearer.

"Three participants were not contacted due to difficulties/adverse events associated with their pregnancy but are still included in the analysis under ITT principles". Commas are required to make sense of the sentence.
Changes have been made to this sentence.

In your flow chart, what does primary refer to, and proximity? Add an explanation or use a term that explains a little better. Also in the flow chart, you say response rate but provide the number of participants. This is not a rate.
The flow diagram has been updated to include consistent terminology. The numbers in the flow diagram are referencing the numbers within each analysis.

In the primary outcome, continue with your phraseology - the 14-day follow-up window rather than "within 14 days of the due date".
This term has been made consistent throughout the paper.

I find the writing of the results very confusing. This sentence below suggests the outcome was completion rate via telephone versus completion rate via something else. "There was evidence to suggest a difference in completion rate via telephone adjusted OR 0.44 (0.22–0.87, p=0.02) which implies those who received the non-personalised text were more likely to complete the questionnaire when completing via the telephone". I'm wondering why you keep saying via telephone. It is particularly confusing when explaining the results. The last part above again says … those who received the non-personalised text were more likely to complete the questionnaire when completing via the telephone". It looks like the method of completion is the purpose of the study.
The main data collection method for MiQuit-3 was via the telephone. As such, within the SWAT protocol for this factorial SWAT, it was decided that the primary outcome would focus specifically on data collected via this method, and as a secondary explore data collected via any method. The results for the primary and associated secondary outcome have been rewritten slightly to clarify that the results are only including those who completed it via the telephone, as opposed to comparing those who did it via the telephone with something else.

When you use the phrase "were more likely to", you must give the details of the comparison, i.e., more likely than who?
Additional words have been added for clarification.

It is implied, but not adequately explained, that some women completed the questionnaire by some other means. It is not clear how this was handled in terms of the numbers analysed throughout the study and this needs to be explained.
For the secondary outcomes (time to completion, and number of attempts to contact) there have been no additional adjustments or changes to the analysis due to the method of data collection. The completion method is only relevant to the primary outcome, where the trial team detailed that their main method of data collection was via the telephone, and thus the primary outcome explores this. However, to explore the effectiveness of the interventions on any data collection method (as is more typically done in RCTs where a number of methods are utilised) a secondary outcome was to explore this via any method.

For your tables 3 and 4, add the number of women. Why does the left column say "response rate for all methods" when the primary outcome is defined as the proportion of the questionnaires completed over the telephone…"
Response rate, completion rate and retention rate are all frequently used synonymously within the SWAT world. As such the terminology within this paper has switched between the terms. In response to this comment the terminology has been made consistent.

The heading in the results section, "Response rates for all method", do you mean completion rates for all methods? A response rate is different.
Response rate has been changed to completion rate throughout.

If you hang your hat on statistically significant evidence, by quoting CIs and p-values, to establish if your SWAT was effective, or not, then the following has no place in your paper. "There is some, non statistically significant, evidence to suggest that there may be a difference in response rate for personalised versus non-personalised text reminders; adjusted OR 0.61 (95% CI 0.30–1.24, p=0.17), in favour of the non-personalised text messages". You cannot say there is non-statistically significant evidence and then support that statement with statistics! Remove this please.
This section has been reworded to reflect your suggestion.

The heading, "Number of attempts to contact required". Replace with, Number of attempts required to contact the women.
The authors have declined to make this change. In some instances the participant was not contacted, therefore it would be incorrect for the heading to referring the number required to contact them, when some were not contactable. Additionally, as explained in a previous comment we will not be changing participant to women.

What do you mean by the maximum number of calls, as stated here "The maximum number of calls was reached for 55 of the 174 participants…"?
This has been clarified both when describing the outcome, and in the results. The researcher tried to contact the participant a maximum of 6 times.

Contacts required is a new term introduced here " There was no evidence of a difference in number of contacts required". What do you mean by it?
This is referring the number of attempts to contact – a word had been omitted, and has been corrected.

"The average time to respond was 6.2 days (ranging from 5 days early to 103 days late)". Respond to what, the phone call or the text, or the questionnaire?
Terminology has been changed.

"This was similar between those who received a personalised text (8.2 days for early versus 7.1 days for late) and those who received the non-personalised text (4.9 days for early versus 4.7 days for late), but there is a slight difference between those who received personalised or non-personalised texts". If it is similar, how can there be a slight difference? What point are you making here?
The wording of the sentence has been changed to clarify this point. The sentence was meant to emphasis the difference in time between the personalised/non-personalised, whilst highlighting the similarities between early/late.

Include the number of participants in the MiQuit Trial earlier in the paper when discussing the 200 randomised for the SWAT.
The authors have included this figure (n=1002) earlier on in the paper.

Discussion
"It did show that there was some evidence that sending a non-personalised text message reminder would have a larger increase in response than sending personalised text messages did". This is misleading because it was not the case when all methods of questionnaire were included. Please amend the statement.
Additional words have been added to explain that these results are only valid for telephone completion

The final sentence of the discussion needs to be reviewed. What do you mean by overall effectiveness?
These words have been removed.
The authors would like to thank you for a very thorough review of this article - please see responds to your comments and suggestions below. We have amended the article to incorporate your corrections and suggestions were we pleased it was suitable to do so.

Summary
This was a SWAT within the MiQuit-3 trial. The purpose of the SWAT was to establish if the timing or personalisation of text messages increases completion of a questionnaire. Study design was a 2 x 2 factorial SWAT. Participants who reached their 36-week gestational follow-up were randomised to receive a personalised or non-personalised text message, either one week or one day prior to the telephone follow-up. Primary outcome was completion rate of questionnaire via telephone. Secondary outcomes included: completion rate via any method, time to completion, and number of reminders required. The authors concluded that timing of the text message did not appear to influence the retention of participants. The authors concluded that personalisation of a text message may be detrimental to retention; however, more SWATs should be undertaken in this field.
Thank you for this well written summary of the paper.

Major Comments
My first comment is on the title, the purpose of the study and the conclusions drawn. The title, correctly refers to the retention of participants in the smoking cessation pregnancy RCT. The purpose of the SWAT though is to evaluate completion of a questionnaire, not retention in the host trial, as claimed in the conclusion in the abstract. I think this is conflated throughout and the authors need to consider this carefully and amend their paper. In fact, the registered SWATs in the NI SWAT repository give the outcomes in both as questionnaire completion. So therefore the conclusions drawn in the abstract and in the discussion are not supported by the data.
The authors have amended the title of the paper, and changed references to retention within the paper to completion of questionnaires throughout. Alongside making the other suggested changes by the reviewer, we hope to have addressed any concerns in relation to the conclusions drawn.

Abstract
"There was evidence that a personalised text would result in fewer completions via telephone compared with a non-personalised text (adjusted OR 0.44, 95% CI 0.22–0.87, p=0.02)". This statement is confusing. The research question is personalised text message versus non-personalised text message. If using via telephone, then it should be also say non-personalised text via telephone.
The wording has been amended to attempt to make this statement clearer, and also reference the results for any data collection method.

Also, use text message, rather than text, throughout.
The term ‘text’ has been changed to ‘text message’ throughout the paper.

"Personalisation of a text message may be detrimental to retention". I don't think your results support a statement this strong. Firstly, this is not a trial with an adequate sample size to make this claim. Secondly, when you included all methods of receiving the questionnaire (I think this is correct interpretation but it is challenging to establish in the current version of the paper as the detail on all methods is unclear) you did not find that personalised or non-personalised texts mattered. At best you can say, personalisation of a text message appears to affect questionnaire completion via telephone.
This sentence has been altered to include ‘data was collected only via the telephone’, and now also references that the results do not hold for data collection via all methods. The results are correct as finding of this work, and the conclusion of the abstract references that more work is needed in this area so no other alterations have been made in regards to this comment.

Introduction
I like the succinct introduction.
Thank you for this comment.

Methods
The methods says "The SWATs are also registered with the Northern Ireland Hub for Trial Methodology Research SWAT Repository (SWATs 35 and 44; both registered December 2015). It's a little confusing to the reader. It looks like you're conducting two SWATs. I understand what you have taken both of these SWATS and conducted the two in a 2 x 2 factorial design. However, make this clear to the reader.
Additional words have been added to this paragraph to make it clear that this was a factorial SWAT involving these two SWATs.

Replace the phrase "carried out" with conducted throughout. I know the phrase carried out is used all the time in literature but it is not correct unless you are describing physically carrying an object.
The phase carried out has been replaced with conducted.

I would use the word women rather than participants, given that it is a trial of pregnant women.
We thank you for the suggestion – however, the authors have decided not to make this change. The protocol for the MiQuit-3 trial refers to those involved both as participants, and as women. The term participants is not incorrect, and as it is made clear that this work is being done in pregnant women in both the introduction and conclusions of this paper, we feel it does not need to be re-emphasised throughout the whole paper.

The phrase "all participants that had not yet had their 36-week gestational follow-up were eligible to participate" suggests that some people were more than 36 weeks pregnant. Is this the case? Otherwise you could simplify it and say women who were 36 weeks pregnant. It's important to be clear on the timing of the intervention given that this is one of your outcomes.
This sentence references those who were eligible to be included in the SWAT. Some participants in the host trial would have been more than 36 weeks pregnant (or have had their baby/babies) when the SWAT was implement, and thus they were not eligible to be included in the SWAT. The wording of this paragraph has been altered in relation to this comment, and the reviewer comment regarding the number of participants involved in the SWAT, hopefully this has made it clearer.

"Participants in MiQuit-3 were blind to their participation in this SWAT". Blinding is different to being unaware. Were they both blinded and unaware?
A correction has been made in regards to the unawareness and blinding of participants.

"The randomisation was undertaken by a statistician independent of the host trial, and of the staff involved in sending the texts". Explain how the statistician did the randomisation, e.g. computer generated. Also, explain how he communicated that randomisation to the researcher assigning the women to each group.
Details have been added to explain how the randomisation was generated (Stata) and how the randomisation sequence was used.

"Block randomisation, stratified by host trial allocation, and whether they had completed the previous follow-up; with varying block sizes of 4, 8, 12 and 16". This is not a sentence.
Corrections have been made to this sentence.

"A £5 voucher was given to all participants who completed a follow-up…". Was this part of the host trial or the SWAT?
Detail has been added to clarify that this is part of the host trial

"…additionally those who provided a saliva sample were given another £30 (£35 total)." Where does the saliva sample come into it? Is this part of the host trial? Explain in the paper.
Detail has been added to explain why a saliva sample was given, and that it, and associated voucher were part of the host trial.

How did you decide on how many to include for the SWAT. I accept that a sample size calculation is not required but a line in the paper on why you decided on (it appears to be 200) would be useful.
Further details have been added to explain why only 200 were included.

In the secondary outcome, explain what you mean by completed by any method.
Details have been added to explain what other methods of completion could be used.

"Time to response, defined as the number of days between the due date of the 36-week gestation follow-up and the date the questionnaire was recorded as complete". Are you certain all follow-up calls were made in 24 hours?
The date of each attempt to contact was recorded. However this outcome captures completion by any follow-up method.

Statistical Analysis
"The primary outcome was completion rate; defined as the proportion of the questionnaires completed over the telephone within the follow-up window (14 days)". However, you then go on to say that you used logistic regression to analyse this. Logistic regression is not suitable for four categories. I suspect what you mean is that you compared the completion rates across the two personalised/not personalised and again early/late. I can see you did this from the table. However, you need to articulate that in the text because it is currently confusing.
A logistic model can only deal with two levels of an outcome; it can handle multiple levels of a covariate. The outcome was completion (yes/no), and the swat allocations are included as covariates in the model. Each swat allocation is include as a separate covariate – as stated in the analysis section, providing two, two-level covariates. Thus the model does not include a four level variable, as you have suggested. Had swat allocation been included as a four level covariate, the model would have been able to do this – but it would not have provided an OR for each of the two SWAT interventions separately. This method is the appropriate method for analysing a binary outcome for a factorial SWAT, as such no changes have been made to the text.

In statistical analysis, a full stop after level and a new sentence for "As this is a factorial..."
Thank you for suggesting this correction – it has been made.

Full stop required in this sentence too. Also, I suspect the word "compared" in this sentence should read completed. - "Time to response (days between questionnaire due and complete) was analysed using a Cox Proportional Hazards regression, those who compared the questionnaire early had their time set to 0.1, those did not complete were censored at either last contact date or 120 days if not contacted, and those who withdrew in the course of the SWAT were set to their withdrawal date".
Corrections to this sentence have been made.

Again the following is not a sentence, "All models were repeated with the inclusion of an interaction term to explore any possible interactions between the two SWAT interventions; with a significance level of 5%."
Additional words have been added.

Results
"Additional participants were excluded from the analysis, where the covariates required for the model were not provided". What additional participants? Quantify and explain please. Why were the covariates not 'provided'? Explain please.
The text in this sentence has been altered to make this clearer.

"Three participants were not contacted due to difficulties/adverse events associated with their pregnancy but are still included in the analysis under ITT principles". Commas are required to make sense of the sentence.
Changes have been made to this sentence.

In your flow chart, what does primary refer to, and proximity? Add an explanation or use a term that explains a little better. Also in the flow chart, you say response rate but provide the number of participants. This is not a rate.
The flow diagram has been updated to include consistent terminology. The numbers in the flow diagram are referencing the numbers within each analysis.

In the primary outcome, continue with your phraseology - the 14-day follow-up window rather than "within 14 days of the due date".
This term has been made consistent throughout the paper.

I find the writing of the results very confusing. This sentence below suggests the outcome was completion rate via telephone versus completion rate via something else. "There was evidence to suggest a difference in completion rate via telephone adjusted OR 0.44 (0.22–0.87, p=0.02) which implies those who received the non-personalised text were more likely to complete the questionnaire when completing via the telephone". I'm wondering why you keep saying via telephone. It is particularly confusing when explaining the results. The last part above again says … those who received the non-personalised text were more likely to complete the questionnaire when completing via the telephone". It looks like the method of completion is the purpose of the study.
The main data collection method for MiQuit-3 was via the telephone. As such, within the SWAT protocol for this factorial SWAT, it was decided that the primary outcome would focus specifically on data collected via this method, and as a secondary explore data collected via any method. The results for the primary and associated secondary outcome have been rewritten slightly to clarify that the results are only including those who completed it via the telephone, as opposed to comparing those who did it via the telephone with something else.

When you use the phrase "were more likely to", you must give the details of the comparison, i.e., more likely than who?
Additional words have been added for clarification.

It is implied, but not adequately explained, that some women completed the questionnaire by some other means. It is not clear how this was handled in terms of the numbers analysed throughout the study and this needs to be explained.
For the secondary outcomes (time to completion, and number of attempts to contact) there have been no additional adjustments or changes to the analysis due to the method of data collection. The completion method is only relevant to the primary outcome, where the trial team detailed that their main method of data collection was via the telephone, and thus the primary outcome explores this. However, to explore the effectiveness of the interventions on any data collection method (as is more typically done in RCTs where a number of methods are utilised) a secondary outcome was to explore this via any method.

For your tables 3 and 4, add the number of women. Why does the left column say "response rate for all methods" when the primary outcome is defined as the proportion of the questionnaires completed over the telephone…"
Response rate, completion rate and retention rate are all frequently used synonymously within the SWAT world. As such the terminology within this paper has switched between the terms. In response to this comment the terminology has been made consistent.

The heading in the results section, "Response rates for all method", do you mean completion rates for all methods? A response rate is different.
Response rate has been changed to completion rate throughout.

If you hang your hat on statistically significant evidence, by quoting CIs and p-values, to establish if your SWAT was effective, or not, then the following has no place in your paper. "There is some, non statistically significant, evidence to suggest that there may be a difference in response rate for personalised versus non-personalised text reminders; adjusted OR 0.61 (95% CI 0.30–1.24, p=0.17), in favour of the non-personalised text messages". You cannot say there is non-statistically significant evidence and then support that statement with statistics! Remove this please.
This section has been reworded to reflect your suggestion.

The heading, "Number of attempts to contact required". Replace with, Number of attempts required to contact the women.
The authors have declined to make this change. In some instances the participant was not contacted, therefore it would be incorrect for the heading to referring the number required to contact them, when some were not contactable. Additionally, as explained in a previous comment we will not be changing participant to women.

What do you mean by the maximum number of calls, as stated here "The maximum number of calls was reached for 55 of the 174 participants…"?
This has been clarified both when describing the outcome, and in the results. The researcher tried to contact the participant a maximum of 6 times.

Contacts required is a new term introduced here " There was no evidence of a difference in number of contacts required". What do you mean by it?
This is referring the number of attempts to contact – a word had been omitted, and has been corrected.

"The average time to respond was 6.2 days (ranging from 5 days early to 103 days late)". Respond to what, the phone call or the text, or the questionnaire?
Terminology has been changed.

"This was similar between those who received a personalised text (8.2 days for early versus 7.1 days for late) and those who received the non-personalised text (4.9 days for early versus 4.7 days for late), but there is a slight difference between those who received personalised or non-personalised texts". If it is similar, how can there be a slight difference? What point are you making here?
The wording of the sentence has been changed to clarify this point. The sentence was meant to emphasis the difference in time between the personalised/non-personalised, whilst highlighting the similarities between early/late.

Include the number of participants in the MiQuit Trial earlier in the paper when discussing the 200 randomised for the SWAT.
The authors have included this figure (n=1002) earlier on in the paper.

Discussion
"It did show that there was some evidence that sending a non-personalised text message reminder would have a larger increase in response than sending personalised text messages did". This is misleading because it was not the case when all methods of questionnaire were included. Please amend the statement.
Additional words have been added to explain that these results are only valid for telephone completion

The final sentence of the discussion needs to be reviewed. What do you mean by overall effectiveness?
These words have been removed.
Competing Interests: I have no competing interests to declare. Close
Report a concern

Views

Reviewer Report 29 Jul 2021

Phil J. Edwards, Department of Population Health, London School of Hygiene and Tropical Medicine, London, UK

Approved

https://doi.org/10.5256/f1000research.55180.r90364

This manuscript reports the results of a 2x2 factorial trial nested within an RCT of MiQuit, a text-message, smoking cessation self-help support programme for pregnant smokers. The nested factorial trial sought to evaluate the effect on response to a questionnaire administered over the telephone of two interventions applied to a pre-notification text message: (i) personalisation (text begins “Hi [name]”, or not), and (ii) timing of text messages (early: one week before follow-up, or late: one day before follow-up).

194 participants who had not yet had their 36-week gestational follow-up were randomised into this nested trial. Analysis of intervention effects was conducted using a logistic regression model.

The study found some evidence that personalised text messages reduced response (OR = 0.44; 95% CI 0.22 to 0.87; p=0.02); and no evidence that the earlier timing of text messages had an effect on response (OR = 0.86; 95% CI 0.44 to 1.67; p=0.65).

The manuscript is a clear and concise account of the study, citing the current literature. The study design is appropriate and the work appears to be technically sound. The authors appropriately recognise that their results are only generalisable to their study population (females aged 17 to 41 years).

The conclusions are adequately supported by the results, and the study makes a useful contribution to the data collection literature.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Evidence-based Data Collection

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Author Response 30 Sep 2021

Elizabeth Coleman, York Trials Unit, Department of Health Sciences, University of York, UK, York, YO10 5DD, UK

30 Sep 2021

Author Response

Thank you for reviewing this article.
Competing Interests: No competing interests were disclosed.
Thank you for reviewing this article.
Thank you for reviewing this article.
Competing Interests: No competing interests were disclosed. Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 30 Sep 2021

Elizabeth Coleman, York Trials Unit, Department of Health Sciences, University of York, UK, York, YO10 5DD, UK

30 Sep 2021

Author Response

Thank you for reviewing this article.
Competing Interests: No competing interests were disclosed.
Thank you for reviewing this article.
Thank you for reviewing this article.
Competing Interests: No competing interests were disclosed. Close
Report a concern

Comments on this article Comments (0)

Version 2

VERSION 2 PUBLISHED 22 Jul 2021

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 2 (revision) 30 Sep 21		read
Version 1 22 Jul 21	read	read

Phil J. Edwards, London School of Hygiene and Tropical Medicine, London, UK
Frances Shiely, University College Cork, Cork, Ireland; University College cork, Cork, Ireland

Comments on this article

All Comments(0)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

12 Views

04 Oct 2021 | for Version 2

12 Views Cite this report Responses(0)

Approved

Thank you to authors for making the suggested changes. I believe these amendments will allow for this SWAT to be replicated. I have no further comments.

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Epidemiology; Trial Methodology; SWATs

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

31 Views

04 Aug 2021 | for Version 1

31 Views Cite this report Responses(1)

Approved With Reservations

"There was evidence that a personalised text would result in fewer completions via telephone compared with a non-personalised text (adjusted OR 0.44, 95% CI 0.22–0.87, p=0.02)". This statement is confusing. The research question is personalised text message versus non-personalised text message. If using via telephone, then it should be also say non-personalised text via telephone.
Also, use text message, rather than text, throughout.
"Personalisation of a text message may be detrimental to retention". I don't think your results support a statement this strong. Firstly, this is not a trial with an adequate sample size to make this claim. Secondly, when you included all methods of receiving the questionnaire (I think this is correct interpretation but it is challenging to establish in the current version of the paper as the detail on all methods is unclear) you did not find that personalised or non-personalised texts mattered. At best you can say, personalisation of a text message appears to affect questionnaire completion via telephone.

Introduction

I like the succinct introduction.

Methods

The methods says "The SWATs are also registered with the Northern Ireland Hub for Trial Methodology Research SWAT Repository (SWATs 35 and 44; both registered December 2015). It's a little confusing to the reader. It looks like you're conducting two SWATs. I understand what you have taken both of these SWATS and conducted the two in a 2 x 2 factorial design. However, make this clear to the reader.
Replace the phrase "carried out" with conducted throughout. I know the phrase carried out is used all the time in literature but it is not correct unless you are describing physically carrying an object.
I would use the word women rather than participants, given that it is a trial of pregnant women.
The phrase "all participants that had not yet had their 36-week gestational follow-up were eligible to participate" suggests that some people were more than 36 weeks pregnant. Is this the case? Otherwise you could simplify it and say women who were 36 weeks pregnant. It's important to be clear on the timing of the intervention given that this is one of your outcomes.
"Participants in MiQuit-3 were blind to their participation in this SWAT". Blinding is different to being unaware. Were they both blinded and unaware?
"The randomisation was undertaken by a statistician independent of the host trial, and of the staff involved in sending the texts". Explain how the statistician did the randomisation, e.g. computer generated. Also, explain how he communicated that randomisation to the researcher assigning the women to each group.
"Block randomisation, stratified by host trial allocation, and whether they had completed the previous follow-up; with varying block sizes of 4, 8, 12 and 16". This is not a sentence.
"A £5 voucher was given to all participants who completed a follow-up…". Was this part of the host trial or the SWAT?
"…additionally those who provided a saliva sample were given another £30 (£35 total)." Where does the saliva sample come into it? Is this part of the host trial? Explain in the paper.
How did you decide on how many to include for the SWAT. I accept that a sample size calculation is not required but a line in the paper on why you decided on (it appears to be 200) would be useful.
In the secondary outcome, explain what you mean by completed by any method.
"Time to response, defined as the number of days between the due date of the 36-week gestation follow-up and the date the questionnaire was recorded as complete". Are you certain all follow-up calls were made in 24 hours?

Statistical Analysis

"The primary outcome was completion rate; defined as the proportion of the questionnaires completed over the telephone within the follow-up window (14 days)". However, you then go on to say that you used logistic regression to analyse this. Logistic regression is not suitable for four categories. I suspect what you mean is that you compared the completion rates across the two personalised/not personalised and again early/late. I can see you did this from the table. However, you need to articulate that in the text because it is currently confusing.
In statistical analysis, a full stop after level and a new sentence for "As this is a factorial..."
Full stop required in this sentence too. Also, I suspect the word "compared" in this sentence should read completed. - "Time to response (days between questionnaire due and complete) was analysed using a Cox Proportional Hazards regression, those who compared the questionnaire early had their time set to 0.1, those did not complete were censored at either last contact date or 120 days if not contacted, and those who withdrew in the course of the SWAT were set to their withdrawal date".
Again the following is not a sentence, "All models were repeated with the inclusion of an interaction term to explore any possible interactions between the two SWAT interventions; with a significance level of 5%."

Results

"Additional participants were excluded from the analysis, where the covariates required for the model were not provided". What additional participants? Quantify and explain please. Why were the covariates not 'provided'? Explain please.
"Three participants were not contacted due to difficulties/adverse events associated with their pregnancy but are still included in the analysis under ITT principles". Commas are required to make sense of the sentence.
In your flow chart, what does primary refer to, and proximity? Add an explanation or use a term that explains a little better. Also in the flow chart, you say response rate but provide the number of participants. This is not a rate.
In the primary outcome, continue with your phraseology - the 14-day follow-up window rather than "within 14 days of the due date".
I find the writing of the results very confusing. This sentence below suggests the outcome was completion rate via telephone versus completion rate via something else. "There was evidence to suggest a difference in completion rate via telephone adjusted OR 0.44 (0.22–0.87, p=0.02) which implies those who received the non-personalised text were more likely to complete the questionnaire when completing via the telephone". I'm wondering why you keep saying via telephone. It is particularly confusing when explaining the results. The last part above again says … those who received the non-personalised text were more likely to complete the questionnaire when completing via the telephone". It looks like the method of completion is the purpose of the study.
When you use the phrase "were more likely to", you must give the details of the comparison, i.e., more likely than who?
It is implied, but not adequately explained, that some women completed the questionnaire by some other means. It is not clear how this was handled in terms of the numbers analysed throughout the study and this needs to be explained.
For your tables 3 and 4, add the number of women. Why does the left column say "response rate for all methods" when the primary outcome is defined as the proportion of the questionnaires completed over the telephone…"
The heading in the results section, "Response rates for all method", do you mean completion rates for all methods? A response rate is different.
If you hang your hat on statistically significant evidence, by quoting CIs and p-values, to establish if your SWAT was effective, or not, then the following has no place in your paper. "There is some, non statistically significant, evidence to suggest that there may be a difference in response rate for personalised versus non-personalised text reminders; adjusted OR 0.61 (95% CI 0.30–1.24, p=0.17), in favour of the non-personalised text messages". You cannot say there is non-statistically significant evidence and then support that statement with statistics! Remove this please.
The heading, "Number of attempts to contact required". Replace with, Number of attempts required to contact the women.
What do you mean by the maximum number of calls, as stated here "The maximum number of calls was reached for 55 of the 174 participants…"?
Contacts required is a new term introduced here " There was no evidence of a difference in number of contacts required". What do you mean by it?
"The average time to respond was 6.2 days (ranging from 5 days early to 103 days late)". Respond to what, the phone call or the text, or the questionnaire?
"This was similar between those who received a personalised text (8.2 days for early versus 7.1 days for late) and those who received the non-personalised text (4.9 days for early versus 4.7 days for late), but there is a slight difference between those who received personalised or non-personalised texts". If it is similar, how can there be a slight difference? What point are you making here?
Include the number of participants in the MiQuit Trial earlier in the paper when discussing the 200 randomised for the SWAT.

Discussion

"It did show that there was some evidence that sending a non-personalised text message reminder would have a larger increase in response than sending personalised text messages did". This is misleading because it was not the case when all methods of questionnaire were included. Please amend the statement.
The final sentence of the discussion needs to be reviewed. What do you mean by overall effectiveness?

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

No

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Epidemiology; Trial Methodology; SWATs

Respond to this report

Responses (1)

Author Response

30 Sep 2021

Elizabeth Coleman, York Trials Unit, Department of Health Sciences, University of York, UK, York, YO10 5DD, UK

The authors would like to thank you for a very thorough review of this article - please see responds to your comments and suggestions below. We have amended the article to incorporate your corrections and suggestions were we pleased it was suitable to do so.

Summary
This was a SWAT within the MiQuit-3 trial. The purpose of the SWAT was to establish if the timing or personalisation of text messages increases completion of a questionnaire. Study design was a 2 x 2 factorial SWAT. Participants who reached their 36-week gestational follow-up were randomised to receive a personalised or non-personalised text message, either one week or one day prior to the telephone follow-up. Primary outcome was completion rate of questionnaire via telephone. Secondary outcomes included: completion rate via any method, time to completion, and number of reminders required. The authors concluded that timing of the text message did not appear to influence the retention of participants. The authors concluded that personalisation of a text message may be detrimental to retention; however, more SWATs should be undertaken in this field.
Thank you for this well written summary of the paper.

Major Comments
My first comment is on the title, the purpose of the study and the conclusions drawn. The title, correctly refers to the retention of participants in the smoking cessation pregnancy RCT. The purpose of the SWAT though is to evaluate completion of a questionnaire, not retention in the host trial, as claimed in the conclusion in the abstract. I think this is conflated throughout and the authors need to consider this carefully and amend their paper. In fact, the registered SWATs in the NI SWAT repository give the outcomes in both as questionnaire completion. So therefore the conclusions drawn in the abstract and in the discussion are not supported by the data.
The authors have amended the title of the paper, and changed references to retention within the paper to completion of questionnaires throughout. Alongside making the other suggested changes by the reviewer, we hope to have addressed any concerns in relation to the conclusions drawn.

Abstract
"There was evidence that a personalised text would result in fewer completions via telephone compared with a non-personalised text (adjusted OR 0.44, 95% CI 0.22–0.87, p=0.02)". This statement is confusing. The research question is personalised text message versus non-personalised text message. If using via telephone, then it should be also say non-personalised text via telephone.
The wording has been amended to attempt to make this statement clearer, and also reference the results for any data collection method.

Also, use text message, rather than text, throughout.
The term ‘text’ has been changed to ‘text message’ throughout the paper.

"Personalisation of a text message may be detrimental to retention". I don't think your results support a statement this strong. Firstly, this is not a trial with an adequate sample size to make this claim. Secondly, when you included all methods of receiving the questionnaire (I think this is correct interpretation but it is challenging to establish in the current version of the paper as the detail on all methods is unclear) you did not find that personalised or non-personalised texts mattered. At best you can say, personalisation of a text message appears to affect questionnaire completion via telephone.
This sentence has been altered to include ‘data was collected only via the telephone’, and now also references that the results do not hold for data collection via all methods. The results are correct as finding of this work, and the conclusion of the abstract references that more work is needed in this area so no other alterations have been made in regards to this comment.

Introduction
I like the succinct introduction.
Thank you for this comment.

Methods
The methods says "The SWATs are also registered with the Northern Ireland Hub for Trial Methodology Research SWAT Repository (SWATs 35 and 44; both registered December 2015). It's a little confusing to the reader. It looks like you're conducting two SWATs. I understand what you have taken both of these SWATS and conducted the two in a 2 x 2 factorial design. However, make this clear to the reader.
Additional words have been added to this paragraph to make it clear that this was a factorial SWAT involving these two SWATs.

Replace the phrase "carried out" with conducted throughout. I know the phrase carried out is used all the time in literature but it is not correct unless you are describing physically carrying an object.
The phase carried out has been replaced with conducted.

I would use the word women rather than participants, given that it is a trial of pregnant women.
We thank you for the suggestion – however, the authors have decided not to make this change. The protocol for the MiQuit-3 trial refers to those involved both as participants, and as women. The term participants is not incorrect, and as it is made clear that this work is being done in pregnant women in both the introduction and conclusions of this paper, we feel it does not need to be re-emphasised throughout the whole paper.

The phrase "all participants that had not yet had their 36-week gestational follow-up were eligible to participate" suggests that some people were more than 36 weeks pregnant. Is this the case? Otherwise you could simplify it and say women who were 36 weeks pregnant. It's important to be clear on the timing of the intervention given that this is one of your outcomes.
This sentence references those who were eligible to be included in the SWAT. Some participants in the host trial would have been more than 36 weeks pregnant (or have had their baby/babies) when the SWAT was implement, and thus they were not eligible to be included in the SWAT. The wording of this paragraph has been altered in relation to this comment, and the reviewer comment regarding the number of participants involved in the SWAT, hopefully this has made it clearer.

"Participants in MiQuit-3 were blind to their participation in this SWAT". Blinding is different to being unaware. Were they both blinded and unaware?
A correction has been made in regards to the unawareness and blinding of participants.

"The randomisation was undertaken by a statistician independent of the host trial, and of the staff involved in sending the texts". Explain how the statistician did the randomisation, e.g. computer generated. Also, explain how he communicated that randomisation to the researcher assigning the women to each group.
Details have been added to explain how the randomisation was generated (Stata) and how the randomisation sequence was used.

"Block randomisation, stratified by host trial allocation, and whether they had completed the previous follow-up; with varying block sizes of 4, 8, 12 and 16". This is not a sentence.
Corrections have been made to this sentence.

"A £5 voucher was given to all participants who completed a follow-up…". Was this part of the host trial or the SWAT?
Detail has been added to clarify that this is part of the host trial

"…additionally those who provided a saliva sample were given another £30 (£35 total)." Where does the saliva sample come into it? Is this part of the host trial? Explain in the paper.
Detail has been added to explain why a saliva sample was given, and that it, and associated voucher were part of the host trial.

How did you decide on how many to include for the SWAT. I accept that a sample size calculation is not required but a line in the paper on why you decided on (it appears to be 200) would be useful.
Further details have been added to explain why only 200 were included.

In the secondary outcome, explain what you mean by completed by any method.
Details have been added to explain what other methods of completion could be used.

"Time to response, defined as the number of days between the due date of the 36-week gestation follow-up and the date the questionnaire was recorded as complete". Are you certain all follow-up calls were made in 24 hours?
The date of each attempt to contact was recorded. However this outcome captures completion by any follow-up method.

Statistical Analysis
"The primary outcome was completion rate; defined as the proportion of the questionnaires completed over the telephone within the follow-up window (14 days)". However, you then go on to say that you used logistic regression to analyse this. Logistic regression is not suitable for four categories. I suspect what you mean is that you compared the completion rates across the two personalised/not personalised and again early/late. I can see you did this from the table. However, you need to articulate that in the text because it is currently confusing.
A logistic model can only deal with two levels of an outcome; it can handle multiple levels of a covariate. The outcome was completion (yes/no), and the swat allocations are included as covariates in the model. Each swat allocation is include as a separate covariate – as stated in the analysis section, providing two, two-level covariates. Thus the model does not include a four level variable, as you have suggested. Had swat allocation been included as a four level covariate, the model would have been able to do this – but it would not have provided an OR for each of the two SWAT interventions separately. This method is the appropriate method for analysing a binary outcome for a factorial SWAT, as such no changes have been made to the text.

In statistical analysis, a full stop after level and a new sentence for "As this is a factorial..."
Thank you for suggesting this correction – it has been made.

Full stop required in this sentence too. Also, I suspect the word "compared" in this sentence should read completed. - "Time to response (days between questionnaire due and complete) was analysed using a Cox Proportional Hazards regression, those who compared the questionnaire early had their time set to 0.1, those did not complete were censored at either last contact date or 120 days if not contacted, and those who withdrew in the course of the SWAT were set to their withdrawal date".
Corrections to this sentence have been made.

Again the following is not a sentence, "All models were repeated with the inclusion of an interaction term to explore any possible interactions between the two SWAT interventions; with a significance level of 5%."
Additional words have been added.

Results
"Additional participants were excluded from the analysis, where the covariates required for the model were not provided". What additional participants? Quantify and explain please. Why were the covariates not 'provided'? Explain please.
The text in this sentence has been altered to make this clearer.

"Three participants were not contacted due to difficulties/adverse events associated with their pregnancy but are still included in the analysis under ITT principles". Commas are required to make sense of the sentence.
Changes have been made to this sentence.

In your flow chart, what does primary refer to, and proximity? Add an explanation or use a term that explains a little better. Also in the flow chart, you say response rate but provide the number of participants. This is not a rate.
The flow diagram has been updated to include consistent terminology. The numbers in the flow diagram are referencing the numbers within each analysis.

In the primary outcome, continue with your phraseology - the 14-day follow-up window rather than "within 14 days of the due date".
This term has been made consistent throughout the paper.

I find the writing of the results very confusing. This sentence below suggests the outcome was completion rate via telephone versus completion rate via something else. "There was evidence to suggest a difference in completion rate via telephone adjusted OR 0.44 (0.22–0.87, p=0.02) which implies those who received the non-personalised text were more likely to complete the questionnaire when completing via the telephone". I'm wondering why you keep saying via telephone. It is particularly confusing when explaining the results. The last part above again says … those who received the non-personalised text were more likely to complete the questionnaire when completing via the telephone". It looks like the method of completion is the purpose of the study.
The main data collection method for MiQuit-3 was via the telephone. As such, within the SWAT protocol for this factorial SWAT, it was decided that the primary outcome would focus specifically on data collected via this method, and as a secondary explore data collected via any method. The results for the primary and associated secondary outcome have been rewritten slightly to clarify that the results are only including those who completed it via the telephone, as opposed to comparing those who did it via the telephone with something else.

When you use the phrase "were more likely to", you must give the details of the comparison, i.e., more likely than who?
Additional words have been added for clarification.

It is implied, but not adequately explained, that some women completed the questionnaire by some other means. It is not clear how this was handled in terms of the numbers analysed throughout the study and this needs to be explained.
For the secondary outcomes (time to completion, and number of attempts to contact) there have been no additional adjustments or changes to the analysis due to the method of data collection. The completion method is only relevant to the primary outcome, where the trial team detailed that their main method of data collection was via the telephone, and thus the primary outcome explores this. However, to explore the effectiveness of the interventions on any data collection method (as is more typically done in RCTs where a number of methods are utilised) a secondary outcome was to explore this via any method.

For your tables 3 and 4, add the number of women. Why does the left column say "response rate for all methods" when the primary outcome is defined as the proportion of the questionnaires completed over the telephone…"
Response rate, completion rate and retention rate are all frequently used synonymously within the SWAT world. As such the terminology within this paper has switched between the terms. In response to this comment the terminology has been made consistent.

The heading in the results section, "Response rates for all method", do you mean completion rates for all methods? A response rate is different.
Response rate has been changed to completion rate throughout.

If you hang your hat on statistically significant evidence, by quoting CIs and p-values, to establish if your SWAT was effective, or not, then the following has no place in your paper. "There is some, non statistically significant, evidence to suggest that there may be a difference in response rate for personalised versus non-personalised text reminders; adjusted OR 0.61 (95% CI 0.30–1.24, p=0.17), in favour of the non-personalised text messages". You cannot say there is non-statistically significant evidence and then support that statement with statistics! Remove this please.
This section has been reworded to reflect your suggestion.

The heading, "Number of attempts to contact required". Replace with, Number of attempts required to contact the women.
The authors have declined to make this change. In some instances the participant was not contacted, therefore it would be incorrect for the heading to referring the number required to contact them, when some were not contactable. Additionally, as explained in a previous comment we will not be changing participant to women.

What do you mean by the maximum number of calls, as stated here "The maximum number of calls was reached for 55 of the 174 participants…"?
This has been clarified both when describing the outcome, and in the results. The researcher tried to contact the participant a maximum of 6 times.

Contacts required is a new term introduced here " There was no evidence of a difference in number of contacts required". What do you mean by it?
This is referring the number of attempts to contact – a word had been omitted, and has been corrected.

"The average time to respond was 6.2 days (ranging from 5 days early to 103 days late)". Respond to what, the phone call or the text, or the questionnaire?
Terminology has been changed.

"This was similar between those who received a personalised text (8.2 days for early versus 7.1 days for late) and those who received the non-personalised text (4.9 days for early versus 4.7 days for late), but there is a slight difference between those who received personalised or non-personalised texts". If it is similar, how can there be a slight difference? What point are you making here?
The wording of the sentence has been changed to clarify this point. The sentence was meant to emphasis the difference in time between the personalised/non-personalised, whilst highlighting the similarities between early/late.

Include the number of participants in the MiQuit Trial earlier in the paper when discussing the 200 randomised for the SWAT.
The authors have included this figure (n=1002) earlier on in the paper.

Discussion
"It did show that there was some evidence that sending a non-personalised text message reminder would have a larger increase in response than sending personalised text messages did". This is misleading because it was not the case when all methods of questionnaire were included. Please amend the statement.
Additional words have been added to explain that these results are only valid for telephone completion

The final sentence of the discussion needs to be reviewed. What do you mean by overall effectiveness?
These words have been removed.

View more View less

Competing Interests

I have no competing interests to declare.

Back to all reports

Reviewer Report

32 Views

29 Jul 2021 | for Version 1

Phil J. Edwards, Department of Population Health, London School of Hygiene and Tropical Medicine, London, UK

32 Views Cite this report Responses(1)

Approved

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Evidence-based Data Collection

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (1)

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] 1. Bower P, Brueton V, Gamble C, et al.: Interventions to improve recruitment and retention in clinical trials: a survey and workshop to assess current practice and future priorities. Trials. 2014; 15(1)1: 399. PubMed Abstract | Publisher Full Text | Free Full Text

[2] 2. Adamson J, Hewitt CE, Torgerson DJ: Producing better evidence on how to improve randomised controlled trials. BMJ. 2015; 351: h4923. PubMed Abstract | Publisher Full Text

[3] 3. Treweek S, Bevan S, Bower P, et al.: Trial Forge Guidance 1: what is a Study Within A Trial (SWAT)? Trials. 2018; 19(1): 139. PubMed Abstract | Publisher Full Text | Free Full Text

[4] 4. Man MS, Tilbrook HE, Jayakody S, et al.: Electronic reminders did not improve postal questionnaire response rates or response times: a randomized controlled trial. J Clin Epidemiol. 2011; 64(9): 1001–1004. PubMed Abstract | Publisher Full Text

[5] 5. Ashby R, Turner G, Cross B, et al.: A randomized trial of electronic reminders showed a reduction in the time to respond to postal questionnaires. J Clin Epidemiol. 2011; 64(2): 208–212. PubMed Abstract | Publisher Full Text

[6] 6. Clark L, Ronaldson S, Dyson L, et al.: Electronic prompts significantly increase response rates to postal questionnaires: a randomized trial within a randomized trial and meta-analysis. J Clin Epidemiol. 2015; 68(12): 1446–1450. PubMed Abstract | Publisher Full Text

[7] 7. Naughton F, Riaz M, Sutton S: Response Parameters for SMS Text Message Assessments Among Pregnant and General Smokers Participating in SMS Cessation Trials. Nicotine Tob Res. 2016; 18(5): 1210–1214. PubMed Abstract | Publisher Full Text | Free Full Text

[8] 8. Sarathy PP, Kottam L, Parker A, et al.: Timing of electronic reminders did not improve trial participant questionnaire response: a randomized trial and meta-analyses. J Clin Epidemiol. 2020; 122: 70–77. PubMed Abstract | Publisher Full Text

[9] 9. Keding A, Brabyn S, MacPherson H, et al.: Text message reminders to improve questionnaire response rates in RCTs: findings from three randomised sub-studies. Trials. 2015; 16(Suppl. 2): 103. PubMed Abstract | Publisher Full Text

[10] 10. Mitchell AS, Cook L, Dean A, et al.: An embedded randomised controlled retention trial of personalised text messages compared to non-personalised text messages in an orthopaedic setting [version 1; peer review: 1 approved]. F1000Res. 2020; 9: 591. Publisher Full Text

[11] 11. Cochrane A, Welch C, Fairhurst C, et al.: An evaluation of a personalised text message reminder compared to a standard text message on postal questionnaire response rates: an embedded randomised controlled trial [version 1; peer review: 2 approved]. F1000Res. 2020; 9: 154. PubMed Abstract | Publisher Full Text | Free Full Text

[12] 12. Whitemore R, Leonardi-Bee J, Naughton F, et al.: Effectiveness and cost-effectiveness of a tailored text-message programme (MiQuit) for smoking cessation in pregnancy: study protocol for a randomised controlled trial (RCT) and meta-analysis. Trials. 2019; 20(1): 280. PubMed Abstract | Publisher Full Text | Free Full Text

[13] 13. StataCorp: Stata Statistical Software: Release 15. College Station, TX: StataCorp LLC, 2017. Reference Source

[14] 14. Bland JM, Altman DG: Multiple significance tests: the Bonferroni method. BMJ. 1995; 310(6973): 170. PubMed Abstract | Publisher Full Text | Free Full Text

[15] 15. Schoenfeld D: Partial residuals for the proportional hazards regression model. Biometrika. 1982; 69(1): 239–241. Publisher Full Text

[16] 16. RStudio Team: RStudio: Integrated Development for R. RStudio, PBC, Boston, MA, 2020. Reference Source

[17] 17. Coleman E: MiQuit_SWAT_Data.csv. figshare. Dataset. 2021. http://www.doi.org/10.6084/m9.figshare.14224319.v1

[18] 18. Coleman E: MiQuit SWAT CONSORT Checklist. figshare. Journal contribution. 2021. http://www.doi.org/10.6084/m9.figshare.14229647.v1

Pre-notification and personalisation of text messages to increase questionnaire completion in a smoking cessation pregnancy RCT: an embedded randomised factorial trial

Abstract

Keywords

Revised Amendments from Version 1

Introduction

Methods

Design

Participants and randomisation

Table 1. Details of the SWAT interventions and combinations.

Interventions

Outcomes

Secondary outcome measures

Statistical analysis

Results

Figure 1. Flow of participants through the SWAT.

Table 2. Baseline characteristics for participants by SWAT allocation.

Primary outcome

Table 3. Primary analysis results.

Table 4. Results for the secondary analyses.

Discussion

Data availability

Underlying data

Reporting guidelines

Acknowledgements

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated