Is predictability of randomisation a historic concern: A survey of practice of recruiters to randomised controlled trials

Introduction

Randomisation is considered the gold standard for allocating participants to interventions within clinical trials as it eliminates potential biases during recruitment, the most notable being selection bias.¹ Selection bias occurs when there are systematic differences between treatment groups stemming from the recruitment process. To avoid this bias, trialists are concerned with ensuring that recruiters are unaware of and unable to predict upcoming allocations (allocation concealment).² Simple randomisation is inherently unpredictable, minimising the risk of selection bias. However, randomisation is commonly adapted to achieve balance with respect to demographic or clinical characteristics, which increases the risk of allocation sequences becoming predictable to recruiters and open to subversion.³

From the literature, there are instances where inadequate allocation concealment has led to bias within trials,^4–6 and research has shown the harmful effects of this bias on treatment effect estimates.^7,8 However, most of these case studies are dated, and research shows that some trialists involved in randomisation method selection were sceptical that this remains an issue in modern trials.⁹

Two previous studies,^10,11 have investigated recruiters’ behaviour when recruiting to trials and found evidence of prediction, however these studies were small and have not quantified the presence or extent of prediction attempts.

This shows that whilst clinical trialists may believe that sequence prediction is not common, the literature suggests there is evidence of subversion happening.

The aim of this study is to describe allocation sequence prediction among recruiters to UK clinical trials and to identify possible methods of prediction by site staff that researchers should consider when evaluating method predictability.

Methods

Results

Recruitment to the survey opened on 19^th August 2024 and closed on 5^th December 2024. Of the R&I departments that circulated the survey, responses had ceased after 123 participants had responded.

One responder declined consent, leaving 122 responders to be included in the analysis. Of those, 65 selected questionnaire one and received the questionnaire preamble which specifically referenced “subversion”, whilst the other 58 selected questionnaire two and received the alternative version.

Table 2 provides a breakdown of characteristics of recruiters, split by the questionnaire version that they received. Around half of respondents were research nurses, over 85% reported undertaking each task involved in patient recruitment and allocation, over 60% had 5 or more years experience, and 84% reported having worked on more than 5 trials. There were no notable differences in characteristics between participants completing each version of the questionnaire.

Table 3 shows the findings of the survey in relation to predicting subsequent allocations. Around 30% of the sample responded that, dependent on the information available, they felt they may be able to predict the next allocation in a sequence. Two respondents answered yes to this question. A higher proportion of those in the prediction group reported knowing a colleague who had attempted to guess the next allocation (53% vs 38%) however unexpectedly, figures were very similar for participants when asked if they had ever tried to guess the next allocation (40% vs 37%). Additionally, around 20% of respondents stated that they currently recruit to a trial where they or a colleague have attempted to guess the next allocation.

Table 4 contains an additional exploration of the relationship between answers to the questions “Have you ever tried to guess the next allocation?” and “Do you currently recruit to any trials where you or a colleague have attempted to guess the next allocation?” conditional on their answer to the question “Do you think you would be able to predict the next allocation in a randomisation sequence?”

There is no notable effect of the questionnaire version provided. Overall, respondents who thought they may be able to predict upcoming allocations reported higher proportions of ever (55% vs 31%) and currently (37% vs 13%), attempting to predict allocations. This suggests that while overall the rate of prediction seen currently in trials by participants or colleagues appears to be around 20%, this proportion is higher in those who believe that they may be able to correctly predict upcoming allocations.

Responders that stated they had ever attempted to guess the next allocation were asked what information they would use to make this guess. Responses were reviewed and the key themes identified are listed in Table 5. Forty-two responders stated that the previous allocation would factor into this choice. Other responses included using the characteristics of the participant, that the prediction was entirely a guess (based on no information) and a very small minority stated that elements of the study design (primarily potential side effects) could be used to identify previous allocations. Six responders specifically stated their prediction would be under the assumption that the treatments alternate. Seven respondents were clear in their statement that their prediction would be entirely a guess.

Discussion

Overall, while research shows that recruiters are aware of the need for allocation concealment and blinding, our survey has shown that this does not stop recruiters from attempting to predict the next allocation, with 20% of respondents stating that predictions are made either by themselves or a colleague in ongoing recruiting trials. This proportion is similar to that found previously in a smaller study.¹⁰ There is a suggestion that those who believe that it is possible to predict upcoming allocations could be more likely to attempt to do so, emphasising the importance of researchers designing randomised trials to guard against prediction as much as possible.

There is limited research looking at the effect of predictability on the final trial results, due to the difficulty in both identifying when subversion has happened and in measuring the effect this has had on outcomes.^12,13 A methodological review,¹⁴ identified a trend between allocation concealment adequacy and the effect size, though this was not statistically significant. Researchers have identified discrepancies in baseline characteristics that suggest subversion could be present.⁴ However, if there is not an obvious flaw in the allocation concealment, and if subversion does not affect baseline characteristics sufficiently to be detectable, subversion would likely go unnoticed.

A strength of this study is that the final sample size, which whilst smaller than the original target, is large compared to other surveys investigating prediction in RCTs,^10,11 which had sample sizes of 25 and 30 respectively. This allowed us to estimate the proportion of recruiters who attempted to guess the next allocation with a margin of error of 9. This was based on a conservative assumption that 50% of the respondents would report attempting to predict the next allocation, but the observed proportion is lower meaning the precision around the obtained proportions might not be markedly reduced despite the smaller sample size. Another strength is that the anonymous nature was likely to make responders feel safer to answer honestly, given previous research has shown that recruiters are aware that they should not be predicting allocations.^10,15

The respondents’ anonymity is also a limitation of this study as it means we cannot validate how representative the sample of survey respondents are compared to the population of recruiters to RCTs. The estimates given could be under-estimating the proportion of recruiters who attempted to guess allocations if those who attempted to guess allocations were less likely to respond to the survey. Equally, the estimate given could be over-estimating this proportion if those who attempt to guess allocations were more likely to respond to the survey. Another limitation is the self-randomisation of participants. Participants were asked to select questionnaire 1 or questionnaire two, with the assumption that there was unlikely to be a bias in this choice, however there is the potential this could have introduced selection bias. However, baseline characteristics of participants were collected to monitor the potential of this bias, and very little difference in groups was detected.

Whilst the aim of these recruiters is likely not to subvert randomisation sequences, this research highlights: 1) that researchers will attempt to make predictions about future allocations, although we do not know the motivation behind these guesses 2) Some recruiters are aware that patterns can arise in sequences which emphasises the continued need for researchers to guard sequences from becoming too predictable.

Declarations

The survey received ethical approval from the University of Nottingham (FMHS 173–0424) on 6^th August 2024, and R&I departments across England were contacted to circulate the survey on 19^th August 2024. It was agreed that the survey would close after reaching the target sample size, or after 3 months (6^th November 2024), however the survey remained open for another month due to delays in recruiting in the first month.