Online algorithm for assignment of specimens to pooled or individual testing using risk models provides a practical way to increase testing capacity

Introduction

The coronavirus disease 2019 (COVID-19) pandemic has placed a demand for massive, rapid, and accurate diagnostic testing. A number of reports recommend pooled testing to help increase testing capacity. For example, a recent report provides specific estimates of cost savings in a pooled testing setting.¹ Biochemically, multiple reports demonstrate that RT-qPCR (reverse transcription quantitative real-time polymerase chain reaction) tests are amenable to pooled testing strategies.^2–4

Pooled testing has a long legacy of quantitative methodologies and some practical implementation successes. A review of pooled testing can be found at Wiley StatsRef: Statistics Reference Online.⁵ Within pooled testing methodologies, hierarchical two-step approach is the oldest and the simplest. The approach involves splitting subjects to be tested in equal size groups (pools) and testing each pool first. If a group test result is negative, so is the entire group. If the group is positive, each individual in the group is tested individually. Many variations of this approach have been proposed over the years. The most relevant set of techniques uses individual-level risks in conjunction with pooled testing to determine appropriate pool sizes for more efficient testing. These methods are typically concerned with optimization for given a collection of specimens (e.g. Ref. 6). Unfortunately, such algorithms do not always fit the workflow of pathology and laboratory medicine. It is desirable to be able to make online (at the time of encounter) decisions to assign a specimen to a pool at a point the specimen is first received in the laboratory. Many labs have limited ability to manipulate and rearrange the specimens multiple times to establish optimal pools. Therefore, an online algorithm that provides an immediate recommendation about better pooling strategies for the specimens is of practical importance for successful implementation of pooled testing strategies. For this reason, fixed size pooling, which cannot account for individual-level risk even when available, is by far the most popular approach in practice.

Informatics and artificial intelligence tools have been mobilized to help with the pandemic by allowing for infection risk prediction at the individual level. These risk predictions can be leveraged workflows to prioritize valuable clinical resources.⁷ In this report I demonstrate that a greedy online algorithm for specimen assignment based on individual risk predictions can increase COVID-19 testing capacity in a way suitable for providing pooling recommendations for specimens as they come in for testing. Figure 1 presents and overview of this approach.

Figure 1. Online greedy algorithm for pool assignment.

A. The algorithm relies on predictive model informed risk that a given specimen to be tested is positive, p_i. Population prevalence rates and arbitrary predictive models can be used based on the available predictors, such as basic demographics, risk factors, symptoms, natural language processing derived features from clinical notes, etc. The probabilities are used to group a stream of specimens to be tested into pools that will be tested together. Should a pool test negative, all of the specimens in the pool are recorded as negative, resulting in increased capacity for testing. For a positive pool, additional ascertainment of each individual specimen will be required for a final result. B. online algorithm makes a decision for any new specimen to either add it to a pool that is being formed or to end forming that pool and start a new one with this specimen. The decision is made based on expected capacity gain by using pooled testing calculated from the p_i of the specimen. EHR: Electronic health record.

Methods

Results

A basic logistic regression model provides moderate predictive accuracy of individual risk

The logistic regression model was meant to provide simplistic estimates of individual risk to demonstrate the feasibility and utility of the approach. The variables included in the data showed statistically significant differences across the training and testing data (Table 1), indicating the potential for suboptimal predictive performance.¹¹ The predictive model provides for a moderate predictive accuracy with 0.62 area under receiver operating characteristic curve estimate in testing data.

Table 1. Distribution of patient characteristics included in the predictive model.

Variable (%)	Overall (32,851)	Data subset (n)		Difference in training vs. testing data, χ2 test, P value (degrees of freedom)	Included in the final predictive model
		Training (25,714)	Testing (7,137)
Positive	4.68	4.71	4.55	0.59 (1)	Response
Repeat visit test	6.07	5.09	9.58	<10-16 (1)	Yes
Previously tested positive	0.88	0.85	0.99	0.27 (1)	Yes
Hospitalized	13	13	12	0.066 (1)	Yes
Tests ordered from hospitala	14	15	13	<10-5 (1)	Nob
Age group (years)					Yes
19–40	28	29	25	<10-12 (2)
40–70	53	52	54
>70	19	19	21
Age group (years) within subjects previously tested positive					Yesc
Total (n)	289	218	71
19 – 40	27	23	38	0.0025 (2)
40-70	43	48	25
>70	30	28	37

a The remaining tests have been ordered from ambulatory or community locations.

b Dropped from the model based on Akaike Information Criteria (AIC) in backward elimination step.

c Included as interaction term of age and indicator of a previous positive test.

Parameter estimates and other details of the model fit are shown in Figure 2. The model demonstrates that age, hospitalization status and whether the individual has been previously tested and/or tested positive are all good high level predictors of risk of positive test. The model is plagued by the imbalance of low and high risk patients, indicative of the relatively low population-level risk. Nonetheless, the predicted and empirical risk seem to correlate well, albeit with large variability in the high risk group.

Figure 2. A simple logistic regression model provides sensible estimates of positive rates.

A. R generalized linear regression function call and output following backwards stepwise elimination is illustrated. The features included in the final model were an indicator of whether this was a follow up test (Follow_up), an indicator of whether the individual had tested positive at any point previously (Previous_positive), age group (18–40, 40–70, >70), and indicator of whether the individual is hospitalized. An interaction of age and previous positivity is likewise retained in the model. B. Model performance evaluation included comparison of the empirical and predicted probabilities for groups of individuals with matching predictor values (follow up testing indicator, previous positivity, age, and hospitalization status). The model shows good concordance between the empirical and predicted risk in the lower risk range (inset), and large variability within the higher risk groups.

Discussion

The online nature of the presented algorithm allows for its easy implementation in many existing laboratory medicine workflows. For example, it may be used to provide pooling recommendations as the specimens are scanned upon receipt in the lab for testing. This feature is unique and important for a feasible and practical solution that fits the existing laboratory medicine workflow. Alternative approaches that optimize the pools globally for a collection of specimens (e.g. Ref. 6) may offer better performance in terms of capacity gains, but require additional manipulation of the specimens to form the pools, which may be feasible in some, but not all workflows. Implementations of these global optimizing approaches may be feasible when pool assignments can happen off-line, for example during transport of a batch of specimens from a collection site to a testing facility. With that respect, the online approach offers simplicity and appeal for laboratory management that is traded for potential global suboptimality.

The exact implementation of online pooling approach may need to meet specific operational constraints to be practical. For example, some laboratories may only be capable of testing in pools of fixed maximum size. These constraints can be naturally incorporated into modified online pooling algorithms. More sophisticated versions of the algorithms are easy to imagine as well. For example, a parallel fulfilment of multiple pools simultaneously can be accommodated in a straightforward extension.

Institutional implementation of any pooled testing approach that utilizes individual-level risks requires solutions to many informatics ecosystem problems. First, the models providing individual-level risk predictions need to be updated frequently to account for changes in prevalence by risk factors, and other contributors to model drift. In practice, a nightly model fit update may be feasible and necessary. Second, the model predictions need to be triggered at an appropriate time between specimen collection and the time it arrives into a laboratory for testing. Third, pooling recommendations have to be either pre-computed or involve only lightweight computations. In either case these recommendations have to be easily available in the laboratory information system. Combined these challenges point to likely requirement of high intra-institutional cooperation between laboratory medicine, analytics, data science, and informatics operations.

The testing capacity increases by pooled testing rely on the quality of the predictive models. When predictors are not available population prevalence rates can be input into the online algorithm, and the resulting two-step groupings will be equivalent to optimal pooling into pools of fixed size. In this paper, the evaluation of the algorithm involved clearly suboptimal sets of predictors and risk prediction approach (multivariable logistic regression). Nonetheless, the online algorithm provides an improvement over simple two-step pooling. Better predictive models will result in even larger capacity gains. Improved model predictivity could result from employing the data that is readily available or computable at the time of diagnostic specimen collection. For example, many of the data elements from the Health and Human Services guidance on laboratory reporting¹⁰ could be included. Other structured data, such as questionnaires collecting evidence and degree of exposure, and telehealth-derived variables can prove useful as well. Likewise, unstructured text data processed by natural language processing techniques can be useful.⁷ Further, more sophisticated machine learning and artificial intelligence approaches may be used to combine all of the available data sources for superior risk estimates in online pooling recommendations.

Conclusions

The results reported herein are immediately translatable to laboratory medicine operations. Even without sophisticated predictive models, given the current state of the pandemic the online pooling algorithm can double the COVID-19 testing capacity. Better risk prediction models may result in even better capacity improvements. In the longer term, similar strategies can be used for implementation of massive scale testing for other diseases.