Better than we thought? The diagnostic performance of an influenza point-of-care test in children, a Bayesian re-analysis

Introduction

Influenza is an infectious disease of global importance and is a target of many near-patient tests^1,2. These tests have been criticized for reported low sensitivity. This relatively poor ability to ‘rule out’ infection has been given as a reason to avoid their use in clinical practice, and instead develop better tests³. There are reasons to suspect some diagnostic-accuracy studies of point-of-care tests (POCTs) may have systematically underestimated sensitivity. If this is the case, the diagnostic accuracy of existing tests may be better than previously thought, with implications for clinical practice and test development.

Classic diagnostic-accuracy studies compare the performance of the index (new) test, with another reference (pre-existing) test, on samples from the same patients. Although rarely explicitly stated, the reference test is assumed to be an infallible ‘gold standard’. Under this assumption, whenever the index test and the reference test results differ, the index test is assumed to be wrong. This prevents the index test outperforming the reference, and may systematically underestimate test performance. Many diagnostic-accuracy studies of point-of-care tests for influenza have used these classical methods, raising the possibility that their diagnostic performance have been artificially suppressed⁴.

Established techniques for when a ‘gold standard’ is not available include: constructing a reference standard by multiple panels of tests, re-testing discrepant results, and statistical modelling⁵. Bayesian latent class models are one such statistical technique^6,7. Unlike many other methods, they offer an opportunity to retrospectively analyse existing data, providing a test has been compared to the same reference standard in more than one population⁶. As far as I can tell, this study is the first attempt at Bayesian re-analysis of point-of-care tests for influenza.

This paper aims to examine the extent to which published estimates of influenza point-of-care test accuracy are constrained by the infallible gold standard assumption, with a view to informing clinical practice, and future diagnostic-accuracy studies.

Methods

Published data were re-analysed using Bayesian latent class modelling and classical analysis. Data were extracted from two studies^8,9 comparing the same reference and index tests (reverse-transcriptase polymerase chain reaction (RT-PCR) vs. QuickVue® influenza A+B influenza), in two separate primary care populations.

Analyses were performed using the free online application Modelling for Infectious diseases CEntre (MICE; http://mice.tropmedres.ac/home.aspx), which has been described elsewhere, and runs parallel analyses of Bayesian latent class models and classical frequentist statistics for diagnostic test accuracy⁷. Data are input into MICE via a simple online portal, and results are stored online or emailed to the user.

MICE employs Markov Chain Monte Carlo (MCMC) simulations. These use the data provided to estimate all unknown parameters: the specificity and sensitivity of both reference and index tests, and the prevalence in the study population(s). The predicted combinations of test results are compared to the actual observed data, and the process is iterated, ideally until the estimates converge on the best fitting values for specificity, sensitivity and prevalence. MICE presents these results in the form of a table, with further graphs of the iterated estimates to allow the user to check convergence of MCMC chains, and Bayesian P values to allow the fit of the final model to the observed data to be assessed.

For this study the ‘two tests in one population’ model was selected and default values were used. Under the default settings non-informative priors (beta distribution 0.5, 0.5) are used to initiate the analysis, with the specificity of both tests constrained to above 40%. MCMC simulation used default initial values for diagnostic accuracy: (90% and 30% for prevalence, 90% and 70% for sensitivity, 90% and 99% for specificity). The analysis ran for 5000 iterations of pre-analysis adjustment (burn in), and 20,000 iterations.

Results

Data were extracted from two studies comparing a QuickVue® point-of-care test to Reverse-Transcriptase PCR in children with influenza-like-illness. Gordon et al⁸ studied 989 children in Nicaragua, Harnden et al⁹ included 157 children in England. Patient characteristics and study procedures were similar, with a low risk of bias (Table 1).

MCMC chains converged for all estimates. Model fit, assessed by Bayesian p value, was close between observed and expected values, with the exception of cases positive by RT-PCR, but negative by point-of-care test in the English study: 26 were predicted and 34 observed (Bayesian p value 0.081 – close to 0.5 indicates a good fit).

In both populations estimated prevalence was lower with Bayesian analysis: 34.0% (95% credible interval (CrI) 29.6-40.3) Vs 45.3% (95% CI 41.2-49.5) in Nicaragua and 18.6% (95% CrI 12.7-27.6) Vs 38.9% (95% confidence interval (CI) 31.3-47.0) in England (Table 2).

RT-PCR performance was assumed to be 100% under the classical model, and estimated by Bayesian modelling. Bayesian sensitivity and negative predictive values were close to the assumed values at 98.8% (95% CrI 94.3-100) and 99.3% (95% CrI 96.7-100), but specificity 80.1% (95% CrI 75.9-87.0) and positive predictive value 68.4% (95% CrI 62.0-81.0) were reduced (Table 2).

The performance estimates for the Quick-Vue® point-of-care test were markedly different under Bayesian assumptions. Sensitivity increased from 66.9% (95% CI 61.4-71.9) to 97.8% (95% CrI 82.1-100). Accordingly, the estimates for negative predictive value also increased, from 79.0% (95% CI 75.2-82.4) to 99.0% (95% CrI 90.7-100). Specificity was more similar between models (Classical 97.8%; 95% CI 95.7-98.9 Vs. Bayesian 98.5%; 95% CrI 96.5-100), as was estimated positive predictive value (Classical 96.0%; 95% CI 92.3-98.0 Vs. Bayesian 96.6%; 95% CrI 92.0-100) (Table 2).

Discussion

The classical results for QuickVue® presented here are typical. A systematic review of all point-of-care tests for influenza reported overall specificity of 98.2% (CI, 97.5% to 98.7%), but sensitivity only 62.3% (95% CI, 57.9% to 66.6%)². In contrast, Bayesian analysis estimated sensitivity of 97.8% (95% CrI 82.1-100), with a negative predictive value of 99.0% (95% CrI 90.7-100), suggesting a test of clinical importance, where there is little room for improvement in the ability to ‘rule out’ infection, apparently answering one of the major criticisms of point-of-care tests³.

The findings suggest false positives by the ‘infallible’ RT-PCR reference test. RT-PCR multiplies nucleic acids exponentially, making it both highly sensitive and vulnerable to false positives. Even the smallest amount of contamination can lead to a false-positive result. This is well recognised, so laboratories often use multiple negative controls¹⁰. Gordon et al did not mention negative controls; Harnden et al used water.

A weakness is that the original data were not collected specifically for this analysis; there may therefore be differences between in the study conduct by Gordon et al. and Harnden et al. other than the populations. Despite this, the studies appear to be remarkably similar. The imperfect fit of the data to an element of the Bayesian model should be balanced against classical modelling, where the fit of the data to the ‘perfect’ reference standard is rarely acknowledged, let alone assessed.

Overall, the findings are consistent with higher sensitivity than previously reported, and this underestimation can be attributed to the use of RT-PCR as a ‘gold standard’. These findings have implications for clinical practice, test development, and diagnostic-accuracy studies.

Data availability

Data used in this analysis are from the articles ‘Performance of an influenza rapid test in children in a primary healthcare setting in Nicaragua’⁸ by Gordon et al. (available at http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0007907) and ‘Near patient testing for influenza in children in primary care: comparison with laboratory test’⁹ by Harnden et al. (available at http://www.bmj.com/content/326/7387/480).