maxRatio improves the detection of samples with abnormal amplification profiles on QIAgen’s artus HIV-1 qPCR assay

Luigi Marongiu; Eric B. Shain; Marianna Martinelli; Matteo Pagliari; Heike Allgayer

doi:10.12688/f1000research.25738.3

Home Browse maxRatio improves the detection of samples with abnormal amplification...

ALL Metrics

Views

Downloads

Get PDF

Get XML

Export

▬

✚

Research Article

Revised

maxRatio improves the detection of samples with abnormal amplification profiles on QIAgen’s artus HIV-1 qPCR assay

[version 3; peer review: 2 approved]

Luigi Marongiu ^1,2, Eric B. Shain³, Marianna Martinelli⁴, Matteo Pagliari⁵, Heike Allgayer¹

Luigi Marongiu ^1,2, Eric B. Shain³, [...] Marianna Martinelli⁴, Matteo Pagliari⁵, Heike Allgayer¹

PUBLISHED 31 Aug 2021

Author details Author details

¹ Department of Experimental Surgery, University of Heidelberg, Medical Faculty in Mannheim, Mannheim, Baden-Wuttenberg, 68167, Germany
² Department of Biochemistry of Nutrition, University of Hohenheim, Garbenstr.30, 70599, Stuttgart, Germany
³ Grove Street Technology LLC, Grove Street Technology LLC, Glencoe, Illinois, 60022, USA
⁴ Department of Medicine and Surgery, University of Milano Bicocca, Monza, Italy, 20900, Italy
⁵ Department of Diagnostics in Animal Health, Istituto Zooprofilattico Sperimentale delle Venezie, Legnaro, Veneto, 35020, Italy

Luigi Marongiu
Roles: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Supervision, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Eric B. Shain
Roles: Conceptualization, Formal Analysis, Software, Validation

Marianna Martinelli
Roles: Formal Analysis, Validation

Matteo Pagliari
Roles: Formal Analysis, Validation

Heike Allgayer
Roles: Funding Acquisition, Project Administration, Supervision

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Pathogens gateway.

Abstract

Background: Accurate viral load (VL) determination is paramount to determine the efficacy of anti-HIV-1 therapy. The conventional method used, fit-point (FP), assumes an equal efficiency in the polymerase chain reaction (PCR) among samples that might not hold for low-input templates. An alternative approach, maxRatio, was introduced to compensate for inhibition in PCR.
Methods: Herein, we assessed whether maxRatio could improve VL quantification using 2,544 QIAgen artus HI virus-1 RT-PCR reactions. The assay’s standard dilutions were used to build external standard curves with either FP or maxRatio that re-calculated the VLs.
Results: FP and maxRatio were highly comparable (Pearson’s ρ=0.994, Cohen’s κ=0.885), and the combination of the two methods identified samples (n=41) with aberrant amplification profiles.
Conclusions: The combination of maxRatio and FP could improve the predictive value of the assay.

Keywords

HIV, qPCR, quantification, maxRatio, viral load.

Corresponding author: Luigi Marongiu

Competing interests: No competing interests were disclosed.

Grant information: This work was supported by the Alfried Krupp von Bohlen und Halbach Foundation, Essen; the Deutsche Krebshilfe, Bonn (70112168); the Deutsche Forschungsgemeinschaft (DFG, grant number AL 465/9-1); the HEiKA Initiative (Karlsruhe Institute of Technology/University of Heidelberg collaborative effort); Dr Hella-Buehler-Foundation, Heidelberg; Ingrid zu Solms Foundation, Frankfurt; the DKFZ-MOST Cooperation, Heidelberg (grant number CA149); the HIPO/POP-Initiative for Personalized Oncology, Heidelberg (H032 and H027).
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Copyright: © 2021 Marongiu L et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Marongiu L, Shain EB, Martinelli M et al. maxRatio improves the detection of samples with abnormal amplification profiles on QIAgen’s artus HIV-1 qPCR assay [version 3; peer review: 2 approved]. F1000Research 2021, 9:1030 (https://doi.org/10.12688/f1000research.25738.3) First published: 24 Aug 2020, 9:1030 (https://doi.org/10.12688/f1000research.25738.1) Latest published: 31 Aug 2021, 9:1030 (https://doi.org/10.12688/f1000research.25738.3)

Revised Amendments from Version 2

1. We re-arranged the acronyms in the paper to make the labelling clearer, specifically by clarifying the use of internal control (IC) in the method section.
2. We corrected a wrong proportion to 45.5%.
3. We encountered no issues in downloading fully functional comma-separated files. We recommend using the "original file format" option provided by Harvard Dataverse when downloading the datasets. We added a dictionary file explaining the fields' names used in the datasets associated with the present work.

See the authors' detailed response to the review by Joel Tellinghuisen

Introduction

Infection with HIV-1 accounts for a global prevalence of 38 million cases and a one million deaths yearly¹. An accurate viral load (VL), typically carried out by quantitative polymerase chain reaction (qPCR), is pivotal for addressing the efficacy of antiviral therapies². The threshold level of detection for the HIV-1 VL has been reported in the range 20–44 viral genomic copies per milliliter (c/mL)^3,4.

qPCR data are usually analyzed by the fit-point (FP) method, which assumes equal amplification efficiency between samples⁵. However, anomalies in the background fluorescence at low template input, can affect the quantification^6–8. An alternative method, maxRatio, was introduced to overcome these issues⁹. It has been reported that maxRatio conferred a marginal increase in assay accuracy over FP^10,11.

FP provides only a quantification cycle (Cq) value, which is then used to calculate VL. MaxRatio, instead, gives two parameters: one associated with the reaction’s efficiency (MR) and one equivalent to, albeit distinct from, Cq. These two parameters can be linked to bestow a quantitative cycle (FCNA) compensated for inhibition.

In the present work, we aimed to determine whether maxRatio could improve the determination of HIV-1 VL. We compared the quantification of HIV-1 VL computed by FP and maxRatio on a dataset generated with the QIAgen artus HIV assay, which has a reported limit of detection of 35.5 c/mL, and we showed that maxRatio could pinpoint samples with abnormal amplification profiles.

Methods

Dataset

The amplification data (see Underlying data¹²) obtained with the QIAgen artus HI Virus-1 RT-PCR kit were collected by the Public Health England Clinical Microbiology and Public Health Laboratory, Addenbrooke’s Hospital, Hills Road, Cambridge CB2 0QW, UK, during the year 2016. All data were anonymized before use. The reactions were subdivided into clinical samples, control dilutions (CDs), and non-template controls (NTCs). The CDs were based on known dilutions of in vitro transcribed HIV-1 RNA provided by the artus kit, corresponding to 405, 4,050, 40,500, and 405,000 c/mL. Each reaction also contained a primer set targeting an internal control (IC) to assess the proper extraction of the samples.

Data analysis

The FP method generated the Cq by registering the fractional cycle where the fluorescence passed the threshold of 0.2 units. The maxRatio transform of the amplification data and determination of the cut-offs were computed as previously described^9,10. Different operators visually inspected the reaction’s profiles and classified each reaction as either passed or failed. Using R v.3.6, linear models (standard curves, SC) were built on the CDs and applied to calculate the copy numbers according to the formula 10^{(x – b)/m} where x is the quantitative cycle (either Cq or FCNA), b and m are the intercept and slope, respectively, of the linear models¹³. Testing the difference between the expected and the calculated copy numbers was carried out with an unpaired t-test. VL correlation was obtained with the Pearson product-moment coefficient ρ¹⁴ and agreement between methods was tested with the Cohen’s κ¹⁵; both are reported with their 95% confidence interval (CI).

Results

The present dataset was derived from 122 individual artus HIV-1 runs, corresponding to 2,544 reactions (480 CDs, 122 NTCs and 1,931 clinical samples). The cut-offs obtained by expectation-maximization analysis were multiplied by 2.7 to generate the values used to filter the maxRatio data, as depicted in Figure 1.

Figure 1. Cut-offs for maxRatio.

Clinical samples (○) and CDs (◊) are plotted on the maxRatio plane; the numbers report the obtained cut-offs for MR (horizontal lines) and FCNA (vertical lines). Upper panel: MR/FCNA pairs for the HIV-1 target. To note how the CDs form four distinct clusters, corresponding to the different standard dilutions. Lower panel: MR/FCNA pairs for the IC target. To note that the data form a single cloud because the IC input was virtually the same for all reactions. The presence of two outlier groups at low and high FCNA values required to instantiate two cut-offs.

The CDs were used to build SCs (Figure 2) that quantified both the CDs (Table 1 and Figure 3) and the clinical samples (Figure 4). Overall, the VLs obtained with the two methods were very strongly correlated (ρ = 0.994, 95% CI: 0.993-0.994) and the agreement in the stratification of the reactions into reactive and non-reactive was noticeably robust (κ = 0.885, 95% CI: 0.863-0.907). Both methods identified 307 (15.9%) and 28 (1.5%) samples within and above the quantification range 405–405,000 c/mL (ρ = 0.988, 95% CI: 0.985-0.991 and ρ = 0.992, 95% CI: 0.982-0.996, respectively), and 1,571 (81.3%) below this range (ρ = 0.844, 95% CI: 0.829-0.858).

Figure 2. Linear models and CD quantification.

Development of the linear models. The Cq (○) and FCNA (●) were used to build SCs for FP (dashed line) and maxRatio (dotted line). The dots and bars represent the mean and standard deviation of the data, respectively. The characteristics of the models are reported.

Table 1. Comparison of copy numbers for the control dilutions.

The mean VL is reported together with the 95% CI (calculated), the difference between the calculated and the expected concentration (difference), and the result of the t-test (p-value). Statistical significance is represented by * and ** for values below 0.05 and 0.01, respectively. The copy numbers are given in c/mL.

Dilution	FP			maxRatio
Expected	Calculated	Difference	p-value	Calculated	Difference	p-value
405	440 (415-466)	35	0.006**	443 (417-469)	38	0.004**
4,050	4,014 (3,835-4,194)	36	0.694	4,028 (3,843-4,212)	22	0.811
40,500	40,777 (39,164-42,389)	277	0.735	40,039 (38,385-41,693)	–461	0.582
405,000	425,384 (408,765-442,002)	20,383	0.017*	430,673 (414,132-447,215)	25,673	0.003**

Figure 3. CD quantification.

The four panels represent the calculated copy numbers obtained using FP (○) or maxRatio (●) for each dilution. The dots and bars represent the mean and the 95% CI of the data, respectively.

Figure 4. Correlation of VL obtained by FP and maxRatio.

The external standard curves were used to calculate the VL for the clinical samples. The following thresholds are depicted: lower (solid lines) and upper (double solid lines) limits of the quantification range; limit of detection of HIV-1 diagnostic in general (dashed line) and artus HI Virus-1 assay in particular (dotted line).

FP quantified 22 reactions below the detection limit of 20 c/mL that were interpreted as non-reactive by maxRatio. Conversely, maxRatio identified 18 reactions below 20 c/mL while FP quantified them above this level. Visual inspection of the amplification profiles of the reactions failed by FP showed that 10 of them (45.5%) had a proper sigmoid shape for the IC signal that, however, was discarded by the FP (as exemplified in Figure. 5A). In contrast, the others had a low signal for either HIV-1 or IC recovered by maxRatio (Figure 5B). Conversely, 15 (83.3%) of the reactions failed by maxRatio showed either a low IC or HIV-1 input (Figure 6A), whereas the FCNA of the remaining reactions produced fractional VL that were rounded to 0 c/mL (Figure 6B).

Figure 5. Samples not quantified only by FP.

Example of raw amplification profiles for the samples whose FP did not provide a VL. The panels display the FP (left) the maxRatio (right) transforms of the amplification data. The solid line represents the HIV-1 template and the dashed line the IC template. (A) The majority of the samples had a proper IC profile but the output was undetermined. (B) The minority of the samples showed low target inputs.

Figure 6. Samples not quantified only by maxRatio.

Example of raw amplification profiles for the samples whose maxRatio did not provide a VL. The panels display the FP (left) the maxRatio (right) transforms of the amplification data. The solid line represents the HIV-1 template and the dashed line the IC template. (A) The majority of the samples had abnormal IC profiles that were overlooked by FP. (B) The minority of the samples showed proper amplification profiles, but the calculation provided fractional VLs that were rounded to zero.

Discussion

The purpose of the present work was to assess the potential benefits of maxRatio in determining HIV-1 VL given its inherent compensation of PCR inhibition. Contrary to our expectation, the SCs we built with either FP or maxRatio were virtually the same. Both methods gave VLs significantly divergent from the expected copy numbers at the lower and upper CDs, and maxRatio was, in general, more discrepant from the expected copy numbers than FP. Even concerning the samples’ quantification, the two methods produced essentially the same VLs.

The main difference between the two methods was in terms of sample’s reactivity. By accepting the reactions identified as non-reactive by FP, but reactive by maxRatio, there would have been 18 false-negative results. Conversely, 15 samples identified as non-reactive by maxRatio showed aberrant IC that raised quality control, rather than false-positive, issues overlooked by FP.

The current use of maxRatio is to confirm the reactivity determined by FP on the Abbott m2000rt platform. Our data supported this combination as the most effective approach for screening purposes. Samples in disagreement between FP and maxRatio would require further assessment that could reduce the workload involved in issuing the results and minimize the risk of providing false results.

The present work had some limitations. Firstly, the sample size was small. Since the correlation between the two methods was high (99.4% overall and 84.4% below the quantification limit of 405 c/mL), a more extensive sample set would provide only a marginal improvement in comparing the two algorithms. However, more samples will provide more instances of amplification profiles that are processed differently by FP and maxRatio. In the present study, 40 reactions out of 1931 samples (2.07%) showed discrepancies in quantification at the clinical threshold of 20 c/mL. Expanding such a subset of discrepant reactions to, say, 4000 could provide a database of profiles that can facilitate (perhaps using machine learning approaches) identifying the characteristics that led to the failure in quantification. Moreover, analysis of qPCR chemistries other than artus HI virus-1 might determine whether such characteristics are common to all reactions or peculiar to the kit used herein. Secondly, the CDs were prepared by diluting the control samples provided in the kit, but the actual concentration was not measured. Finally, we did not have access to the actual issued results; thus, we could not confirm the official VL values.

In conclusion, we compared FP and maxRatio in providing HIV-1 VL. Contrary to our expectations, maxRatio did not give a better quantification than FP, but combining the two methods could minimize issuing false results.

Data availability

Underlying data

Harvard Dataverse: artusHIV_amplificationData. https://doi.org/10.7910/DVN/0QQNPF

This project contains the following underlying data:

amplificationDataRaw.tab. (Raw amplification data, file is comma-delimited.)
viralLoads.tab. (Raw viral loads data, file is comma-delimited.)
dictionary.tab (explanation of the fields’ names used in the other files, file is comma-delimited.)

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Acknowledgements

We would like to thank Dr. Martin D. Curran (Public Health England, Clinical Microbiology and Public Health Laboratory, Addenbrooke's Hospital, Cambridge UK) for providing the amplification data.

We would also like to thank Prof. Clementina Elvezia Cocuzza (University of Milano Bicocca, Department of Medicine and Surgery, Monza, Italy) and Dr. Maria Serena Beato (‘Istituto Zooprofilattico Sperimentale delle Venezie’, Department of Diagnostics in Animal Health, Legnaro, Italy) for their moral support.

Faculty Opinions recommended

References

1. Wang H, Wolock TM, Carter A, et al.: Estimates of global, regional, and national incidence, prevalence, and mortality of HIV 1980–2015: the Global Burden of Disease Study 2015. Lancet HIV. 2016; 3(8): e361–87. PubMed Abstract | Publisher Full Text | Free Full Text
2. Marschner IC, Collier AC, Coombs RW, et al.: Use of Changes in Plasma Levels of Human Immunodeficiency Virus Type 1 RNA to Assess the Clinical Benefit of Antiretroviral Therapy. J Infect Dis. 1998; 177(1): 40–7. PubMed Abstract | Publisher Full Text
3. Adams P, Vancutsem E, Nicolaizeau C, et al.: Multicenter evaluation of the cobas® HIV-1 quantitative nucleic acid test for use on the cobas® 4800 system for the quantification of HIV-1 plasma viral load. J Clin Virol. 2019; 114: 43–9. PubMed Abstract | Publisher Full Text
4. Sizmann D, Glaubitz J, Simon CO, et al.: Improved HIV-1 RNA quantitation by COBAS® AmpliPrep/COBAS® TaqMan® HIV-1 Test, v2.0 using a novel dual-target approach. J Clin Virol. 2010; 49(1): 41–6. PubMed Abstract | Publisher Full Text
5. Bustin SA: Absolute quantification of mRNA using real-time reverse transcription polymerase chain reaction assays. J Mol Endocrinol. 2000; 25(2): 169–93. PubMed Abstract | Publisher Full Text
6. Bar T, Kubista M, Tichopad A: Validation of kinetics similarity in qPCR. Nucleic Acids Res. 2012; 40(4): 1395–406. PubMed Abstract | Publisher Full Text | Free Full Text
7. Chen P, Huang X: Comparison of Analytic Methods for Quantitative Real-Time Polymerase Chain Reaction Data. J Comput Biol. 2015; 22(11): 988–96. PubMed Abstract | Publisher Full Text | Free Full Text
8. Ramakers C, Ruijter JM, Lekanne Deprez RH, et al.: Assumption-free analysis of quantitative real-time polymerase chain reaction (PCR) data. Neurosci Lett. 2003; 339(1): 62–6. PubMed Abstract | Publisher Full Text
9. Shain EB, Clemens JM: A new method for robust quantitative and qualitative analysis of real-time PCR. Nucleic Acids Res. 2008; 36(14): e91. PubMed Abstract | Publisher Full Text | Free Full Text
10. Marongiu L, Shain E, Drumright L, et al.: Analysis of TaqMan Array Cards Data by an Assumption-Free Improvement of the maxRatio Algorithm Is More Accurate than the Cycle-Threshold Method. PLoS One. 2016; 11(11): e0165282. PubMed Abstract | Publisher Full Text | Free Full Text
11. Marongiu L, Shain E, Shain K, et al.: Filtering maxRatio results with machine learning models increases quantitative PCR accuracy over the fit point method. J Microbiol Methods. 2020; 169(1): 105803. PubMed Abstract | Publisher Full Text
12. Marongiu L: "artusHIV_amplificationData". Harvard Dataverse, V1, 2020. http://www.doi.org/10.7910/DVN/0QQNPF
13. Scott Adams P: Data Analysis and Reporting. In: Dorak T editor. Real-time PCR. 1st ed. New: Taylor & Francis; 2006; 39–62. Reference Source
14. Schober P, Schwarte LA: Correlation coefficients: Appropriate use and interpretation. Anesth Analg. 2018; 126(5): 1763–8. PubMed Abstract | Publisher Full Text
15. Viera AJ, Garrett JM: Understanding interobserver agreement: The kappa statistic. Fam Med. 2005; 37(5): 360–3. PubMed Abstract

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 24 Aug 2020

Author details Author details

Eric B. Shain
Roles: Conceptualization, Formal Analysis, Software, Validation

Marianna Martinelli
Roles: Formal Analysis, Validation

Matteo Pagliari
Roles: Formal Analysis, Validation

Heike Allgayer
Roles: Funding Acquisition, Project Administration, Supervision

Competing interests

No competing interests were disclosed.

Grant information

This work was supported by the Alfried Krupp von Bohlen und Halbach Foundation, Essen; the Deutsche Krebshilfe, Bonn (70112168); the Deutsche Forschungsgemeinschaft (DFG, grant number AL 465/9-1); the HEiKA Initiative (Karlsruhe Institute of Technology/University of Heidelberg collaborative effort); Dr Hella-Buehler-Foundation, Heidelberg; Ingrid zu Solms Foundation, Frankfurt; the DKFZ-MOST Cooperation, Heidelberg (grant number CA149); the HIPO/POP-Initiative for Personalized Oncology, Heidelberg (H032 and H027).
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Article Versions (3)

version 3

Revised

Published: 31 Aug 2021, 9:1030

https://doi.org/10.12688/f1000research.25738.3

version 2

Revised

Published: 29 Jun 2021, 9:1030

https://doi.org/10.12688/f1000research.25738.2

version 1

Published: 24 Aug 2020, 9:1030

https://doi.org/10.12688/f1000research.25738.1

© 2021 Marongiu L et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

SEE MORE DETAILS

CITE

how to cite this article

Marongiu L, Shain EB, Martinelli M et al. maxRatio improves the detection of samples with abnormal amplification profiles on QIAgen’s artus HIV-1 qPCR assay [version 3; peer review: 2 approved]. F1000Research 2021, 9:1030 (https://doi.org/10.12688/f1000research.25738.3)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Version 3

VERSION 3

PUBLISHED 31 Aug 2021

Revised

Views

Reviewer Report 14 Sep 2021

Joel Tellinghuisen, Department of Chemistry, Vanderbilt University, Nashville, TN, USA

Approved

https://doi.org/10.5256/f1000research.58856.r93097

With minor exceptions, I accept the author’s explanations and clarifications. Regarding the data, I wish I had saved what I downloaded initially but I did not. I recall clicking somewhere on the left side of the download page, and in two attempts, I got nothing sensible. I was able to get the data now, though they seem to be space-separated rather than csv; and the file would be much smaller if the fluorescence data didn’t have 13 figures after the decimal point!

My suggestion of just specifying a minimal plateau level was based on the observation that most of the questionable rxs never actually grow, and those that grow only a little for late cycles ultimately give 0 copies anyway. I did not intend this to mean anything about threshold levels.

I agree that my thinking was flawed when I suggested that the MR cycle would roughly equal the FDM: Maximum first difference indeed does not equal maximum ratio. My comment about instrument dependence was also wrong, as indeed the ratios will not change with scaling. On the other hand, two identical growth profiles superimposed on different baselines will give different MRs, and that would seem to make their FCNAs different. In this connection, is the conversion from FCN to FCNA used here as given in eq 2 in ref 9? That eqn is FCNA = FCN – log2(MR). When I apply that, for example, to the green curve in Fig. 5B (MR), I get FCNA ~ 36 – log2(0.021) = 41.6, which is much smaller than the given 46.525. Should that eqn perhaps have a factor of 2 with the log2 term?

Competing Interests: No competing interests were disclosed.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Respond or Comment

Views

Reviewer Report 01 Sep 2021

Emiliano Panieri, Department of Physiology and Pharmacology, Faculty of Pharmacy and Medicine, Sapienza University of Rome, Rome, Italy

Approved

https://doi.org/10.5256/f1000research.58856.r93096

I have ... Continue reading

CITE

Report a concern

Respond or Comment

Version 2

VERSION 2

PUBLISHED 29 Jun 2021

Revised

Views

Reviewer Report 13 Jul 2021

Joel Tellinghuisen, Department of Chemistry, Vanderbilt University, Nashville, TN, USA

Not Approved

https://doi.org/10.5256/f1000research.57778.r89311

The authors use a method apparently first described in their ref 9 (2008), which was coauthored by the 2^nd author of this work. This maxRatio method seems to be a useful tool for automated screening of very large numbers of qPCR reaction profiles for non-detects, though it is not clear to me that it is much better than, for example, just setting a low limit on acceptable plateau levels, perhaps combined with an upper limit on acceptable C_q values, beyond which the template number would round to zero. I did not see any comparisons of this sort either here or in ref 9. And I am not well enough acquainted with the various instruments to know what other possible tools there are for this kind of discrimination. However, it does appear to me that the title is not quite a truthful description of the work, since the authors note in Results that the MR method and the “fixed point” (FP) method with which they compare it gave nearly identical “stratification of the reactions into reactive and non-reactive.” And then early in the Discussion (and again in the final paragraph of the Conclusion), they acknowledge that the two methods gave virtually identical standard curves (SC). On the other hand, it is not clear to me how the discrimination was done in the FP method. This may have been by manual inspection (bottom of 1^st column, p 3), in which case the MR method would still be advantageous. (But, as noted, the “inspection” could also be automated.)

The near-agreement in the 2 standard curves is not surprising, because the FCNA is essentially a first-derivative maximum (FDM), shifted slightly from the use of finite differences and even more by the correction –log2(MR) (which is not mentioned here but is given in eq 2 in ref 9). Incidentally, this correction must only apply for a particular instrument and its fluorescence axis scale, because it clearly changes as the data are scaled. In any event, both C_t (used here for C_q in the FP method) and the FDM are legitimate C_q markers, so they should indeed give similar SCs. In fact, C_t is arguably the poorest C_q marker, for reasons Spiess and I have explained, most recently in Biomol. Detect. Quantif. 17 (2019) 100084.¹ Thus, the use of, for example, a relative threshold obtained from a fit model that includes the baseline function, should give even better FP results.

I wanted to check some of the authors’ results but unfortunately could not easily get the data from their referenced source (p 9 and ref 12). The Excel files seem not to be .csv, as labeled; the .tab files are text, but the application for opening them is not given (GitHub perhaps?). Nor is their content explained. I did attempt to reproduce their SCs in Figure 2, by digitizing the plotted data (WebPlotDigitizer, https://apps.automeris.io/wpd/). My results more or less confirmed theirs with one exception: My uncertainties for the back-calculated N₀ values for FP in Table 1 were ~10% smaller than theirs for the first 3 concentrations but ~40% larger for the highest. The relative errors here depend on only the structure of the data, so the last should also have been small by ~10% (this discrepancy from digitization limitations). The larger error makes the discrepancy < 2s for this case, removing one of the two values judged to be discordant.

Some specific problems:

Some labels are not defined: EM (1^st para of Results), SD (5^th line of Discussion). And IC appears in Fig. 1 but isn’t defined until several paragraphs after Fig. 1 is mentioned.
I don’t understand the introductory sentence in the last para of Results. And if the 10 which had sigmoidal appearance were of 22, that would be 45.4%, not 43.4%.
While it is clear that the reaction profiles in Figs 5B and 6A should not give results, I don’t understand why those in 5A do not.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

No
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

No
Are the conclusions drawn adequately supported by the results?

Partly

References

1. Tellinghuisen J, Spiess AN: qPCR data analysis: Better results through iconoclasm.Biomol Detect Quantif. 2019; 17: 100084 PubMed Abstract | Publisher Full Text

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: statistical data analysis

I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.

CITE

Report a concern

Author Response 19 Jul 2021

Luigi Marongiu, Department of Experimental Surgery, University of Heidelberg, Medical Faculty in Mannheim, Mannheim, 68167, Germany

19 Jul 2021

Author Response

Eric B. Shain is indeed the developer of maxRatio, as reported in Ref. 9. Shain and Marongiu collaborated to apply maxRatio in qPCR classification as either reactive or non-reactive in ... Continue reading Eric B. Shain is indeed the developer of maxRatio, as reported in Ref. 9. Shain and Marongiu collaborated to apply maxRatio in qPCR classification as either reactive or non-reactive in other works. The purpose of the present study was to assess whether maxRatio could provide more accurate copy numbers than the classical FP due to maxRatio's inherent capability to compensate for qPCR inhibition. Contrary to our expectations, maxRatio did not provide a significant improvement over FP. Hence we considered our results as "grey," and we embraced the F1000Research policy of welcoming "confirmatory and negative results, as well as null studies... irrespective of the perceived level of interest or novelty".

Our title reflects the fact that maxRatio helped identify reactions that were classified as non-reactive by FP. Unsurprisingly, the classification of the majority of the reactions was similar between the two methods. The focus of the analysis was instead on the few disagreed reactions, which are also those where an operator might have more difficulties inspect. About half of the reactions classified as non-reactive by FP had a proper amplification profile, suggesting false-negative results. Conversely, this ratio was only about a fifth using maxRatio. Despite the small sample set, we believe that when applied to the vast volumes characteristic of diagnostic laboratories, such a minor improvement could still be beneficial.

As reported in the Method section, the amplification profile of the individual reactions was visually inspected by a trained operator, and the operator classified each reaction as either reactive or non-reactive. Conversely, classification by maxRatio was done directly by mathematical methods. The evaluation of the amplification profiles generated by maxRatio was done post hoc purposedly to compare the two methods.
For the sake of brevity, we did not report the details of the FCNA's calculation. However, we referred to the original paper cited on maxRatio, as correctly noted by the reviewer, including the details of the expectation-maximization (EM) step reported in the first paragraph of the results section. We agree with the reviewer that the slope of the standard curves built with either FP or maxRatio should be similar. The focus of the work was not on the subtle differences between the standard curves based on the two methods but their applications.

The driving force of this and the previous works on the application of maxRatio done by Shain and Marongiu was to avoid the definition of a baseline threshold level. Firstly, the guidelines for the threshold value are too generic to give universal outcomes. Secondly, the threshold value might not be the same for each PCR assay. Thirdly, in our experience, the threshold level is set without much mathematical consideration. Conversely, maxRatio provides an objective way to quantify the reactions that dispense the threshold level altogether. As the reviewer remarked in his cited paper (Biomol Detect Quantif 2019;17:100084) the threshold level does not hold throughout the PCR while "most workers have taken [it] as a level near baseline where it is hoped that eq. 2 [associating the copy number to the efficiency of the reaction] remains valid". We agree with the conclusion of the author in his paper -- that is, that the absolute threshold is a poor choice -- because we believe that maxRatio allows overcoming its use. We disagree with the reviewer when he stated, "it is not clear to me that [maxRatio] is much better than, for example, just setting a low limit on acceptable plateau levels". If the reviewer meant to lower the threshold, there would be firstly the problem mentioned above of where to set the cut-off level, and secondly, there would be a trade-off between picking up more true positives at the expense of more false positives. This trade-off would require further determination of the impact of the analytical method on the sensitivity and specificity of the assay that (a) was beyond the scope of our work and (b) is skipped altogether by using maxRatio.

Furthermore, we disagree with the reviewer when he remarked, "FCNA is essentially a first-derivative maximum (FDM), shifted slightly from the use of finite differences and even more by the correction –log2(MR)". In fact, maxRatio is not the same as FDM. maxRatio does not use differences. maxRatio consistently identifies a cycle number earlier and closer to the exponential portion of the amplification curve. In addition, maxRatio provides a second measure (MR) which is highly useful in evaluating reactive-nonreactive status. FCNA helps correct the quantitation of PCR results where the response is partially inhibited. FDM cannot do this. We also disagree with his statement, "This correction must only apply for a particular instrument and its fluorescence axis scale because it clearly changes as the data are scaled." maxRatio works by calculating ratios. It is entirely unaffected by multiplicative scaling effects.

We loaded the datasets following the Harvard Dataverse's guidelines, essentially uploading a comma-delimited table. We quickly downloaded the original data simply by clicking on the "download" icon and selecting the download option. On downloading the data saved on Harvard Dataverse, the comma-separated values original file format)" can be opened directly with a spreadsheet application (in our case: LibreOffice Calc).
On downloading the alternative version "tab-delimited", Harvard Dataverse automatically adds the extension .tab, but the file's layout remains comma-delimited, implying that the spreadsheet applications do not appropriately recognize such file format. We recommend downloading the dataset only in its original format. Nevertheless, the procedure of data downloading is under the domain of Harvard Dataverse.

We believe that the names of the fields in the datasets were self-explicatory, but we have now included a dictionary to supply more information.

We assume that the discrepant back-calculated copy numbers might be due to the different approaches taken (actual calculation based on the quantification thresholds in our paper, digitalization applied by the reviewer).
Eric B. Shain is indeed the developer of maxRatio, as reported in Ref. 9. Shain and Marongiu collaborated to apply maxRatio in qPCR classification as either reactive or non-reactive in other works. The purpose of the present study was to assess whether maxRatio could provide more accurate copy numbers than the classical FP due to maxRatio's inherent capability to compensate for qPCR inhibition. Contrary to our expectations, maxRatio did not provide a significant improvement over FP. Hence we considered our results as "grey," and we embraced the F1000Research policy of welcoming "confirmatory and negative results, as well as null studies... irrespective of the perceived level of interest or novelty".

Our title reflects the fact that maxRatio helped identify reactions that were classified as non-reactive by FP. Unsurprisingly, the classification of the majority of the reactions was similar between the two methods. The focus of the analysis was instead on the few disagreed reactions, which are also those where an operator might have more difficulties inspect. About half of the reactions classified as non-reactive by FP had a proper amplification profile, suggesting false-negative results. Conversely, this ratio was only about a fifth using maxRatio. Despite the small sample set, we believe that when applied to the vast volumes characteristic of diagnostic laboratories, such a minor improvement could still be beneficial.

As reported in the Method section, the amplification profile of the individual reactions was visually inspected by a trained operator, and the operator classified each reaction as either reactive or non-reactive. Conversely, classification by maxRatio was done directly by mathematical methods. The evaluation of the amplification profiles generated by maxRatio was done post hoc purposedly to compare the two methods.
For the sake of brevity, we did not report the details of the FCNA's calculation. However, we referred to the original paper cited on maxRatio, as correctly noted by the reviewer, including the details of the expectation-maximization (EM) step reported in the first paragraph of the results section. We agree with the reviewer that the slope of the standard curves built with either FP or maxRatio should be similar. The focus of the work was not on the subtle differences between the standard curves based on the two methods but their applications.

The driving force of this and the previous works on the application of maxRatio done by Shain and Marongiu was to avoid the definition of a baseline threshold level. Firstly, the guidelines for the threshold value are too generic to give universal outcomes. Secondly, the threshold value might not be the same for each PCR assay. Thirdly, in our experience, the threshold level is set without much mathematical consideration. Conversely, maxRatio provides an objective way to quantify the reactions that dispense the threshold level altogether. As the reviewer remarked in his cited paper (Biomol Detect Quantif 2019;17:100084) the threshold level does not hold throughout the PCR while "most workers have taken [it] as a level near baseline where it is hoped that eq. 2 [associating the copy number to the efficiency of the reaction] remains valid". We agree with the conclusion of the author in his paper -- that is, that the absolute threshold is a poor choice -- because we believe that maxRatio allows overcoming its use. We disagree with the reviewer when he stated, "it is not clear to me that [maxRatio] is much better than, for example, just setting a low limit on acceptable plateau levels". If the reviewer meant to lower the threshold, there would be firstly the problem mentioned above of where to set the cut-off level, and secondly, there would be a trade-off between picking up more true positives at the expense of more false positives. This trade-off would require further determination of the impact of the analytical method on the sensitivity and specificity of the assay that (a) was beyond the scope of our work and (b) is skipped altogether by using maxRatio.

Furthermore, we disagree with the reviewer when he remarked, "FCNA is essentially a first-derivative maximum (FDM), shifted slightly from the use of finite differences and even more by the correction –log2(MR)". In fact, maxRatio is not the same as FDM. maxRatio does not use differences. maxRatio consistently identifies a cycle number earlier and closer to the exponential portion of the amplification curve. In addition, maxRatio provides a second measure (MR) which is highly useful in evaluating reactive-nonreactive status. FCNA helps correct the quantitation of PCR results where the response is partially inhibited. FDM cannot do this. We also disagree with his statement, "This correction must only apply for a particular instrument and its fluorescence axis scale because it clearly changes as the data are scaled." maxRatio works by calculating ratios. It is entirely unaffected by multiplicative scaling effects.

We loaded the datasets following the Harvard Dataverse's guidelines, essentially uploading a comma-delimited table. We quickly downloaded the original data simply by clicking on the "download" icon and selecting the download option. On downloading the data saved on Harvard Dataverse, the comma-separated values original file format)" can be opened directly with a spreadsheet application (in our case: LibreOffice Calc).
On downloading the alternative version "tab-delimited", Harvard Dataverse automatically adds the extension .tab, but the file's layout remains comma-delimited, implying that the spreadsheet applications do not appropriately recognize such file format. We recommend downloading the dataset only in its original format. Nevertheless, the procedure of data downloading is under the domain of Harvard Dataverse.

We believe that the names of the fields in the datasets were self-explicatory, but we have now included a dictionary to supply more information.

We assume that the discrepant back-calculated copy numbers might be due to the different approaches taken (actual calculation based on the quantification thresholds in our paper, digitalization applied by the reviewer).
Competing Interests: None Close
Report a concern
Respond or Comment

COMMENTS ON THIS REPORT

Author Response 19 Jul 2021

Luigi Marongiu, Department of Experimental Surgery, University of Heidelberg, Medical Faculty in Mannheim, Mannheim, 68167, Germany

19 Jul 2021

Author Response

Eric B. Shain is indeed the developer of maxRatio, as reported in Ref. 9. Shain and Marongiu collaborated to apply maxRatio in qPCR classification as either reactive or non-reactive in ... Continue reading Eric B. Shain is indeed the developer of maxRatio, as reported in Ref. 9. Shain and Marongiu collaborated to apply maxRatio in qPCR classification as either reactive or non-reactive in other works. The purpose of the present study was to assess whether maxRatio could provide more accurate copy numbers than the classical FP due to maxRatio's inherent capability to compensate for qPCR inhibition. Contrary to our expectations, maxRatio did not provide a significant improvement over FP. Hence we considered our results as "grey," and we embraced the F1000Research policy of welcoming "confirmatory and negative results, as well as null studies... irrespective of the perceived level of interest or novelty".

Our title reflects the fact that maxRatio helped identify reactions that were classified as non-reactive by FP. Unsurprisingly, the classification of the majority of the reactions was similar between the two methods. The focus of the analysis was instead on the few disagreed reactions, which are also those where an operator might have more difficulties inspect. About half of the reactions classified as non-reactive by FP had a proper amplification profile, suggesting false-negative results. Conversely, this ratio was only about a fifth using maxRatio. Despite the small sample set, we believe that when applied to the vast volumes characteristic of diagnostic laboratories, such a minor improvement could still be beneficial.

As reported in the Method section, the amplification profile of the individual reactions was visually inspected by a trained operator, and the operator classified each reaction as either reactive or non-reactive. Conversely, classification by maxRatio was done directly by mathematical methods. The evaluation of the amplification profiles generated by maxRatio was done post hoc purposedly to compare the two methods.
For the sake of brevity, we did not report the details of the FCNA's calculation. However, we referred to the original paper cited on maxRatio, as correctly noted by the reviewer, including the details of the expectation-maximization (EM) step reported in the first paragraph of the results section. We agree with the reviewer that the slope of the standard curves built with either FP or maxRatio should be similar. The focus of the work was not on the subtle differences between the standard curves based on the two methods but their applications.

The driving force of this and the previous works on the application of maxRatio done by Shain and Marongiu was to avoid the definition of a baseline threshold level. Firstly, the guidelines for the threshold value are too generic to give universal outcomes. Secondly, the threshold value might not be the same for each PCR assay. Thirdly, in our experience, the threshold level is set without much mathematical consideration. Conversely, maxRatio provides an objective way to quantify the reactions that dispense the threshold level altogether. As the reviewer remarked in his cited paper (Biomol Detect Quantif 2019;17:100084) the threshold level does not hold throughout the PCR while "most workers have taken [it] as a level near baseline where it is hoped that eq. 2 [associating the copy number to the efficiency of the reaction] remains valid". We agree with the conclusion of the author in his paper -- that is, that the absolute threshold is a poor choice -- because we believe that maxRatio allows overcoming its use. We disagree with the reviewer when he stated, "it is not clear to me that [maxRatio] is much better than, for example, just setting a low limit on acceptable plateau levels". If the reviewer meant to lower the threshold, there would be firstly the problem mentioned above of where to set the cut-off level, and secondly, there would be a trade-off between picking up more true positives at the expense of more false positives. This trade-off would require further determination of the impact of the analytical method on the sensitivity and specificity of the assay that (a) was beyond the scope of our work and (b) is skipped altogether by using maxRatio.

Furthermore, we disagree with the reviewer when he remarked, "FCNA is essentially a first-derivative maximum (FDM), shifted slightly from the use of finite differences and even more by the correction –log2(MR)". In fact, maxRatio is not the same as FDM. maxRatio does not use differences. maxRatio consistently identifies a cycle number earlier and closer to the exponential portion of the amplification curve. In addition, maxRatio provides a second measure (MR) which is highly useful in evaluating reactive-nonreactive status. FCNA helps correct the quantitation of PCR results where the response is partially inhibited. FDM cannot do this. We also disagree with his statement, "This correction must only apply for a particular instrument and its fluorescence axis scale because it clearly changes as the data are scaled." maxRatio works by calculating ratios. It is entirely unaffected by multiplicative scaling effects.

We loaded the datasets following the Harvard Dataverse's guidelines, essentially uploading a comma-delimited table. We quickly downloaded the original data simply by clicking on the "download" icon and selecting the download option. On downloading the data saved on Harvard Dataverse, the comma-separated values original file format)" can be opened directly with a spreadsheet application (in our case: LibreOffice Calc).
On downloading the alternative version "tab-delimited", Harvard Dataverse automatically adds the extension .tab, but the file's layout remains comma-delimited, implying that the spreadsheet applications do not appropriately recognize such file format. We recommend downloading the dataset only in its original format. Nevertheless, the procedure of data downloading is under the domain of Harvard Dataverse.

We believe that the names of the fields in the datasets were self-explicatory, but we have now included a dictionary to supply more information.

We assume that the discrepant back-calculated copy numbers might be due to the different approaches taken (actual calculation based on the quantification thresholds in our paper, digitalization applied by the reviewer).
Eric B. Shain is indeed the developer of maxRatio, as reported in Ref. 9. Shain and Marongiu collaborated to apply maxRatio in qPCR classification as either reactive or non-reactive in other works. The purpose of the present study was to assess whether maxRatio could provide more accurate copy numbers than the classical FP due to maxRatio's inherent capability to compensate for qPCR inhibition. Contrary to our expectations, maxRatio did not provide a significant improvement over FP. Hence we considered our results as "grey," and we embraced the F1000Research policy of welcoming "confirmatory and negative results, as well as null studies... irrespective of the perceived level of interest or novelty".

Our title reflects the fact that maxRatio helped identify reactions that were classified as non-reactive by FP. Unsurprisingly, the classification of the majority of the reactions was similar between the two methods. The focus of the analysis was instead on the few disagreed reactions, which are also those where an operator might have more difficulties inspect. About half of the reactions classified as non-reactive by FP had a proper amplification profile, suggesting false-negative results. Conversely, this ratio was only about a fifth using maxRatio. Despite the small sample set, we believe that when applied to the vast volumes characteristic of diagnostic laboratories, such a minor improvement could still be beneficial.

As reported in the Method section, the amplification profile of the individual reactions was visually inspected by a trained operator, and the operator classified each reaction as either reactive or non-reactive. Conversely, classification by maxRatio was done directly by mathematical methods. The evaluation of the amplification profiles generated by maxRatio was done post hoc purposedly to compare the two methods.
For the sake of brevity, we did not report the details of the FCNA's calculation. However, we referred to the original paper cited on maxRatio, as correctly noted by the reviewer, including the details of the expectation-maximization (EM) step reported in the first paragraph of the results section. We agree with the reviewer that the slope of the standard curves built with either FP or maxRatio should be similar. The focus of the work was not on the subtle differences between the standard curves based on the two methods but their applications.

The driving force of this and the previous works on the application of maxRatio done by Shain and Marongiu was to avoid the definition of a baseline threshold level. Firstly, the guidelines for the threshold value are too generic to give universal outcomes. Secondly, the threshold value might not be the same for each PCR assay. Thirdly, in our experience, the threshold level is set without much mathematical consideration. Conversely, maxRatio provides an objective way to quantify the reactions that dispense the threshold level altogether. As the reviewer remarked in his cited paper (Biomol Detect Quantif 2019;17:100084) the threshold level does not hold throughout the PCR while "most workers have taken [it] as a level near baseline where it is hoped that eq. 2 [associating the copy number to the efficiency of the reaction] remains valid". We agree with the conclusion of the author in his paper -- that is, that the absolute threshold is a poor choice -- because we believe that maxRatio allows overcoming its use. We disagree with the reviewer when he stated, "it is not clear to me that [maxRatio] is much better than, for example, just setting a low limit on acceptable plateau levels". If the reviewer meant to lower the threshold, there would be firstly the problem mentioned above of where to set the cut-off level, and secondly, there would be a trade-off between picking up more true positives at the expense of more false positives. This trade-off would require further determination of the impact of the analytical method on the sensitivity and specificity of the assay that (a) was beyond the scope of our work and (b) is skipped altogether by using maxRatio.

Furthermore, we disagree with the reviewer when he remarked, "FCNA is essentially a first-derivative maximum (FDM), shifted slightly from the use of finite differences and even more by the correction –log2(MR)". In fact, maxRatio is not the same as FDM. maxRatio does not use differences. maxRatio consistently identifies a cycle number earlier and closer to the exponential portion of the amplification curve. In addition, maxRatio provides a second measure (MR) which is highly useful in evaluating reactive-nonreactive status. FCNA helps correct the quantitation of PCR results where the response is partially inhibited. FDM cannot do this. We also disagree with his statement, "This correction must only apply for a particular instrument and its fluorescence axis scale because it clearly changes as the data are scaled." maxRatio works by calculating ratios. It is entirely unaffected by multiplicative scaling effects.

We loaded the datasets following the Harvard Dataverse's guidelines, essentially uploading a comma-delimited table. We quickly downloaded the original data simply by clicking on the "download" icon and selecting the download option. On downloading the data saved on Harvard Dataverse, the comma-separated values original file format)" can be opened directly with a spreadsheet application (in our case: LibreOffice Calc).
On downloading the alternative version "tab-delimited", Harvard Dataverse automatically adds the extension .tab, but the file's layout remains comma-delimited, implying that the spreadsheet applications do not appropriately recognize such file format. We recommend downloading the dataset only in its original format. Nevertheless, the procedure of data downloading is under the domain of Harvard Dataverse.

We believe that the names of the fields in the datasets were self-explicatory, but we have now included a dictionary to supply more information.

We assume that the discrepant back-calculated copy numbers might be due to the different approaches taken (actual calculation based on the quantification thresholds in our paper, digitalization applied by the reviewer).
Competing Interests: None Close
Report a concern

Views

Reviewer Report 29 Jun 2021

Emiliano Panieri, Department of Physiology and Pharmacology, Faculty of Pharmacy and Medicine, Sapienza University of Rome, Rome, Italy

Approved

https://doi.org/10.5256/f1000research.57778.r88542

The authors properly addressed all my previous requests. I am ... Continue reading

CITE

Report a concern

Respond or Comment

Version 1

VERSION 1

PUBLISHED 24 Aug 2020

Views

Reviewer Report 10 Jun 2021

Emiliano Panieri, Department of Physiology and Pharmacology, Faculty of Pharmacy and Medicine, Sapienza University of Rome, Rome, Italy

Approved

https://doi.org/10.5256/f1000research.28406.r83772

General comment:
The paper from Marongiu et al. presents an alternative approach to the conventional use of fit-point (FP) or MaxRatio methods, which can be used to accurately determine the viral load of HIV-1 patients under experimental conditions wherein the amplification efficiency might vary across the samples, as in the case of low-copy number of viral RNA or in presence of PCR inhibitors. This is a clinically relevant issue since the VLs widely differ among HIV-1 infected individuals, potentially leading to issues of false-positive/negative results when the viremia is very low or the efficiency of PCR reactions is not uniform. It is therefore of utmost importance to improve not only the analytical sensitivity of qPCR methods but also their predictive value to avoid that a substantial fraction of HIV-1 patients will be incorrectly diagnosed, causing a negative impact on their prognosis and treatment monitoring. In this regard, the approach proposed by the authors, based on both FP and MaxRatio is straightforward and when correctly applied can represent a cost-effective alternative to the time-consuming and labor-intensive work required to validate false positive and false negative results. In general, the manuscript is well written, concise and presented in a clear, scientifically-sound way. The data are quite exhaustive, corroborated by appropriate statistical analysis and firmly support the authors’ conclusions.

However, some minor changes will be required to improve the MS quality, which are listed below:

The authors correctly indicate that the small-size of the tested samples represents a limitation to their study. Can also the authors discuss whether increasing the sample size is expected to proportionally increase the accuracy of the proposed method? In other words, in the authors’ opinion, is the number of tested samples expected to influence the predictive value of the FP + MaxRatio use? It might be useful to further elaborate this aspect in the discussion to help the readers to gain insights on this limitation.
The authors should indicate which kind of t-test was used to assess the statistical significance of the data contained in Table 1 (i.e paired or unpaired, with or without other corrections such as Welch’s or Bonferroni’s) and which software was used to perform the statistical analysis.
In the Results paragraph it is suggested to mitigate the following statement: ”…the agreement in the stratification of the reactions into reactive and non-reactive was almost perfect (κ = 0.885, 95% CI:0.863-0.907). ”. Instead, “noticeably robust” can be a good alternative.
Few lines below it is suggested to slightly rearrange the sentence “…the FP identified 22 reactions above the limit of detection of 20 c/mL, while maxRatio failed the quantification” (i.e. while Max Ratio was unsuccessful).
Few lines below it is suggested to substitute “sigmoid outlook” with “sigmoidal shape”.
Few lines below it is suggested to change “…of the remainder of the reactions” into “…of the remaining reactions”.
In the Figure 1 legend, the authors describe a panel A (MR/FCNA pairs for the HIV-1 target) and a panel B (MR/FCNA pairs for the IC) which are however absent in the Figure 1. Please add the correct indication or either refer to the panels as upper and lower.
In the Figure 5-6 legend, it should be corrected “the panels displays” with “the panels display”.
In the last three lines of the Discussion section please rephrase the sentence: “…maxRatio did not give a better quantification than FP, but combining the two methods could minimize issuing false results” into “maxRatio did not result in a better quantification than FP but when combined together these two methods could…”

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Oncology, cellular and molecular biology

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

CITE

Report a concern

Respond or Comment

Comments on this article Comments (0)

Version 3

VERSION 3 PUBLISHED 24 Aug 2020

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2
Version 3 (revision) 31 Aug 21	read	read
Version 2 (revision) 29 Jun 21	read	read
Version 1 24 Aug 20	read

Emiliano Panieri, Sapienza University of Rome, Rome, Italy
Joel Tellinghuisen, Vanderbilt University, Nashville, USA

Comments on this article

All Comments(0)

Add a comment

Browse by related subjects

Back to all reports

Reviewer Report

4 Views

14 Sep 2021 | for Version 3

Joel Tellinghuisen, Department of Chemistry, Vanderbilt University, Nashville, TN, USA

4 Views Cite this report Responses(0)

Approved

Competing Interests

No competing interests were disclosed.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

4 Views

01 Sep 2021 | for Version 3

Emiliano Panieri, Department of Physiology and Pharmacology, Faculty of Pharmacy and Medicine, Sapienza University of Rome, Rome, Italy

4 Views Cite this report Responses(0)

Approved

I have no further comments

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Oncology, cellular and molecular biology

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

21 Views

13 Jul 2021 | for Version 2

Joel Tellinghuisen, Department of Chemistry, Vanderbilt University, Nashville, TN, USA

21 Views Cite this report Responses(1)

Not Approved

Some labels are not defined: EM (1^st para of Results), SD (5^th line of Discussion). And IC appears in Fig. 1 but isn’t defined until several paragraphs after Fig. 1 is mentioned.
I don’t understand the introductory sentence in the last para of Results. And if the 10 which had sigmoidal appearance were of 22, that would be 45.4%, not 43.4%.
While it is clear that the reaction profiles in Figs 5B and 6A should not give results, I don’t understand why those in 5A do not.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

No
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

No
Are the conclusions drawn adequately supported by the results?

Partly

References

1. Tellinghuisen J, Spiess AN: qPCR data analysis: Better results through iconoclasm.Biomol Detect Quantif. 2019; 17: 100084 PubMed Abstract | Publisher Full Text

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

statistical data analysis

Respond to this report

Responses (1)

Author Response

19 Jul 2021

Luigi Marongiu, Department of Experimental Surgery, University of Heidelberg, Medical Faculty in Mannheim, Mannheim, 68167, Germany

Eric B. Shain is indeed the developer of maxRatio, as reported in Ref. 9. Shain and Marongiu collaborated to apply maxRatio in qPCR classification as either reactive or non-reactive in other works. The purpose of the present study was to assess whether maxRatio could provide more accurate copy numbers than the classical FP due to maxRatio's inherent capability to compensate for qPCR inhibition. Contrary to our expectations, maxRatio did not provide a significant improvement over FP. Hence we considered our results as "grey," and we embraced the F1000Research policy of welcoming "confirmatory and negative results, as well as null studies... irrespective of the perceived level of interest or novelty".

Our title reflects the fact that maxRatio helped identify reactions that were classified as non-reactive by FP. Unsurprisingly, the classification of the majority of the reactions was similar between the two methods. The focus of the analysis was instead on the few disagreed reactions, which are also those where an operator might have more difficulties inspect. About half of the reactions classified as non-reactive by FP had a proper amplification profile, suggesting false-negative results. Conversely, this ratio was only about a fifth using maxRatio. Despite the small sample set, we believe that when applied to the vast volumes characteristic of diagnostic laboratories, such a minor improvement could still be beneficial.

As reported in the Method section, the amplification profile of the individual reactions was visually inspected by a trained operator, and the operator classified each reaction as either reactive or non-reactive. Conversely, classification by maxRatio was done directly by mathematical methods. The evaluation of the amplification profiles generated by maxRatio was done post hoc purposedly to compare the two methods.
For the sake of brevity, we did not report the details of the FCNA's calculation. However, we referred to the original paper cited on maxRatio, as correctly noted by the reviewer, including the details of the expectation-maximization (EM) step reported in the first paragraph of the results section. We agree with the reviewer that the slope of the standard curves built with either FP or maxRatio should be similar. The focus of the work was not on the subtle differences between the standard curves based on the two methods but their applications.

The driving force of this and the previous works on the application of maxRatio done by Shain and Marongiu was to avoid the definition of a baseline threshold level. Firstly, the guidelines for the threshold value are too generic to give universal outcomes. Secondly, the threshold value might not be the same for each PCR assay. Thirdly, in our experience, the threshold level is set without much mathematical consideration. Conversely, maxRatio provides an objective way to quantify the reactions that dispense the threshold level altogether. As the reviewer remarked in his cited paper (Biomol Detect Quantif 2019;17:100084) the threshold level does not hold throughout the PCR while "most workers have taken [it] as a level near baseline where it is hoped that eq. 2 [associating the copy number to the efficiency of the reaction] remains valid". We agree with the conclusion of the author in his paper -- that is, that the absolute threshold is a poor choice -- because we believe that maxRatio allows overcoming its use. We disagree with the reviewer when he stated, "it is not clear to me that [maxRatio] is much better than, for example, just setting a low limit on acceptable plateau levels". If the reviewer meant to lower the threshold, there would be firstly the problem mentioned above of where to set the cut-off level, and secondly, there would be a trade-off between picking up more true positives at the expense of more false positives. This trade-off would require further determination of the impact of the analytical method on the sensitivity and specificity of the assay that (a) was beyond the scope of our work and (b) is skipped altogether by using maxRatio.

Furthermore, we disagree with the reviewer when he remarked, "FCNA is essentially a first-derivative maximum (FDM), shifted slightly from the use of finite differences and even more by the correction –log2(MR)". In fact, maxRatio is not the same as FDM. maxRatio does not use differences. maxRatio consistently identifies a cycle number earlier and closer to the exponential portion of the amplification curve. In addition, maxRatio provides a second measure (MR) which is highly useful in evaluating reactive-nonreactive status. FCNA helps correct the quantitation of PCR results where the response is partially inhibited. FDM cannot do this. We also disagree with his statement, "This correction must only apply for a particular instrument and its fluorescence axis scale because it clearly changes as the data are scaled." maxRatio works by calculating ratios. It is entirely unaffected by multiplicative scaling effects.

We loaded the datasets following the Harvard Dataverse's guidelines, essentially uploading a comma-delimited table. We quickly downloaded the original data simply by clicking on the "download" icon and selecting the download option. On downloading the data saved on Harvard Dataverse, the comma-separated values original file format)" can be opened directly with a spreadsheet application (in our case: LibreOffice Calc).
On downloading the alternative version "tab-delimited", Harvard Dataverse automatically adds the extension .tab, but the file's layout remains comma-delimited, implying that the spreadsheet applications do not appropriately recognize such file format. We recommend downloading the dataset only in its original format. Nevertheless, the procedure of data downloading is under the domain of Harvard Dataverse.

We believe that the names of the fields in the datasets were self-explicatory, but we have now included a dictionary to supply more information.

We assume that the discrepant back-calculated copy numbers might be due to the different approaches taken (actual calculation based on the quantification thresholds in our paper, digitalization applied by the reviewer).

View more View less

Competing Interests

None

Back to all reports

Reviewer Report

4 Views

29 Jun 2021 | for Version 2

Emiliano Panieri, Department of Physiology and Pharmacology, Faculty of Pharmacy and Medicine, Sapienza University of Rome, Rome, Italy

4 Views Cite this report Responses(0)

Approved

The authors properly addressed all my previous requests. I am satisfied with their reply and I have no additional issues.

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Oncology, cellular and molecular biology

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

8 Views

10 Jun 2021 | for Version 1

Emiliano Panieri, Department of Physiology and Pharmacology, Faculty of Pharmacy and Medicine, Sapienza University of Rome, Rome, Italy

8 Views Cite this report Responses(0)

Approved

The authors correctly indicate that the small-size of the tested samples represents a limitation to their study. Can also the authors discuss whether increasing the sample size is expected to proportionally increase the accuracy of the proposed method? In other words, in the authors’ opinion, is the number of tested samples expected to influence the predictive value of the FP + MaxRatio use? It might be useful to further elaborate this aspect in the discussion to help the readers to gain insights on this limitation.
The authors should indicate which kind of t-test was used to assess the statistical significance of the data contained in Table 1 (i.e paired or unpaired, with or without other corrections such as Welch’s or Bonferroni’s) and which software was used to perform the statistical analysis.
In the Results paragraph it is suggested to mitigate the following statement: ”…the agreement in the stratification of the reactions into reactive and non-reactive was almost perfect (κ = 0.885, 95% CI:0.863-0.907). ”. Instead, “noticeably robust” can be a good alternative.
Few lines below it is suggested to slightly rearrange the sentence “…the FP identified 22 reactions above the limit of detection of 20 c/mL, while maxRatio failed the quantification” (i.e. while Max Ratio was unsuccessful).
Few lines below it is suggested to substitute “sigmoid outlook” with “sigmoidal shape”.
Few lines below it is suggested to change “…of the remainder of the reactions” into “…of the remaining reactions”.
In the Figure 1 legend, the authors describe a panel A (MR/FCNA pairs for the HIV-1 target) and a panel B (MR/FCNA pairs for the IC) which are however absent in the Figure 1. Please add the correct indication or either refer to the panels as upper and lower.
In the Figure 5-6 legend, it should be corrected “the panels displays” with “the panels display”.
In the last three lines of the Discussion section please rephrase the sentence: “…maxRatio did not give a better quantification than FP, but combining the two methods could minimize issuing false results” into “maxRatio did not result in a better quantification than FP but when combined together these two methods could…”

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Oncology, cellular and molecular biology

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Respond to this report

Responses (0)

Alongside their report, reviewers assign a status to the article:

Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions

[1] 1. Wang H, Wolock TM, Carter A, et al.: Estimates of global, regional, and national incidence, prevalence, and mortality of HIV 1980–2015: the Global Burden of Disease Study 2015. Lancet HIV. 2016; 3(8): e361–87. PubMed Abstract | Publisher Full Text | Free Full Text

[2] 2. Marschner IC, Collier AC, Coombs RW, et al.: Use of Changes in Plasma Levels of Human Immunodeficiency Virus Type 1 RNA to Assess the Clinical Benefit of Antiretroviral Therapy. J Infect Dis. 1998; 177(1): 40–7. PubMed Abstract | Publisher Full Text

[3] 3. Adams P, Vancutsem E, Nicolaizeau C, et al.: Multicenter evaluation of the cobas® HIV-1 quantitative nucleic acid test for use on the cobas® 4800 system for the quantification of HIV-1 plasma viral load. J Clin Virol. 2019; 114: 43–9. PubMed Abstract | Publisher Full Text

[4] 4. Sizmann D, Glaubitz J, Simon CO, et al.: Improved HIV-1 RNA quantitation by COBAS® AmpliPrep/COBAS® TaqMan® HIV-1 Test, v2.0 using a novel dual-target approach. J Clin Virol. 2010; 49(1): 41–6. PubMed Abstract | Publisher Full Text

[5] 5. Bustin SA: Absolute quantification of mRNA using real-time reverse transcription polymerase chain reaction assays. J Mol Endocrinol. 2000; 25(2): 169–93. PubMed Abstract | Publisher Full Text

[6] 6. Bar T, Kubista M, Tichopad A: Validation of kinetics similarity in qPCR. Nucleic Acids Res. 2012; 40(4): 1395–406. PubMed Abstract | Publisher Full Text | Free Full Text

[7] 7. Chen P, Huang X: Comparison of Analytic Methods for Quantitative Real-Time Polymerase Chain Reaction Data. J Comput Biol. 2015; 22(11): 988–96. PubMed Abstract | Publisher Full Text | Free Full Text

[8] 8. Ramakers C, Ruijter JM, Lekanne Deprez RH, et al.: Assumption-free analysis of quantitative real-time polymerase chain reaction (PCR) data. Neurosci Lett. 2003; 339(1): 62–6. PubMed Abstract | Publisher Full Text

[9] 9. Shain EB, Clemens JM: A new method for robust quantitative and qualitative analysis of real-time PCR. Nucleic Acids Res. 2008; 36(14): e91. PubMed Abstract | Publisher Full Text | Free Full Text

[10] 10. Marongiu L, Shain E, Drumright L, et al.: Analysis of TaqMan Array Cards Data by an Assumption-Free Improvement of the maxRatio Algorithm Is More Accurate than the Cycle-Threshold Method. PLoS One. 2016; 11(11): e0165282. PubMed Abstract | Publisher Full Text | Free Full Text

[11] 11. Marongiu L, Shain E, Shain K, et al.: Filtering maxRatio results with machine learning models increases quantitative PCR accuracy over the fit point method. J Microbiol Methods. 2020; 169(1): 105803. PubMed Abstract | Publisher Full Text

[12] 12. Marongiu L: "artusHIV_amplificationData". Harvard Dataverse, V1, 2020. http://www.doi.org/10.7910/DVN/0QQNPF

[13] 13. Scott Adams P: Data Analysis and Reporting. In: Dorak T editor. Real-time PCR. 1st ed. New: Taylor & Francis; 2006; 39–62. Reference Source

[14] 14. Schober P, Schwarte LA: Correlation coefficients: Appropriate use and interpretation. Anesth Analg. 2018; 126(5): 1763–8. PubMed Abstract | Publisher Full Text

[15] 15. Viera AJ, Garrett JM: Understanding interobserver agreement: The kappa statistic. Fam Med. 2005; 37(5): 360–3. PubMed Abstract

maxRatio improves the detection of samples with abnormal amplification profiles on QIAgen’s artus HIV-1 qPCR assay

Abstract

Keywords

Revised Amendments from Version 2

Introduction

Methods

Dataset

Data analysis

Results

Figure 1. Cut-offs for maxRatio.

Figure 2. Linear models and CD quantification.

Table 1. Comparison of copy numbers for the control dilutions.

Figure 3. CD quantification.

Figure 4. Correlation of VL obtained by FP and maxRatio.

Figure 5. Samples not quantified only by FP.

Figure 6. Samples not quantified only by maxRatio.

Discussion

Data availability

Underlying data

Acknowledgements

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated