Performance characteristics of a modified HIV-1 drug resistance genotyping method for use in resource-limited settings

Background: HIV-1 drug resistance (HIVDR) assays are critical components of HIV clinical management programs in the face of emerging drug resistance. However, the high costs associated with existing commercial HIVDR assays prohibit their routine usage in resource-limited settings. We present the performance characteristics of a modified commercial HIVDR testing assay. Methods: A total of 26 plasma samples were used to validate and assess the accuracy, precision, reproducibility and amplification sensitivity of a modified HIVDR assay by HIV genotyping. In addition, a cost comparison between the original and the modified assay was performed using the ingredient costing approach. Results: The performance characteristics of the modified assay were in agreement with the original assay. Accuracy, precision and reproducibility showed nucleotide sequence identity of 98.5% (confidence interval (CI), 97.9–99.1%), 98.67% (CI, 98.1–99.23) and 98.7% (CI, 98.1–99.3), respectively. There was no difference in the type of mutations detected by the two assays (χ 2 = 2.36, p = 0.26). Precision and reproducibility showed significant mutation agreement between replicates (kappa = 0.79 and 0.78), respectively ( p < 0.05). The amplification sensitivity of the modified assay was 100% and 62.5% for viremia ≥1000 copies/ml and <1000 copies/ml respectively. Our assay modification translates to a 39.2% reduction in the cost of reagents. Conclusions: Our findings underscore the potential of modifying commercially available HIVDR testing assays into cost-effective, yet accurate assays for use in resource-limited settings.


Introduction
There has been an unprecedented increase in support for HIV diagnosis, treatment and monitoring programmes. Various multinational groups including the U.S. President's Emergency Plan for AIDS Relief (PEPFAR) and Global Fund have increased funding of HIV management programs leading to improved access to antiretroviral therapy (ART) 1,2 . The outcome of the expanded access to ART is significant decline of HIV/AIDSassociated morbidity and mortality 2 . As a result, there has been an increase in both life expectancy and duration for patients on lifelong ART translating to high risk of HIV drug resistance (HIVDR) development. To sustain the success achieved following improved ART coverage, continuous clinical, virologic and immunological monitoring of patients on ART is required to ensure these positive treatment outcomes are maintained 3,4 . Unfortunately, development and transmission of HIV drug resistance threatens to derail these achievements 5 . HIV drug resistance testing (DRT) is routinely used for clinical care in high-income countries; however, it is not available to a majority of patients in resource-limited settings due to the high costs of implementation and limited trained manpower.
The World Health Organization (WHO) launched the Global Action Plan (GAP) against HIVDR for resource-limited countries whereby two of the proposed approaches included continuous innovation and capacity building of laboratory staff 6 .
Scientists have attempted to develop alternative affordable methods for HIV drug resistance monitoring to improve access 5,7-9 . One such test was developed by Centre for Disease Control and Prevention (CDC) and is currently distributed by Thermo Fisher Scientific 10 , and is herein referred to as the original assay. The assay is used to genotype the genetically diverse HIV-1 virus from plasma samples to detect resistance mutations in the protease and reverse transcriptase genes with a good subtype inclusivity rate compared to other commercial DRT assays in the market 11 .
In the present study, we modified this commercial HIV DRT protocol by a 50% reduction of reagent volumes for nested PCR and cycle sequencing reactions for genotyping of HIV-1 drug resistance. The performance characteristics of the modified assay were assessed and evaluated for suitability and reliability using WHO guidelines for assay validation. In addition, cost comparison was performed to determine the cost implication of the assay modification.  Table 1. Cycle sequencing was performed using the HIV-1 Genotyping Kit: Cycle Sequencing Module. The resulting cycle sequencing products were analyzed using an ABI 3730 genetic analyzer. The consensus sequences were generated using ReCall (requires free registration) and drug resistance mutations were interpreted using the Stanford HIVdb genotyping resistance interpretation algorithm. Cycle sequencing was performed using the HIV Genotyping kit: Cycle Sequencing module. Six overlapping primers, labelled F1, F2, F3, R1, R2, and R3 were used. One microliter nested PCR product was added to 9 µl of each cycle sequencing mix. The cycle sequencing reaction conditions were 25 cycles of 96°C for 10 seconds, 50°C for 5 seconds, and 60°C for 4 minutes.

Samples
The Big Dye XTerminator purification kit was used to purify the sequencing reaction by adding 10 µl of the Big Dye XTerminator and 45 µl SAM solution to cycle sequencing products. The reaction plate was vortexed at 1,800 rpm for 30 minutes. The plate was then centrifuged at 1000g for 2 minutes at room temperature. Next, 30 µl of purified cycle sequencing products were transferred to a reaction plate and analyzed using an ABI 3730 genetic analyzer (Applied Biosystem, CA, USA). Sequences were generated using ReCall and drug resistance mutations interpreted using the Stanford HIVdb genotyping resistance interpretation algorithm (output files available as Extended data 12 ) and the International AIDS Society (IAS) 2011 mutation list 13 .

Modified assay validation
Performance characteristics of the modified assay were assessed using the WHO/HIV ResNet guidelines, including accuracy, precision, reproducibility and amplification sensitivity 14 .
Accuracy. Accuracy refers to the agreement between a result and an expected reference value. In genotyping assays, nucleotide sequence identity is used. Ten samples were analyzed using both methods and the degree of concordance in mutations identified was compared based on the 2017 IAS mutations list 13 . Nucleotide sequence identity between the paired sequences was assessed using the EMBOSS pairwise alignment tool. The WHO recommends 90% nucleotide sequence similarity as the cutoff point for assay performance characteristics 14 .
Precision. This is the ability of an assay to generate the same result on multiple replicates of the same sample within a test run. Three samples were analyzed using the modified method in n=4 replicates. The degree of concordance of detected mutations within replicates was determined. Nucleotide sequence identity was also determined using the EMBOSS program for pairwise alignment 14 .
Reproducibility. This is the ability of a test to produce the same result on multiple aliquots of the same sample in different test runs. Ten samples were analyzed in duplicates using the modified method on different days. Nucleotide sequence identity of sequences obtained was assessed using EMBOSS program for pairwise alignment 14 .
Amplification sensitivity. Amplification sensitivity is defined as the percentage of successful genotyping tests amongst specimens with a specific viral load range. In this study, sixteen samples with viral loads ranging between 207 and 86,040 copies/ml were analyzed using the modified assay to determine the viral load ranges at which ≥95% of the samples were successfully genotyped 14 .

Reagent cost comparison
An ingredient costing approach was utilized to estimate reagent costs for both original and modified assays at RNA extraction, DNA/RNA amplification, gel electrophoresis, and sequencing steps. All costs were converted to US dollar using 2019 conversion rate.

Statistical analyses
Quantitative variables were expressed as mean ± standard deviations (SD). The McNemar test was used to assess significance in the discordant mutations between the modified and the original assay. Precision and reproducibility were assessed using the Cohen kappa statistic. Wilcoxon signed-rank test was used to compare the original and modified assays in base calling for mixed bases between the two methods. Statistical Package for Social Sciences (SPSS) version 22 (IBM, NY, USA) was used for all data analysis.

Results
Validation of the modified HIV-1 drug resistance genotyping assay Accuracy of sequence identity and mutation detection. The mean nucleotide identity was 98.5% (confidence interval (CI), 97.92-99.1%). A total of 68 mutations were detected, including 9 (13%) mutations in the protease gene and 59 (87%) in the reverse transcriptase (RT) gene as shown in  K101P and K70N) in the original assay and as shown in Table 3. A high mutation concordance was obtained between the two assays at χ 2 = 2.36, p = 0.26. The significance of mixture base-calling between the two methods was determined using the Wilcoxon signed rank test, which showed no significant difference in mixtures detected by the two methods at p = 0.089.

Precision and reproducibility of the modified assay.
Assessment to examine the precision (intra-assay precision) and reproducibility (inter-assay precision) of the modified assay were performed by analyzing n=3 samples in quadruplicate in a single test run for precision. The mean nucleotide sequence identity within the replicates was 98.67% (CI, 98.1-99.3). Reproducibility was assessed by testing 10 samples in duplicate on different days using the modified assay. The mean nucleotide sequence identity was 98.6% (CI, 98.2-99.0). The overall agreement of drug resistance mutations detected by precision and reproducibility was significant at kappa value of 0.792 and 0.778, respectively, as shown in Table 4.

Amplification sensitivity of the modified assay.
A total of 16 samples with a median viral load of 832.5 copies/ml (IQR 403.5-2454.25) were used to assess amplification sensitivity.
The modified assay showed an amplification sensitivity of 100% for samples with viral load ≥1000 copies/ml and 62.5% for samples with viral load <1000 copies/ml as shown in Table 5.

Comparison of costs of reagents between the original and modified assays
After a 50% reduction in reagent volumes in the amplification and sequencing reactions, we compared the cost between the original and the modified method. The cost of analyzing one sample from extraction to sequencing was decreased from $97 to $59. This represents about 39.2% cost reduction as shown in Table 6.     15 . Furthermore, the modified assay showed 100% sensitivity for VL >1000 copies/ml which is the recommended indicator for HIV treatment failure 16 . In general, the modified assay met all the requirements in the WHO Genotyping assay validation recommendations. Altogether our findings demonstrate that reducing the volumes of reagents in the original assay by up to half results in a low-cost HIV drug resistance assay that could be utilized in resource-limited settings without compromising the quality of testing.
Accuracy was assessed by analyzing 10 samples with viral load ranges (4258 -78924 copies per ml) by both the original and the modified assays and all 10 samples were successfully amplified and genotyped with high nucleotide sequence identity (98.5±0.94%). All the clinically relevant mutations were reproducible by both methods in spite of some minor discordance between the two methods. The WHO recommends a minimum of 90% sequence similarity for genotyping assays 14 .
Our findings of 98.5±0.94% nucleotide sequence similarity are in agreement with those reported by Zhou et al. 7 who reported 99.41%. In addition, analysis of mutation concordance between the original and the modified assay showed high mutation concordance also reported by Inzaule et al. 8 . Our findings for mutation concordance (93%) were lower than reported by Manasa et al. (100%) 5 . The modified assay precision assessment showed a high degree of nucleotide sequence identity within replicates. Similar results for precision (98.22%) were reported by Zhou et al. 7 . High sequence similarity was also obtained for reproducibility of the modified assay. Our findings for reproducibility (98.6%) were similar to those reported by Zhou et al., who obtained (98.94%) 7 . Amplification sensitivity was defined as the percentage of samples successfully genotyped at a given plasma viral load range. In this study, all samples with viral load ≥1000 copies/ml were successfully genotyped, a finding that was similar to that reported by a previous study 8 .
For samples with plasma viral loads <1000 copies/ml, our modified assay successfully genotyped 62.5% of the samples which is slightly below 63.6% reported by a previous study 8 . The WHO criteria for HIVDR testing requires that patients with persistent viral load ≥1000 copies/ml to be tested for HIV drug resistance 17 . Our assay was 100% efficient in genotyping these samples. Furthermore, with 62.5% amplification sensitivity for low level viremia (LLV) samples, highlighted the possibility of using the modified assay in genotyping patients with LLV. However, there is need to perform further large-scale studies to clearly establish the performance of the modified method in LLV patients.
We detected six discordant mutations in both original and modified assays. Out of the six discordant mutations, four were mixed-base mutations, which were detected exclusively by the modified assay. The differential detection of discordant mutations between the two assays could be attributed to detection of nucleotide mixtures as a result of subjectivity in basecalling or amplification bias 18,19 . Previous studies have also reported a significant contribution of nucleotide mixtures to discrepancies of mutations between Viroseq and an in-house assay 6,7 .
The comparison of cost between the two assays revealed a reduction of HIV drug resistance testing cost by 39.2% per test. These findings suggest that access to HIV drug resistance services in resource-limited settings could be increased through innovative modifications of existing commercially available assays. A small sample size was used because this was a small modification of a validated test already in use. The applicability of this assay can be demonstrated further by testing a larger number of samples. One limitation of the study is the samples used were from patients failing first line treatment and not exposed to protease inhibitors leading to under representation of drug resistant mutations in the protease gene.

Conclusion
Our findings underscore the potential utility of a modified cost-effective HIV-1 drug resistance assay for testing plasma samples in resource-limited settings. The performance characteristics of the modified assay were satisfactory and therefore the cheaper assay could be one of the approaches of increasing access to HIVDR tests.

Data availability
Underlying data GenBank accession numbers for HIV-1 pol and gag sequences isolated from each participant using each technique are available in Table 7.

Grant information
The author(s) declared that no grants were involved in supporting this work. MN240792 Modified the commercial assay. Furthermore, the authors have demonstrated that these assay modifications resulted in a significant decrease in cost of implementation. I however wonder why the authors did not present results for reducing reaction volumes for the RT-PCR step. It is conceivable that they may have at least attempted and therefore should comment if the results did not meet the WHO criterion for assay validation.

If applicable, is the statistical analysis and its interpretation appropriate? Yes
Are all the source data underlying the results available to ensure full reproducibility?

Samples
Are 26 samples enough to validate an assay? And how did you arrive upon this sample size?
How were the samples collected? By venipuncture or cardiac puncture? Be very clear by which method.
After clinic, delete the word 'laboratory'. Patients attend clinics, not clinic laboratories.
-80 is already a freezing condition. Delete the word 'freezer', the statement will still make sense without the subtle repetition.
What are 'remnant plasma samples'? Were they the ones stored in the freezer? What defines them as 'remnant'?

Ethical approval
It is claimed here that consent was waived, yet in the sample collection, it is clearly stated that the samples were collected from patients. How and why was the consent waived, and by whom?

Viral RNA extraction
Add 'the' before manufacturer's.

HIV DRT
As this was a study validating an assay that relied heavily on reagent volumes, why aren't the volumes of the original assay and their reaction temperatures not included here?

Reagent cost comparison
As the year is not out yet and conversion rates keep fluctuating, which conversion rate did you use? Precision, Reproducibility, Accuracy and Provide references for definition of the following words: amplification sensitivity

Statistics
Significance was set at what value? p Results Only two drug resistance genes are targeted in this assay, what informed the decision to arrive on these two sets of genes? Table 5 should be the one introducing the results subtopic. Introduce the samples, the viral loads before the validation assay results are presented.
Remove the grids from the tables.