Discordant results among major histocompatibility complex binding affinity prediction tools

Austin Nguyen; Abhinav Nellore; Reid F. Thompson

doi:10.12688/f1000research.132538.1

Home Browse Discordant results among major histocompatibility complex binding...

ALL Metrics

-

Views

-

Downloads

Get PDF

Get XML

Export

▬

✚

Research Article

Discordant results among major histocompatibility complex binding affinity prediction tools

[version 1; peer review: 2 approved with reservations, 1 not approved]

Austin Nguyen^1,2, Abhinav Nellore ^1-3, Reid F. Thompson^1,2,4-6

PUBLISHED 07 Jun 2023

Author details Author details

¹ Department of Biomedical Engineering, Oregon Health & Science University, Portland, Oregon, 97239, USA
² Computational Biology, Oregon Health & Science University, Portland, Oregon, 97239, USA
³ Department of Surgery, Oregon Health & Science University, Portland, Oregon, 97239, USA
⁴ Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, Oregon, 97239, USA
⁵ Department of Radiation Medicine, Oregon Health & Science University, Portland, Oregon, 97239, USA
⁶ Division of Hospital and Specialty Medicine, VA Portland Healthcare System, Portland, Oregon, 97239, USA

Austin Nguyen
Roles: Conceptualization, Data Curation, Formal Analysis, Methodology, Project Administration, Resources, Software, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Abhinav Nellore
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Review & Editing

Reid F. Thompson
Roles: Conceptualization, Funding Acquisition, Investigation, Project Administration, Supervision, Writing – Review & Editing

OPEN PEER REVIEW

REVIEWER STATUS

This article is included in the Bioinformatics gateway.

Abstract

Background: Human leukocyte antigen (HLA) alleles are critical components of the immune system’s ability to recognize and eliminate tumors and infections. A large number of machine learning-based major histocompatibility complex (MHC) binding affinity (BA) prediction tools have been developed and are widely used for both investigational and therapeutic applications, so it is important to explore differences in tool outputs.
Methods: We examined predictions of four popular tools (netMHCpan, HLAthena, MHCflurry, and MHCnuggets) across a range of possible peptide sources (human, viral, and randomly generated) and MHC class I alleles.
Results: We uncovered inconsistencies in predictions of BA, allele promiscuity and the relationship between physical properties of peptides by source and BA predictions, as well as quality of training data. We found amount of training data does not explain inconsistencies between tools and yet for all tools, predicted binding quantities are similar between human and viral proteomes. Lastly, we find peptide physical properties are associated with allele-specific binding predictions.
Conclusions: Our work raises fundamental questions about the fidelity of peptide-MHC binding prediction tools and their real-world implications. The real-world use of these prediction tools for theoretical binding of peptides to alleles is worrying, as the range of allele promiscuity is substantial yet does not differentiate between potential foreign versus self-antigens. Evaluating more viruses – as well as bacteria, fungi, and other pathogens – and linking these analyses with metrics such as evolutionary distance may give greater insight into the relationship between HLA evolution and disease.

Keywords

Human leukocyte antigen, major histocompatibility complex, MHC, binding, allele, netMHCpan, immune system

Corresponding authors: Abhinav Nellore, Reid F. Thompson

Competing interests: No competing interests were disclosed.

Grant information: The author(s) declared that no grants were involved in supporting this work.

Copyright: © 2023 Nguyen A et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

How to cite: Nguyen A, Nellore A and Thompson RF. Discordant results among major histocompatibility complex binding affinity prediction tools [version 1; peer review: 2 approved with reservations, 1 not approved]. F1000Research 2023, 12:617 (https://doi.org/10.12688/f1000research.132538.1) First published: 07 Jun 2023, 12:617 (https://doi.org/10.12688/f1000research.132538.1) Latest published: 07 Jun 2023, 12:617 (https://doi.org/10.12688/f1000research.132538.1)

Introduction

Human leukocyte antigen (HLA) alleles are critical components of the immune system’s ability to recognize and eliminate tumors and infections.¹ Infectious diseases in particular are thought to be a major source of selective pressure on the major histocompatibility complex (MHC) region which encodes HLA alleles and is one of the most diverse regions of the human genome.²^–⁸ There is large diversity in the antigenic peptide sequences which individual HLA alleles can recognize and ultimately present to the adaptive immune system,⁹ with a positive correlation between increased sequence diversity recognition and fitness.¹⁰

Tools that can predict the extent to which a given HLA allele may have an affinity for a given peptide have critical implications for our ability to understand and translationally leverage antigen-specific immune response pathways. For instance, MHC binding affinity predictors have been – or otherwise have the potential to be – used to evaluate an individual or population’s susceptibility to viral infection,¹¹ to develop an understanding of specific autoimmune conditions,¹² to improve transplantation technologies,¹³ or even to assist in the development of personalized cancer vaccines.¹⁴^–¹⁸ Numerous peptide-MHC binding prediction tools exist, and are key components in broader antigen prediction methodologies.¹⁹^–²²

The most widely adopted MHC binding prediction tools rely on neural network models trained on binding affinity (BA) and/or eluted ligand (EL) data. The most commonly cited tool, netMHCpan,²³^,²⁴ uses both BA and EL data in a neural network architecture with a single hidden layer to predict allele-specific binding affinities. MHCflurry²⁵ attempts to improve upon netMHCpan by increasing the number of hidden layers and augmenting BA and EL training data with unobserved decoys. MHCnuggets²⁶ again trains on BA and EL data but uses a different architecture, with a long short-term memory layer and a fully connected layer to improve its predictions further across different peptide lengths. Lastly, HLAthena,²⁷ while most similar in architecture to netMHCpan, relies on independently generated EL data from mono-allelic cell lines for training.

We sought to better characterize the outputs of these tools over a large and diverse set of peptides, across different tools and HLA alleles, as well as quantify the stability of these predictions. We also sought to measure allelic binding preferences and whether they may enrich for foreign v. self peptides. In this study, we performed a comprehensive in silico analysis of peptides from multiple viral proteomes, the human proteome, and randomly generated peptides across HLA class I alleles.

Methods

Sequence retrieval, peptide filtering, and kmerization

FASTA-formatted protein sequence data was retrieved from the National Center of Biotechnology Information (NCBI)²⁸^,²⁹ using RefSeq as of 1-31-22 for BK, SARS-CoV-2, HHV-5, HHV-6, HSV-1, HSV-2, HSV-4, and Human. Protein sequence data was inputted into netchop v3.0 “C-term” model with a cleavage threshold of 0.1 to remove peptides that were not predicted to undergo canonical MHC class I antigen processing via proteasomal cleavage (of the peptide’s C-terminus). The results from netchop v3.0 were then kmerized sequentially into 8- to 12-mers. Code used for kmerization and netchop filtering can be found at: https://github.com/Boeinco/peptide-MHCassess. We additionally generated a set of 1 million random peptides of length 8-12 drawn uniformly at random. Peptide sets had negligible overlap (<1% shared between human vs viral vs random peptides).

Peptide-MHC class I binding affinity predictions

MHC class I binding affinity predictions were performed for the peptides generated from the kmerization process above using four tools: netMHCpan v4.1,²³ HLAthena v1.0,²⁷ MHCflurry v2.0,²⁵ and MHCnuggets v2.3.³⁰ netMHCpan was run with default options with the ‘-l’ option to specify peptides of lengths 8–12. MHCflurry was run with default options. MHCnuggets was run with default options. HLAthena was run using the dockerized version of HLAthena with default options, which predicts peptides of length 8–11. MHC class I binding affinity predictions were performed for each of 24, 26, and 2, HLA-A, -B, and -C alleles, respectively. Only alleles that were in common between all four tools were used (52 total alleles in common between 2489 possible alleles). Binding affinity values were converted to binding probability values for MHCflurry and MHCnuggets using 1- log (binding affinity) /log(50000) in order to match HLAthena and netMHCpan binding probability predictions. Alleles were grouped into supertypes when applicable using the HLA class I revised classification.³¹

Dimensional reduction and binning analysis

Peptides were converted into physical property matrices using amino acid sequence mapping into a 4*kmer length matrix containing each amino acid’s properties in sequence. The following physical properties of the amino acids were encoded: side chain polarity was recorded as its isoelectric point (pI),³² the molecular volume of each side chain was recorded as its partial molar volume at 37°C,³³ the hydrophobicity of each side chain was characterized by its simulated contact angle with nanodroplets of water³⁴ and conformational entropy was derived from peptide bond angular observations among protein sequences without observed secondary structure.³⁵

Each dimensional reduction was performed on the pooled set of k-mers. UMAP dimensionality was performed using uwot UMAP R implementation v0.1.11. PCA was performed using default prcomp() functions in base R v4.1.3.

For each peptide source, binned matrices were computed using the bin2() function with 40×40 (1600) bins from the Ash v1.0.15 package³⁶ in R v4.1.3. Bin values were then divided by the total number of peptides to create bins with the % of total peptides. In order to compare between two peptide sources, a matrix, called the difference matrix, is created by subtracting one matrix of a peptide source from another. Taking the absolute value of each bin in the difference matrix, then summing the values together, results in a single metric ranging from 0–2 measuring the difference in binned density between two peptide sources, the value 2 indicating that no peptides were shared between bins and the value 0 indicating the same percentage of peptides in every bin (Figure 1).

Figure 1. Schematic of peptide binned density metric.

Bin values were then divided by the total number of peptides to create bins with the % of total peptides. In order to compare between two peptide sources, a matrix, called the difference matrix, is created by subtracting one matrix of a peptide source from another. Taking the absolute value of each bin in the difference matrix, then summing the values together, results in a single metric ranging from 0–2.

Allele ordering similarity

For each allele-peptide source combination, the percentage of peptides predicted to bind with a binding probability score of 0.5 or greater was calculated for all processed peptides.³⁷ A binding score of 0.5 is estimated to be equivalent to 250–300 nM depending on the tool used. For each peptide source, alleles were ranked from best to worst binders (most to least peptides ≥ 0.5 score) t. In order to compute allele ordering similarity between two peptide sources for a single tool, Spearman’s Rank Correlation Coefficient was calculated between the two sets of allele ranks.

For the random group 1 vs random group 2 analysis, we conducted 100 replicates of dividing the randomly generated peptides into two random groups and performed a Spearman rank test of allele ordering between these groups for each of the tools.

Interrater reliability

Intraclass correlation coefficients (ICCs) were calculated using the ICC () function from the IRR v0.84.1 R package.³⁸ Binding prediction scores for all one million randomly generated peptides were separated by tool and HLA allele, and an ICC was calculated as the interrater reliability metric between the four tools for each allele. ICC was also between the four tools on a per peptide basis, each peptide receiving a score across four tools using predictions separated by tool and peptide.

An earlier version of this article can be found on bioRxiv (https://doi.org/10.1101/2022.12.04.518984). Source code can be found on Github.³⁹

Results

Peptide predictions are inconsistent across tools

We first assessed the consistency of peptide-specific MHC I binding affinity predictions across four tools (MHCnuggets, MHCflurry, HLAthena, netMHCpan) and 52 different HLA alleles. We found substantial disagreement in peptide-specific predictions between each tool, independent of allele (Figure 2A), with median intraclass correlation coefficient (ICC) of 0.207 and only 0.48% of peptides having ICC >0.75. On a per-allele basis, we found a wide range in consistency of predictions across tools, with a mean intraclass correlation as low as 0.12 for A02:07 and as high as 0.64 for A23:01 (Figure 2B). Among all of the peptides predicted by at least one tool to bind to at least one allele, only 7.9% were consistently predicted across all tools to bind to the same allele (Figure 2C).

Figure 2. Inconsistency of peptide predictions across tools.

A) Histogram of intraclass correlation coefficients (ICC) calculated for a set of 1 million random peptides across four tools (MHCnuggets, MHCflurry, HLAthena, netMHCpan), with ICC calculated as the overall correlation among tools across 52 HLA alleles. The dotted vertical line indicates the median ICC value (0.207) across all peptides. B) Histogram of ICCs for 52 HLA alleles between four tools (MHCnuggets, MHCflurry, HLAthena, netMHCpan). The number of alleles is shown on the y-axis and the ICC is shown on the x-axis. The dotted lines show the mean ICC for alleles belonging to each HLA class. Red, green, and blue colors represent data from -A, -B, and -C alleles, respectively. C) Detailed comparison of the complete set of random peptides predicted to bind (binding score ≥ 0.5) to HLA alleles according to each of four tools. Patterns of agreement or disagreement among groups of peptides predicted by different combinations of tools across 1 million random peptides are shown along each column (e.g. the first column corresponds to peptides predicted by HLAthena while the final column corresponds to peptides predicted by all tools). Each row indicates the predictions associated with the indicated tool. The number of peptides in each column (vertical bars) corresponds to the size of the subset predicted by the indicated combination of tools.

We next investigated aggregate peptide binding predictions across different HLA alleles according to each tool. As others have noted differential HLA allelic promiscuity in peptide presentation,³¹^,⁴⁰^–⁴² we too found a wide range in the proportion of peptides a given allele was predicted to bind (Underlying data: Supplementary Figure 1³⁷). We uncovered significant inconsistencies in these predictions between tools (Figure 3). Note that this phenomenon is independent of binding affinity threshold (Underlying data: Supplementary Figure 2³⁷).

Figure 3. The correlation of HLA allelic presentation of 8-11mers from the random proteome between tools.

The lower left grouping of plots displays scatter plots of peptides predicted to bind (≥ 0.5 binding probability score) between 2 tools with each point representing the number of predicted binders for each HLA allele. The upper right grouping represents the Spearman correlation of the number of peptides predicted to bind to all alleles between tools. Note that MHCnuggets has a number of alleles with 0 random peptides predicted to bind. The diagonal panels show distribution of HLA allelic presentation from the random proteome for each tool. The number of peptides that putatively bind to each of the HLA alleles is shown along the x-axis as a series of horizontal bars with green, orange, and purple colors representing HLA-A, -B, and -C alleles, respectively, sorted in order of decreasing quantity of binders.

Amount of training data does not explain inconsistencies between tools

As each allele has a different amount of training data, we were next interested in exploring to what extent the quantity and quality of training data available to each tool might influence its allele-specific predictions. Indeed, some netMHCpan predictive models for some alleles are based on as few as 101 peptides, while others from MHCflurry are based on as many as 31,775 peptides (Underlying data: Supplementary Table 1³⁷). Note that we excluded from consideration the ~95% of alleles (4108) that were available for prediction but had no underlying allele-specific training data available (Underlying data: Supplementary Table 2³⁷). Ultimately, we found that the amount of training data available was not significantly related to the consistency of binding predictions between tools (Figure 4A), nor was it clearly related to the quantity of binding peptides predicted by tools (Figure 4B).

Figure 4. The relationship between training data and consistency of predictions.

A) Scatterplot of ICC vs mean training data across 4 tools with each point representing data for a single HLA allele. The mean number of training peptides is shown on the x-axis while the ICC score is shown on the y-axis. B) Scatterplot of the relationship between training data and predicted peptide binding. The number of peptides used as training data for an allele is shown on the x-axis whereas the number of peptides predicted to bind for the same allele is shown on the y-axis. Each dot is a single allele with each color representing a different tool: red circles (HLAthena), green triangles (MHCflurry), blue squares (MHCnuggets), purple plus signs (netMHCpan). We note that netMHCpan does not make all of their training data available, thus the depicted quantity of training data represents an estimate.

Predicted binding quantities are similar between human and viral proteomes

According to the pathogen driven selection theory of MHC evolution, different HLA alleles are anticipated to be particularly attuned to foreign as opposed to self-antigens.³^,⁸^,⁴³^–⁴⁶ We therefore sought to compare the predicted capacity of different HLA alleles to present different viral vs. self-antigens. Further, we wished to establish which specific alleles had the propensity to bind a larger fraction of peptides in general (allele promiscuity) by observing the relationship between an allele’s ability to bind random peptides versus peptides from a viral or human proteome.

We examined distribution of predicted allelic promiscuity across alleles for nine sets of peptides of viral, human, and random origin (See Methods). Confining attention to human and viral proteomes, we again found a wide range in the proportion of peptides a given allele was predicted to bind and also significant inconsistencies between tools (Underlying data: Supplementary Figure 3³⁷).

We found that the alleles with highest mean binding percentage for human and viral peptides were B15:03 (2.68%) and B15:02 (2.36%) and the allele lowest mean binding percentage were B18:01 (0.24%) and A01:01 (0.33%) (Underlying data: Supplementary Table 3³⁷). No alleles were predicted by any tool to preferentially present either viral or human peptides. Further, the distribution of predicted allelic promiscuity across alleles was highly consistent between human and viral proteomes, but not when applied to a set of random peptides (Figure 5). We noted that this phenomenon holds for closely related viruses across all tools and to a lesser extent for more distantly related viruses (Underlying data: Supplementary Figure 4³⁷).

Figure 5. The correlation between peptide sources of predicted allelic promiscuity across alleles.

A) Heatmap of spearman correlation between peptide sources for HLAthena-based predictions for human peptides, viral peptides, and randomly generated peptides. Numbers show Spearman correlation coefficients between each pair respectively, while color reflects the Spearman correlation with red approaching a Spearman correlation of 1. Analogous data is shown for netMHCpan, MHCflurry, and MHCnuggets in panels B, C, and D, respectively.

Confining attention to the nine alleles whose predictive models were likely most robust (based on a minimum of 2000 training peptides for every tool), we again found that the distribution of predicted allelic promiscuity across alleles was consistent between closely related viruses and to a lesser extent between more distantly related viruses (Underlying data: Supplementary Figure 5³⁷).

Peptide physical properties are associated with allele-specific binding predictions

Reasoning that differences in peptide characteristics were the likeliest explanation for predicted differences in binding affinity between different alleles and peptide sources, we next studied the distribution of physical properties among different peptide sets. Human, viral, and random peptide sets all exhibited the same range of physical properties but were differentially enriched among different physical properties (Underlying data: Supplementary Figure 6³⁷). Between individual peptide sets, the differential enrichment ranged from 10% (CMV v. human) to 63% (BK v. random) of peptides (Underlying data: Supplementary Figure 7³⁷).

We next sought to discover the relationship between the peptide similarity in physical property space and distribution of predicted allelic promiscuity across alleles. Across all tools, there was a positive relationship between similarity in physical property space and distribution of predicted allelic promiscuity across alleles as evidenced by the negative correlation between peptide set difference and Spearman correlation coefficient (Figure 6).

Figure 6. The relationship between physical property similarity vs peptide binding similarity.

A) Scatterplot for HLAthena-based predictions, where each point represents predictions for a species vs species pair. Peptide dissimilarity is shown on the x-axis, whereas Spearman correlation coefficients of predicted allelic promiscuity across alleles. Color represents the length of peptide, with 8-, 9-, 10-, and 11-mers shown in red, green, blue and purple, respectively. Analogous data is shown for netMHCpan, MHCflurr, and MHCnuggets in panels B, C, and D, respectively.

Next, we found that each allele has distinct preferences for different peptide physical properties, independent of peptide length (Figure 7A, Supplementary Figure 8³⁷). Some alleles (e.g. A01:01 and B08:01) have stronger preference for certain physical properties (Figure 7B, C), while others (B45:01) do not have as clear of a preference (Figure 7D).

Figure 7. Differential distributions of physical properties for 9-mer peptides predicted to bind to HLA alleles.

A) The plotting coordinates represent the first two dimensions of a UMAP transform of peptide physical properties, which is divided into 1600 (40×40) equivalently-sized square bins (see Methods). For each bin where there is at least one HLA allele with >0.2% difference in proportion of all peptides predicted to bind v. non-binders, the identity of the most enriched allele is shaded in the color corresponding to that allele’s supertype as corresponding to the legend. B-D) Example plots of three different alleles (A01:01, B08:01, and B45:01) with different distributions of binders. Each box represents enrichment as the percent peptide difference between predicted binders and non-binders for the given allele. The color scale shows the percent of peptides difference in the given box, with red meaning a larger number of predicted binders and blue meaning a larger number of predicted non-binders.

Discussion

To the best of our knowledge, this is the first study to examine the consistency of predictions of peptide-MHC binding across different tools, and to explore the quality and quantity of training data in this context. We note several limitations to this work. Firstly, we confined attention to MHC class I peptides and did not include predictions for MHC class II,⁴⁷ of which there are numerous alleles. We also excluded from consideration any potential contributions of proteasomal cleavage or other antigen processing machinery to MHC binding.⁴⁸^–⁵⁰ We did not seek to comprehensively assess all available tools for peptide-MHC binding affinity prediction, but rather confined our attention to four of the most widely used tools. The majority of our randomly generated peptides are not known to be found in nature and may not represent the optimal background distribution for measuring allele promiscuity or interrater reliability between tools primarily used for human and pathogenic peptides. While our analysis of peptides leveraged four essential and well-described amino acid physical properties, there may exist unassessed latent features that could capture additional variance and improve dimensionally-reduced comparisons. We did not assess the extent to which mass spectrometry biases in the training datasets might affect peptide-MHC predictions.⁵¹^–⁵⁴ Lastly, we did not evaluate individual tool performance based on known epitopes as this has been previously reported.²³^–²⁷^,⁵⁵^–⁵⁹

Conclusions

Our work raises fundamental questions about the fidelity of peptide-MHC binding prediction tools. Why, for instance, can predictions be so discordant among tools for which training datasets are otherwise so similar? We especially worry about the real-world use of these prediction tools for alleles without any direct basis in training data. Why is the predicted range of allele promiscuity so substantial, and yet not demonstrative of any meaningful differences in enrichment between potential foreign versus self-antigens? Moreover, is this differential promiscuity a universal biological phenomenon, with certain alleles being generally poor functional presenters of antigen? If this is the case, what selective advantage might have evolutionarily maintained these alleles in the population? Evaluating more viruses – as well as bacteria, fungi, and other pathogens – and linking these analyses with metrics such as evolutionary distance may give greater insight into the relationship between HLA evolution and disease.

Data availability

Underlying data

Zenodo: Underlying data for ‘Discordant results among MHC binding affinity prediction tools’, https://doi.org/10.5281/zenodo.7850939.³⁷

This project contains the following underlying data:

• Supplementary figures.docx
• Supplementary tables 1–3.xlsx

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Software availability

Source code available from: https://github.com/Boeinco/peptide-MHCassess

Archived source code at time of publication: https://doi.org/10.5281/zenodo.7803676.³⁹

License: MIT

References

1. Dendrou CA, Petersen J, Rossjohn J, et al.: HLA variation and disease. Nat. Rev. Immunol. 2018 May; 18(5): 325–339. Publisher Full Text
2. Blackwell JM, Jamieson SE, Burgner D: HLA and Infectious Diseases. Clin. Microbiol. Rev. 2009 Apr; 22(2): 370–385. PubMed Abstract | Publisher Full Text | Free Full Text
3. Meyer D, Aguiar VRC, Bitarello BD, et al.: A genomic perspective on HLA evolution. Immunogenetics. 2018; 70(1): 5–27. PubMed Abstract | Publisher Full Text | Free Full Text
4. Zernich D, Purcell AW, Macdonald WA, et al.: Natural HLA Class I Polymorphism Controls the Pathway of Antigen Presentation and Susceptibility to Viral Evasion. J. Exp. Med. 2004 Jun 28; 200(1): 13–24. PubMed Abstract | Publisher Full Text | Free Full Text
5. Bihl F, Frahm N, Giammarino LD, et al.: Impact of HLA-B Alleles, Epitope Binding Affinity, Functional Avidity, and Viral Coinfection on the Immunodominance of Virus-Specific CTL Responses. J. Immunol. 2006 Apr 1; 176(7): 4094–4101. PubMed Abstract | Publisher Full Text
6. Berger CT, Carlson JM, Brumme CJ, et al.: Viral adaptation to immune selection pressure by HLA class I–restricted CTL responses targeting epitopes in HIV frameshift sequences. J. Exp. Med. 2010 Jan 18; 207(1): 61–75. PubMed Abstract | Publisher Full Text | Free Full Text
7. Schellens IM, Meiring HD, Hoof I, et al.: Measles Virus Epitope Presentation by HLA: Novel Insights into Epitope Selection, Dominance, and Microvariation. Front. Immunol. 2015 [cited 2019 Nov 15]; 6. PubMed Abstract | Publisher Full Text | Free Full Text
8. Kaufman J: Generalists and Specialists: A New View of How MHC Class I Molecules Fight Infectious Pathogens. Trends Immunol. 2018 May 1; 39(5): 367–379. PubMed Abstract | Publisher Full Text | Free Full Text
9. Barbosa CRR, Barton J, Shepherd AJ, et al.: Mechanistic diversity in MHC class I antigen recognition. Biochem. J. 2021 Dec 23; 478(24): 4187–4202. PubMed Abstract | Publisher Full Text | Free Full Text
10. Slade JWG, Watson MJ, MacDougall-Shackleton EA: “Balancing” balancing selection? Assortative mating at the major histocompatibility complex despite molecular signatures of balancing selection. Ecol. Evol. 2019 Apr 13; 9(9): 5146–5157. PubMed Abstract | Publisher Full Text | Free Full Text
11. Nguyen A, David JK, Maden SK, et al.: Human Leukocyte Antigen Susceptibility Map for Severe Acute Respiratory Syndrome Coronavirus 2. J. Virol. 2020 Apr 17 [cited 2022 Jul 19]; 94. PubMed Abstract | Publisher Full Text | Free Full Text
12. Mishto M, Mansurkhodzhaev A, Rodriguez-Calvo T, et al.: Potential Mimicry of Viral and Pancreatic β Cell Antigens Through Non-Spliced and cis-Spliced Zwitter Epitope Candidates in Type 1 Diabetes. Front. Immunol. 2021 [cited 2022 Sep 29]; 12. PubMed Abstract | Publisher Full Text | Free Full Text
13. Geneugelijk K, Thus KA, Spierings E: Predicting Alloreactivity in Transplantation. J. Immunol. Res. 2014 Apr 28; 2014: 1–12. Publisher Full Text
14. Hu Z, Ott PA, Wu CJ: Towards personalized, tumour-specific, therapeutic vaccines for cancer. Nat. Rev. Immunol. 2018 Mar; 18(3): 168–182. PubMed Abstract | Publisher Full Text | Free Full Text
15. Blass E, Ott PA: Advances in the development of personalized neoantigen-based therapeutic cancer vaccines. Nat. Rev. Clin. Oncol. 2021 Apr; 18(4): 215–229. PubMed Abstract | Publisher Full Text | Free Full Text
16. Nelde A, Maringer Y, Bilich T, et al.: Immunopeptidomics-Guided Warehouse Design for Peptide-Based Immunotherapy in Chronic Lymphocytic Leukemia. Front. Immunol. 2021 [cited 2022 Sep 30]; 12. PubMed Abstract | Publisher Full Text | Free Full Text
17. Terasaki M, Shibui S, Narita Y, et al.: Phase I trial of a personalized peptide vaccine for patients positive for human leukocyte antigen--A24 with recurrent or progressive glioblastoma multiforme. J. Clin. Oncol. Off. J. Am. Soc. Clin. Oncol. 2011 Jan 20; 29(3): 337–344. Publisher Full Text
18. Kibe S, Yutani S, Motoyama S, et al.: Phase II study of personalized peptide vaccination for previously treated advanced colorectal cancer. Cancer Immunol. Res. 2014 Dec; 2(12): 1154–1162. PubMed Abstract | Publisher Full Text
19. Bjerregaard AM, Nielsen M, Hadrup SR, et al.: MuPeXI: prediction of neo-epitopes from tumor sequencing data. Cancer Immunol. Immunother. CII. 2017 Sep; 66(9): 1123–1130. PubMed Abstract | Publisher Full Text
20. Wood MA, Nguyen A, Struck AJ, et al.: neoepiscope improves neoepitope prediction with multivariant phasing. Bioinformatics. 2020 Feb 1; 36(3): 713–720. PubMed Abstract | Publisher Full Text
21. Hundal J, Carreno BM, Petti AA, et al.: pVAC-Seq: A genome-guided in silico approach to identifying tumor neoantigens. Genome Med. 2016 Jan 29; 8(1): 11. PubMed Abstract | Publisher Full Text | Free Full Text
22. Bais P, Namburi S, Gatti DM, et al.: CloudNeo: a cloud pipeline for identifying patient-specific tumor neoantigens. Bioinforma. Oxf. Engl. 2017 Oct 1; 33(19): 3110–3112. PubMed Abstract | Publisher Full Text | Free Full Text
23. Reynisson B, Alvarez B, Paul S, et al.: NetMHCpan-4.1 and NetMHCIIpan-4.0: improved predictions of MHC antigen presentation by concurrent motif deconvolution and integration of MS MHC eluted ligand data. Nucleic Acids Res. 2020 Jul 2; 48(W1): W449–W454. PubMed Abstract | Publisher Full Text | Free Full Text
24. Jurtz V, Paul S, Andreatta M, et al.: NetMHCpan-4.0: Improved Peptide–MHC Class I Interaction Predictions Integrating Eluted Ligand and Peptide Binding Affinity Data. J. Immunol. 2017 Nov 1; 199(9): 3360–3368. PubMed Abstract | Publisher Full Text | Free Full Text
25. O’Donnell TJ, Rubinsteyn A, Laserson U: MHCflurry 2.0: Improved Pan-Allele Prediction of MHC Class I-Presented Peptides by Incorporating Antigen Processing. Cell Syst. 2020 Jul; 11(1): 42–48.e7. PubMed Abstract | Publisher Full Text
26. Shao XM, Bhattacharya R, Huang J, et al.: High-Throughput Prediction of MHC Class I and II Neoantigens with MHCnuggets. Cancer Immunol. Res. 2020; 8: 396–408. PubMed Abstract | Publisher Full Text | Free Full Text
27. Sarkizova S, Klaeger S, Le PM, et al.: A large peptidome dataset improves HLA class I epitope prediction across most of the human population. Nat. Biotechnol. 2020 Feb; 38(2): 199–209. PubMed Abstract | Publisher Full Text | Free Full Text
28. O’Leary NA, Wright MW, Brister JR, et al.: Reference sequence (RefSeq) database at NCBI: current status, taxonomic expansion, and functional annotation. Nucleic Acids Res. 2016 Jan 4; 44(D1): D733–D745. PubMed Abstract | Publisher Full Text | Free Full Text
29. Brister JR, Ako-Adjei D, Bao Y, et al.: NCBI viral genomes resource. Nucleic Acids Res. 2015 Jan; 43(Database issue): D571–D577. PubMed Abstract | Publisher Full Text | Free Full Text
30. Shao XM, Bhattacharya R, Huang J, et al.: High-throughput prediction of MHC class I and class II neoantigens with MHCnuggets. Cancer. Immunol. Res. 2019 Dec 23; canimm.0464.2019.
31. Sidney J, Peters B, Frahm N, et al.: HLA class I supertypes: a revised and updated classification. BMC Immunol. 2008 Jan 22; 9(1): 1. PubMed Abstract | Publisher Full Text | Free Full Text
32. Lide D: CRC handbook of chemistry and physics, 1992-1993: a ready-reference book of chemical and physical data.1992 [cited 2022 Sep 4]. Reference Source
33. Hackel M, Hinz HJ, Hedwig GR: A new set of peptide-based group heat capacities for use in protein stability calculations - ScienceDirect. [cited 2022 Sep 4]. Reference Source
34. Zhu C, Gao Y, Li H, et al.: Characterizing hydrophobicity of amino acid side chains in a protein environment via measuring contact angle of a water nanodroplet on planar peptide network. Proc. Natl. Acad. Sci. U. S. A. 2016 Nov 15; 113(46): 12946–12951. PubMed Abstract | Publisher Full Text | Free Full Text
35. Fogolari F, Corazza A, Fortuna S, et al.: Distance-Based Configurational Entropy of Proteins from Molecular Dynamics Simulations. PLoS One. 2015; 10(7): e0132356. PubMed Abstract | Publisher Full Text | Free Full Text
36. Kaluzny S original by DWSR port by AG adopted to recent SP by S. ash: David Scott’s ASH Routines.2015 [cited 2022 Jul 11]. Reference Source
37. Nguyen A: Underlying data for ‘Discordant results among MHC binding affinity prediction tools’. [Dataset]. Zenodo. 2023. Publisher Full Text
38. Shrout PE, Fleiss JL: Intraclass correlations: uses in assessing rater reliability. Psychol. Bull. 1979 Mar; 86(2): 420–428. Publisher Full Text
39. Nguyen A: Boeinco/peptide-MHCassess: F1000ReleaseV1 (Version F1000). [Code]. Zenodo. 2023; Publisher Full Text
40. Paul S, Weiskopf D, Angelo MA, et al.: HLA class I alleles are associated with peptide-binding repertoires of different size, affinity, and immunogenicity. J. Immunol. Baltim. Md. 1950. 2013 Dec 15; 191(12): 5831–5839. Publisher Full Text
41. Pavlos R, McKinnon EJ, Ostrov DA, et al.: Shared peptide binding of HLA Class I and II alleles associate with cutaneous nevirapine hypersensitivity and identify novel risk alleles. Sci. Rep. 2017 Aug 17; 7(1): 8653. PubMed Abstract | Publisher Full Text | Free Full Text
42. Abelin JG, Keskin DB, Sarkizova S, et al.: Mass Spectrometry Profiling of HLA-Associated Peptidomes in Mono-allelic Cells Enables More Accurate Epitope Prediction. Immunity. 2017 Feb; 46(2): 315–326. PubMed Abstract | Publisher Full Text | Free Full Text
43. Spurgin LG, Richardson DS: How pathogens drive genetic diversity: MHC, mechanisms and misunderstandings. Proc. R. Soc. B Biol. Sci. 2010 Apr 7; 277(1684): 979–988. PubMed Abstract | Publisher Full Text | Free Full Text
44. Manczinger M, Boross G, Kemény L, et al.: Pathogen diversity drives the evolution of generalist MHC-II alleles in human populations. PLoS Biol. 2019 Jan 31; 17(1): e3000131. PubMed Abstract | Publisher Full Text | Free Full Text
45. White CF, Pellis L, Keeling MJ, et al.: Detecting HLA-infectious disease associations for multi-strain pathogens. Infect. Genet. Evol. 2020 Sep 1; 83: 104344. PubMed Abstract | Publisher Full Text
46. Prugnolle F, Manica A, Charpentier M, et al.: Pathogen-Driven Selection and Worldwide HLA Class I Diversity. Curr. Biol. 2005 Jun 7; 15(11): 1022–1027. PubMed Abstract | Publisher Full Text
47. Roche PA, Furuta K: The ins and outs of MHC class II-mediated antigen processing and presentation. Nat. Rev. Immunol. 2015 Apr; 15(4): 203–216. PubMed Abstract | Publisher Full Text | Free Full Text
48. Weeder B, Wood MA, Li E, et al.: pepsickle rapidly and accurately predicts proteasomal cleavage sites for improved neoantigen identification. Bioinformatics|Oxford Academic. [cited 2022 Oct 2]; 37: 3723–3733. PubMed Abstract | Publisher Full Text Reference Source
49. Ritz U, Seliger B: The Transporter Associated With Antigen Processing (TAP): Structural Integrity, Expression, Function, and Its Clinical Relevance. Mol. Med. 2001 Mar; 7(3): 149–158. PubMed Abstract | Publisher Full Text | Free Full Text
50. López de Castro JA: How ERAP1 and ERAP2 Shape the Peptidomes of Disease-Associated MHC-I Proteins. Front. Immunol. 2018 [cited 2022 Oct 2]; 9. PubMed Abstract | Publisher Full Text | Free Full Text
51. Dincer AB, Lu Y, Schweppe DK, et al.: Reducing Peptide Sequence Bias in Quantitative Mass Spectrometry Data with Machine Learning. J. Proteome Res. 2022 Jul 1; 21(7): 1771–1782. PubMed Abstract | Publisher Full Text | Free Full Text
52. Edwards NJ: Novel peptide identification from tandem mass spectra using ESTs and sequence database compression. Mol. Syst. Biol. 2007 Jan; 3(1): 102. PubMed Abstract | Publisher Full Text | Free Full Text
53. Prakash A, Piening B, Whiteaker J, et al.: Assessing Bias in Experiment Design for Large Scale Mass Spectrometry-based Quantitative Proteomics. Mol. Cell. Proteomics. 2007 Oct 1; 6(10): 1741–1748. PubMed Abstract | Publisher Full Text
54. Timp W, Timp G: Beyond mass spectrometry, the next step in proteomics. Sci. Adv. 2020 Jan 10; 6(2): eaax8978. PubMed Abstract | Publisher Full Text | Free Full Text
55. Paul S, Croft NP, Purcell AW, et al.: Benchmarking predictions of MHC class I restricted T cell epitopes in a comprehensively studied model system. PLoS Comput. Biol. 2020 May 26; 16(5): e1007757. PubMed Abstract | Publisher Full Text | Free Full Text
56. Trolle T, Metushi IG, Greenbaum JA, et al.: Automated benchmarking of peptide-MHC class I binding predictions. Bioinformatics. 2015 Jul 1; 31(13): 2174–2181. PubMed Abstract | Publisher Full Text | Free Full Text
57. Zhao W, Sher X: Systematically benchmarking peptide-MHC binding predictors: From synthetic to naturally processed epitopes. PLoS Comput. Biol. 2018 Nov 8; 14(11): e1006457. PubMed Abstract | Publisher Full Text | Free Full Text
58. Venkatesh G, Grover A, Srinivasaraghavan G, et al.: MHCAttnNet: predicting MHC-peptide bindings for MHC alleles classes I and II using an attention-based deep neural model. Bioinformatics. 2020 Jul; 36(Suppl 1): i399–i406. PubMed Abstract | Publisher Full Text | Free Full Text
59. Bhattacharya R, Sivakumar A, Tokheim C, et al.: Evaluation of machine learning methods to predict peptide binding to MHC Class I proteins. bioRxiv. 2017 Jul 27; 154757.

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 07 Jun 2023

Author details Author details

¹ Department of Biomedical Engineering, Oregon Health & Science University, Portland, Oregon, 97239, USA
² Computational Biology, Oregon Health & Science University, Portland, Oregon, 97239, USA
³ Department of Surgery, Oregon Health & Science University, Portland, Oregon, 97239, USA
⁴ Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, Oregon, 97239, USA
⁵ Department of Radiation Medicine, Oregon Health & Science University, Portland, Oregon, 97239, USA
⁶ Division of Hospital and Specialty Medicine, VA Portland Healthcare System, Portland, Oregon, 97239, USA

Austin Nguyen
Roles: Conceptualization, Data Curation, Formal Analysis, Methodology, Project Administration, Resources, Software, Validation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing

Abhinav Nellore
Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Review & Editing

Reid F. Thompson
Roles: Conceptualization, Funding Acquisition, Investigation, Project Administration, Supervision, Writing – Review & Editing

Competing interests

No competing interests were disclosed.

Grant information

The author(s) declared that no grants were involved in supporting this work.

Article Versions (1)

version 1

Published: 07 Jun 2023, 12:617

https://doi.org/10.12688/f1000research.132538.1

Copyright

© 2023 Nguyen A et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download

Export To

metrics

	Views	Downloads
F1000Research	-	-
PubMed Central Data from PMC are received and updated monthly.	-	-

Citations

0

SEE MORE DETAILS

CITE

how to cite this article

Nguyen A, Nellore A and Thompson RF. Discordant results among major histocompatibility complex binding affinity prediction tools [version 1; peer review: 2 approved with reservations, 1 not approved]. F1000Research 2023, 12:617 (https://doi.org/10.12688/f1000research.132538.1)

NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.

track

receive updates on this article

Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?

Key to Reviewer Statuses VIEW HIDE

ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested

Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.

Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions

Version 1

VERSION 1

PUBLISHED 07 Jun 2023

Views

26

Reviewer Report 20 Nov 2023

Nikolaos G Sgourakis, University of Pennsylvania, Philadelphia, Pennsylvania, USA

Sagar Gupta, University of Pennsylvania, Philadelphia, Pennsylvania, USA

Not Approved

https://doi.org/10.5256/f1000research.145468.r210811

This body of work presents a comparison of four peptide/HLA binding prediction tools. The authors evaluate the consistency in binding predictions for netMHCPan, HLAthena, MHCflurry, and MHCnuggets initially across a set of one million random peptides of varying lengths to ... Continue reading

This body of work presents a comparison of four peptide/HLA binding prediction tools. The authors evaluate the consistency in binding predictions for netMHCPan, HLAthena, MHCflurry, and MHCnuggets initially across a set of one million random peptides of varying lengths to 52 alleles. After discovering that these methods do not converge in their predictions, they attempt to rationalize differences based on the amount of training data used for each allele. Next, Nguyen et al. investigated the allelic promiscuity by comparing binding to human, viral, and random peptides for all four tools. They again found inconsistencies between the methods. The authors found that the difference in physical properties of the peptides roughly correlated with predicted allelic promiscuity. Finally, they showed that alleles have different preferences for the peptides they are predicted to bind. Overall, this manuscript supports that popular, well-established tools to predict binding between peptides and HLAs are inconsistent, suggesting their inaccuracy.

Major issues:

What is “kmerization”? This concept is not explained clearly in the manuscript.
How many human and viral peptides were evaluated and what is the breakdown per virus?
Figure 2C is referenced in the article and described in the figure legend, but the plot is not in Figure 2 itself. Thus, the conclusions made in the main text cannot be evaluated.
As different alleles have a variable binding affinity that can be considered “strong”, is there a particular reason why predicted binding affinity is converted to the binding score rather than converting a percentile-based value as given in NetMHCPan? Also, the rationale for converting MHCflurry and MHCnuggets values using the “1- log (binding affinity) /log(50000)” equation is not clear.
How was the training data estimated for netMHCPan? The legend of Figure 4 does not provide any details on how this prediction was conducted.
The correlation coefficients of the best-fit lines shown in Figure 6 should be provided so readers can understand how strong the “negative correlation” indicated in the text is.
It is not clear which method is being used to predict binders in Figure 7 and Supplementary Figure 8. Additionally, throughout the manuscript, the authors maintain that the four tools they tested are inconsistent, yet likely rely on these same tools to draw conclusions on allelic binding preferences.
To advance the field, the authors should consider proposing potential solutions to improving these tools or at least provide a brief summary of newer, less popular tools and potential improvements, if any, have been made/different approaches that have been pursued. An example is Motmaen et al., PNAS 2023 who use a structure-based prediction method and see remarkable accuracy beyond pHLAs.

Other comments:

Abbreviations should be clarified explicitly on first use. For instance, in the ‘Sequence retrieval, peptide filtering, and kmerization’ of Methods, it is unclear what ‘BK’ stands for.
The exact value pertaining to each dotted line in Figure 2B should be stated as was done in Figure 2A.
In Supplementary Figure 1, the boxplots need to be defined in the legends in terms of minima, maxima, center, bounds of box and whiskers, and percentile.
What do the red point, black circle, and red line refer to in the plots in Figure 3 and in Supplementary Figures 2 and 3?
The number of alleles should be delineated in Figure 4 for clarity.
The number of peptides tested should be made clear in Supplementary Figure 3.
Panel letters are missing in Supplementary Figure 4. The number of peptides in each column should be indicated in the legend.
The number of peptides evaluated should be listed in the legend of Figure 5.
The alleles analyzed in Supplementary Figure 5 should be provided in a Supplementary Table.
The number of peptides analyzed in each subplot of Supplementary Figure 6 should be delineated. Also, it is not clear what these plots are showing – are they the result of dimension reductionality? What are the x and y-axes?
The number of peptides analyzed in each plot should be delineated in Supplementary Figure 7.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Molecular Immunology, Bioinformatics

We confirm that we have read this submission and believe that we have an appropriate level of expertise to state that we do not consider it to be of an acceptable scientific standard, for reasons outlined above.

CITE

Report a concern

Respond or Comment

Views

20

Reviewer Report 01 Nov 2023

Michael E Birnbaum, Koch Institute for Integrative Cancer Research, Cambridge, MA, USA

Approved with Reservations

https://doi.org/10.5256/f1000research.145468.r210784

In the manuscript “Discordant results among major histocompatibility complex binding affinity prediction tools”, Nguyen and colleagues discuss the discordances when four MHC binding tools, netMHCpan, HLAthena, MHCflurry, and MHCnuggets, are compared against one another. Surprisingly, despite being similarly designed and ... Continue reading

In the manuscript “Discordant results among major histocompatibility complex binding affinity prediction tools”, Nguyen and colleagues discuss the discordances when four MHC binding tools, netMHCpan, HLAthena, MHCflurry, and MHCnuggets, are compared against one another. Surprisingly, despite being similarly designed and trained, the authors report large discordances in the results of the tools. This raises important questions about where these discrepancies come from, as such predictions are regularly used for the study of immunology and the design of immunogens.

While this work is of interest, I think there are areas that would help better contextualize these results:

From Figures 3-4, it seems like a large source of the discordances are from MHCnuggets. This is a relatively newer tool. It would be interesting to understand exactly why this difference is occurring here. It is especially curious that random peptides are not predicted to bind at all for some alleles.
More generally, I am somewhat curious/concerned about setting a binding probability of 0.5 as a cut-off for assessing tools, especially when some of the tools need their output transformed to match this metric. It is possible this is an issue regarding scaling of predictions rather than totally missing predictions. What happens if the threshold is changed from 0.5?
It would be interesting to understand more about the sequence properties of peptides that are more universally well predicted, vs. those that were only predicted as binders for certain tools. Seeing these as (for example) sequence logos would be helpful. A comparison to orthogonal experimental data would be even better, but understood that may be hard to come by.
I think the authors somewhat overstate the importance of sequence differences between self and pathogen peptide sequences (such as the conclusion) – since the motifs that demarcate peptide binding to MHC molecules tend to be relatively minimal (certain residue preferences as P2 and P9/10, with contributions possible at positions such as P3), there are not likely to be strong signals for those that bind to self peptides vs those that bind to pathogen peptides, especially when taking into account the common constraints set by protein secondary structure (alpha helices, beta sheets) are common for all phyla of life.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

I cannot comment. A qualified statistician is required.
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests: I am a founder, consultant and equity holder of Kelonia Therapeutics and Abata Therapeutics, an equity holder in 3T Biosciences, hold patents on technologies related to pMHC-TCR discovery, and have received consulting fees from Repertoire Immune Medicines.

Reviewer Expertise: I am an immunologist studying pMHC-TCR interactions

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Respond or Comment

Views

22

Reviewer Report 13 Jul 2023

André Leier, Department of Genetics, School of Medicine, The University of Alabama at Birmingham, Birmingham, Alabama, USA

Approved with Reservations

https://doi.org/10.5256/f1000research.145468.r177522

The work by Nguyen et al. compares outputs of four ML-based predictors of MHC binding affinity, namely NetMHCpan (2015), HLAthena, MHCflurry (2018), and MHCnuggets (2017), which, according to the authors, are widely used in investigational and therapeutic applications. Their comparison ... Continue reading

The work by Nguyen et al. compares outputs of four ML-based predictors of MHC binding affinity, namely NetMHCpan (2015), HLAthena, MHCflurry (2018), and MHCnuggets (2017), which, according to the authors, are widely used in investigational and therapeutic applications. Their comparison is based on predictions involving human, viral, and randomly generated peptide sources and 52 MHC class I alleles.

To assess the consistency of these prediction tools the authors present results from various analyses, such as:

(a) an intraclass correlation coefficient analysis,

(b) tool-vs-tool comparison of predicted binder numbers per allele, Spearman correlations between tools based on peptide numbers predicted to bind to all alleles, and each tool’s HLA allelic presentation obtained per allele (for 1 million random peptides),

(c) scatterplot analysis of the relationship between size of training data and consistency of predictions,

(d) distributions of predicted allelic promiscuity across alleles and correlation between peptide sources (human, viral, random),

(e) correlations between peptide physical property similarity and peptide binding similarity for each tool, and

(f) differential distributions of UMAP-transformed physical properties of peptides predicted to bind to specific HLA alleles or, combined, for all alleles.

In summary, the data shows convincingly the inconsistencies in prediction outcomes between the tools for the given MHC class I alleles. However, the manuscript has also some weaknesses.

Criticisms:

The authors compare four popular tools. However, there are many other and also newer tools available: NetMHCcons (2012) is a tool based on a consensus approach including 4(?) individual predictors. Its already older but has still been very competitive for 10- and 11-mer MHC class I binders (Prediction of Major Histocompatibility Complex Binding with Bilateral and Variable Long Short Term Memory Networks, Jiang et al., 2022). NetMHCpan 4.0 (2017) learns from mass spec data, which has improved the prediction accuracy compared to earlier versions. Recently, several deep learning methods have been developed, which perform better than previous neural networks: AI-MHC (2018), a deep learning architecture for human Class I and Class II MHC binding prediction, BVLSTM-MHC (2021) and BVMHC (2022) (Prediction of Major Histocompatibility Complex Binding with Bilateral and Variable Long Short Term Memory Networks, Jiang et al., 2022). I am sure the reader would be interested in having these included in the comparison. The manuscript should cite these and other (especially newer) predictors. That said, comparing even more tools could further reduce the number of alleles that the tools have in common. Lastly, to be fair, the authors acknowledge this point in their discussion stating that they deliberately chose four of the most widely used tools.
To compare the four tools, the authors must convert binding affinities used by MHCflurry and MHCnuggets into binding probabilities used by NetMHCpan and HLAthena. It is not clear how the formula 1- log (binding affinity) /log(50000) was derived. What does it mean that binding affinities were converted “in order to match [..] binding probability predictions”?
The main text refers to a subfigure 2C. The caption to figure 2 describes 2C - but there is no subfigure C in the figure!
It is not clear which prediction tool/data was used for obtaining allele-specific differential distributions of physical properties (Fig. 7). The authors show that different alleles have different preferences for peptide physical properties but is this distribution similar across the four tools? What is the conclusion of this observation (e.g., what does this mean for the consistency of predictions)?
The discussion only summarizes the limitations of the work. The conclusions lists questions that this work raises. In both sections I would have liked to see more content. What are the implications of your study? This brings me to my last point:
While I find this analysis interesting, I am a bit at loss what the take home message is. Should we not use any of the tools, or use all as part of a new consensus approach? Which tool is better/more accurate? This raises the question if there is not a better benchmark test for MHC binding affinity predictors? Why not use experimentally verified MHC binding peptides? The four tools are already at least 6 years old. Would it not make sense to compare the tools based on allele-peptide pairs (or just peptides) that the ML/DL algorithm has not been trained on? By the same token, would it not make sense to base the promiscuity analysis on verified bindings rather than predictions of tools that are obviously inconsistent? These and other questions could be discussed also in the discussion/conclusion section.

Minor issues:

The authors write “as well as quantify the stability of these predictions”. It is not entirely clear what the stability of a prediction is. Do the authors mean accuracy or consistency?
Could the authors provide the list of the 52 alleles that all four tools have in common?
Sentence: “For each peptide source, alleles were ranked from best to worst binders (most to least peptides ≥ 0.5 score) t.” -> Remove t.
“and the allele lowest mean binding”-> “and the alleles with the lowest mean binding”.
Does the entire analysis presented here, in particular the investigation of allelic promiscuity and the correlation between peptide sources, is based only on the 52 alleles that all four tools have in common?
Fig. 6, …”is shown on the x-axis, whereas Spearman correlation coefficients of predicted allelic promiscuity across alleles.” -> “is shown on the y-axis” is missing.

Note: I have not tested the source code.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Bioinformatics, Computational Biology, Machine learning, Gene-targeted Therapies, RNA Biochemistry

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

CITE

Report a concern

Respond or Comment

Comments on this article Comments (0)

Version 1

VERSION 1 PUBLISHED 07 Jun 2023

Open Peer Review

Reviewer Status

Reviewer Reports

	Invited Reviewers
	1	2	3
Version 1 07 Jun 23	read	read	read

André Leier, The University of Alabama at Birmingham, Birmingham, USA
Michael E Birnbaum, Koch Institute for Integrative Cancer Research, Cambridge, USA
Nikolaos G Sgourakis, University of Pennsylvania, Philadelphia, USA

Sagar Gupta, University of Pennsylvania, Philadelphia, USA

Comments on this article

All Comments(0)

Add a comment

Sign up for content alerts

Browse by related subjects

Back to all reports

Reviewer Report

26 Views

20 Nov 2023 | for Version 1

Nikolaos G Sgourakis, University of Pennsylvania, Philadelphia, Pennsylvania, USA

Sagar Gupta, University of Pennsylvania, Philadelphia, Pennsylvania, USA

26 Views Cite this report Responses(0)

Not Approved

This body of work presents a comparison of four peptide/HLA binding prediction tools. The authors evaluate the consistency in binding predictions for netMHCPan, HLAthena, MHCflurry, and MHCnuggets initially across a set of one million random peptides of varying lengths to 52 alleles. After discovering that these methods do not converge in their predictions, they attempt to rationalize differences based on the amount of training data used for each allele. Next, Nguyen et al. investigated the allelic promiscuity by comparing binding to human, viral, and random peptides for all four tools. They again found inconsistencies between the methods. The authors found that the difference in physical properties of the peptides roughly correlated with predicted allelic promiscuity. Finally, they showed that alleles have different preferences for the peptides they are predicted to bind. Overall, this manuscript supports that popular, well-established tools to predict binding between peptides and HLAs are inconsistent, suggesting their inaccuracy.

Major issues:

What is “kmerization”? This concept is not explained clearly in the manuscript.
How many human and viral peptides were evaluated and what is the breakdown per virus?
Figure 2C is referenced in the article and described in the figure legend, but the plot is not in Figure 2 itself. Thus, the conclusions made in the main text cannot be evaluated.
As different alleles have a variable binding affinity that can be considered “strong”, is there a particular reason why predicted binding affinity is converted to the binding score rather than converting a percentile-based value as given in NetMHCPan? Also, the rationale for converting MHCflurry and MHCnuggets values using the “1- log (binding affinity) /log(50000)” equation is not clear.
How was the training data estimated for netMHCPan? The legend of Figure 4 does not provide any details on how this prediction was conducted.
The correlation coefficients of the best-fit lines shown in Figure 6 should be provided so readers can understand how strong the “negative correlation” indicated in the text is.
It is not clear which method is being used to predict binders in Figure 7 and Supplementary Figure 8. Additionally, throughout the manuscript, the authors maintain that the four tools they tested are inconsistent, yet likely rely on these same tools to draw conclusions on allelic binding preferences.
To advance the field, the authors should consider proposing potential solutions to improving these tools or at least provide a brief summary of newer, less popular tools and potential improvements, if any, have been made/different approaches that have been pursued. An example is Motmaen et al., PNAS 2023 who use a structure-based prediction method and see remarkable accuracy beyond pHLAs.

Other comments:

Abbreviations should be clarified explicitly on first use. For instance, in the ‘Sequence retrieval, peptide filtering, and kmerization’ of Methods, it is unclear what ‘BK’ stands for.
The exact value pertaining to each dotted line in Figure 2B should be stated as was done in Figure 2A.
In Supplementary Figure 1, the boxplots need to be defined in the legends in terms of minima, maxima, center, bounds of box and whiskers, and percentile.
What do the red point, black circle, and red line refer to in the plots in Figure 3 and in Supplementary Figures 2 and 3?
The number of alleles should be delineated in Figure 4 for clarity.
The number of peptides tested should be made clear in Supplementary Figure 3.
Panel letters are missing in Supplementary Figure 4. The number of peptides in each column should be indicated in the legend.
The number of peptides evaluated should be listed in the legend of Figure 5.
The alleles analyzed in Supplementary Figure 5 should be provided in a Supplementary Table.
The number of peptides analyzed in each subplot of Supplementary Figure 6 should be delineated. Also, it is not clear what these plots are showing – are they the result of dimension reductionality? What are the x and y-axes?
The number of peptides analyzed in each plot should be delineated in Supplementary Figure 7.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Partly
Are sufficient details of methods and analysis provided to allow replication by others?

Partly
If applicable, is the statistical analysis and its interpretation appropriate?

Partly
Are all the source data underlying the results available to ensure full reproducibility?

Partly
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Molecular Immunology, Bioinformatics

We confirm that we have read this submission and believe that we have an appropriate level of expertise to state that we do not consider it to be of an acceptable scientific standard, for reasons outlined above.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

20 Views

01 Nov 2023 | for Version 1

Michael E Birnbaum, Koch Institute for Integrative Cancer Research, Cambridge, MA, USA

20 Views Cite this report Responses(0)

Approved With Reservations

In the manuscript “Discordant results among major histocompatibility complex binding affinity prediction tools”, Nguyen and colleagues discuss the discordances when four MHC binding tools, netMHCpan, HLAthena, MHCflurry, and MHCnuggets, are compared against one another. Surprisingly, despite being similarly designed and trained, the authors report large discordances in the results of the tools. This raises important questions about where these discrepancies come from, as such predictions are regularly used for the study of immunology and the design of immunogens.

While this work is of interest, I think there are areas that would help better contextualize these results:

From Figures 3-4, it seems like a large source of the discordances are from MHCnuggets. This is a relatively newer tool. It would be interesting to understand exactly why this difference is occurring here. It is especially curious that random peptides are not predicted to bind at all for some alleles.
More generally, I am somewhat curious/concerned about setting a binding probability of 0.5 as a cut-off for assessing tools, especially when some of the tools need their output transformed to match this metric. It is possible this is an issue regarding scaling of predictions rather than totally missing predictions. What happens if the threshold is changed from 0.5?
It would be interesting to understand more about the sequence properties of peptides that are more universally well predicted, vs. those that were only predicted as binders for certain tools. Seeing these as (for example) sequence logos would be helpful. A comparison to orthogonal experimental data would be even better, but understood that may be hard to come by.
I think the authors somewhat overstate the importance of sequence differences between self and pathogen peptide sequences (such as the conclusion) – since the motifs that demarcate peptide binding to MHC molecules tend to be relatively minimal (certain residue preferences as P2 and P9/10, with contributions possible at positions such as P3), there are not likely to be strong signals for those that bind to self peptides vs those that bind to pathogen peptides, especially when taking into account the common constraints set by protein secondary structure (alpha helices, beta sheets) are common for all phyla of life.

Is the work clearly and accurately presented and does it cite the current literature?

Yes
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

I cannot comment. A qualified statistician is required.
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Partly

Competing Interests

I am a founder, consultant and equity holder of Kelonia Therapeutics and Abata Therapeutics, an equity holder in 3T Biosciences, hold patents on technologies related to pMHC-TCR discovery, and have received consulting fees from Repertoire Immune Medicines.

Reviewer Expertise

I am an immunologist studying pMHC-TCR interactions

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Respond to this report

Responses (0)

Back to all reports

Reviewer Report

22 Views

13 Jul 2023 | for Version 1

André Leier, Department of Genetics, School of Medicine, The University of Alabama at Birmingham, Birmingham, Alabama, USA

22 Views Cite this report Responses(0)

Approved With Reservations

The work by Nguyen et al. compares outputs of four ML-based predictors of MHC binding affinity, namely NetMHCpan (2015), HLAthena, MHCflurry (2018), and MHCnuggets (2017), which, according to the authors, are widely used in investigational and therapeutic applications. Their comparison is based on predictions involving human, viral, and randomly generated peptide sources and 52 MHC class I alleles.

To assess the consistency of these prediction tools the authors present results from various analyses, such as:

(a) an intraclass correlation coefficient analysis,

(b) tool-vs-tool comparison of predicted binder numbers per allele, Spearman correlations between tools based on peptide numbers predicted to bind to all alleles, and each tool’s HLA allelic presentation obtained per allele (for 1 million random peptides),

(c) scatterplot analysis of the relationship between size of training data and consistency of predictions,

(d) distributions of predicted allelic promiscuity across alleles and correlation between peptide sources (human, viral, random),

(e) correlations between peptide physical property similarity and peptide binding similarity for each tool, and

(f) differential distributions of UMAP-transformed physical properties of peptides predicted to bind to specific HLA alleles or, combined, for all alleles.

In summary, the data shows convincingly the inconsistencies in prediction outcomes between the tools for the given MHC class I alleles. However, the manuscript has also some weaknesses.

Criticisms:

The authors compare four popular tools. However, there are many other and also newer tools available: NetMHCcons (2012) is a tool based on a consensus approach including 4(?) individual predictors. Its already older but has still been very competitive for 10- and 11-mer MHC class I binders (Prediction of Major Histocompatibility Complex Binding with Bilateral and Variable Long Short Term Memory Networks, Jiang et al., 2022). NetMHCpan 4.0 (2017) learns from mass spec data, which has improved the prediction accuracy compared to earlier versions. Recently, several deep learning methods have been developed, which perform better than previous neural networks: AI-MHC (2018), a deep learning architecture for human Class I and Class II MHC binding prediction, BVLSTM-MHC (2021) and BVMHC (2022) (Prediction of Major Histocompatibility Complex Binding with Bilateral and Variable Long Short Term Memory Networks, Jiang et al., 2022). I am sure the reader would be interested in having these included in the comparison. The manuscript should cite these and other (especially newer) predictors. That said, comparing even more tools could further reduce the number of alleles that the tools have in common. Lastly, to be fair, the authors acknowledge this point in their discussion stating that they deliberately chose four of the most widely used tools.
To compare the four tools, the authors must convert binding affinities used by MHCflurry and MHCnuggets into binding probabilities used by NetMHCpan and HLAthena. It is not clear how the formula 1- log (binding affinity) /log(50000) was derived. What does it mean that binding affinities were converted “in order to match [..] binding probability predictions”?
The main text refers to a subfigure 2C. The caption to figure 2 describes 2C - but there is no subfigure C in the figure!
It is not clear which prediction tool/data was used for obtaining allele-specific differential distributions of physical properties (Fig. 7). The authors show that different alleles have different preferences for peptide physical properties but is this distribution similar across the four tools? What is the conclusion of this observation (e.g., what does this mean for the consistency of predictions)?
The discussion only summarizes the limitations of the work. The conclusions lists questions that this work raises. In both sections I would have liked to see more content. What are the implications of your study? This brings me to my last point:
While I find this analysis interesting, I am a bit at loss what the take home message is. Should we not use any of the tools, or use all as part of a new consensus approach? Which tool is better/more accurate? This raises the question if there is not a better benchmark test for MHC binding affinity predictors? Why not use experimentally verified MHC binding peptides? The four tools are already at least 6 years old. Would it not make sense to compare the tools based on allele-peptide pairs (or just peptides) that the ML/DL algorithm has not been trained on? By the same token, would it not make sense to base the promiscuity analysis on verified bindings rather than predictions of tools that are obviously inconsistent? These and other questions could be discussed also in the discussion/conclusion section.

Minor issues:

The authors write “as well as quantify the stability of these predictions”. It is not entirely clear what the stability of a prediction is. Do the authors mean accuracy or consistency?
Could the authors provide the list of the 52 alleles that all four tools have in common?
Sentence: “For each peptide source, alleles were ranked from best to worst binders (most to least peptides ≥ 0.5 score) t.” -> Remove t.
“and the allele lowest mean binding”-> “and the alleles with the lowest mean binding”.
Does the entire analysis presented here, in particular the investigation of allelic promiscuity and the correlation between peptide sources, is based only on the 52 alleles that all four tools have in common?
Fig. 6, …”is shown on the x-axis, whereas Spearman correlation coefficients of predicted allelic promiscuity across alleles.” -> “is shown on the y-axis” is missing.

Note: I have not tested the source code.

Is the work clearly and accurately presented and does it cite the current literature?

Partly
Is the study design appropriate and is the work technically sound?

Yes
Are sufficient details of methods and analysis provided to allow replication by others?

Yes
If applicable, is the statistical analysis and its interpretation appropriate?

Yes
Are all the source data underlying the results available to ensure full reproducibility?

Yes
Are the conclusions drawn adequately supported by the results?

Yes

Competing Interests

No competing interests were disclosed.

Reviewer Expertise

Bioinformatics, Computational Biology, Machine learning, Gene-targeted Therapies, RNA Biochemistry

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Respond to this report

Responses (0)

[1] 1. Dendrou CA, Petersen J, Rossjohn J, et al.: HLA variation and disease. Nat. Rev. Immunol. 2018 May; 18(5): 325–339. Publisher Full Text

[2] 2. Blackwell JM, Jamieson SE, Burgner D: HLA and Infectious Diseases. Clin. Microbiol. Rev. 2009 Apr; 22(2): 370–385. PubMed Abstract | Publisher Full Text | Free Full Text

[3] 3. Meyer D, Aguiar VRC, Bitarello BD, et al.: A genomic perspective on HLA evolution. Immunogenetics. 2018; 70(1): 5–27. PubMed Abstract | Publisher Full Text | Free Full Text

[4] 4. Zernich D, Purcell AW, Macdonald WA, et al.: Natural HLA Class I Polymorphism Controls the Pathway of Antigen Presentation and Susceptibility to Viral Evasion. J. Exp. Med. 2004 Jun 28; 200(1): 13–24. PubMed Abstract | Publisher Full Text | Free Full Text

[5] 5. Bihl F, Frahm N, Giammarino LD, et al.: Impact of HLA-B Alleles, Epitope Binding Affinity, Functional Avidity, and Viral Coinfection on the Immunodominance of Virus-Specific CTL Responses. J. Immunol. 2006 Apr 1; 176(7): 4094–4101. PubMed Abstract | Publisher Full Text

[6] 6. Berger CT, Carlson JM, Brumme CJ, et al.: Viral adaptation to immune selection pressure by HLA class I–restricted CTL responses targeting epitopes in HIV frameshift sequences. J. Exp. Med. 2010 Jan 18; 207(1): 61–75. PubMed Abstract | Publisher Full Text | Free Full Text

[7] 7. Schellens IM, Meiring HD, Hoof I, et al.: Measles Virus Epitope Presentation by HLA: Novel Insights into Epitope Selection, Dominance, and Microvariation. Front. Immunol. 2015 [cited 2019 Nov 15]; 6. PubMed Abstract | Publisher Full Text | Free Full Text

[8] 8. Kaufman J: Generalists and Specialists: A New View of How MHC Class I Molecules Fight Infectious Pathogens. Trends Immunol. 2018 May 1; 39(5): 367–379. PubMed Abstract | Publisher Full Text | Free Full Text

[9] 9. Barbosa CRR, Barton J, Shepherd AJ, et al.: Mechanistic diversity in MHC class I antigen recognition. Biochem. J. 2021 Dec 23; 478(24): 4187–4202. PubMed Abstract | Publisher Full Text | Free Full Text

[10] 10. Slade JWG, Watson MJ, MacDougall-Shackleton EA: “Balancing” balancing selection? Assortative mating at the major histocompatibility complex despite molecular signatures of balancing selection. Ecol. Evol. 2019 Apr 13; 9(9): 5146–5157. PubMed Abstract | Publisher Full Text | Free Full Text

[11] 11. Nguyen A, David JK, Maden SK, et al.: Human Leukocyte Antigen Susceptibility Map for Severe Acute Respiratory Syndrome Coronavirus 2. J. Virol. 2020 Apr 17 [cited 2022 Jul 19]; 94. PubMed Abstract | Publisher Full Text | Free Full Text

[12] 12. Mishto M, Mansurkhodzhaev A, Rodriguez-Calvo T, et al.: Potential Mimicry of Viral and Pancreatic β Cell Antigens Through Non-Spliced and cis-Spliced Zwitter Epitope Candidates in Type 1 Diabetes. Front. Immunol. 2021 [cited 2022 Sep 29]; 12. PubMed Abstract | Publisher Full Text | Free Full Text

[13] 13. Geneugelijk K, Thus KA, Spierings E: Predicting Alloreactivity in Transplantation. J. Immunol. Res. 2014 Apr 28; 2014: 1–12. Publisher Full Text

[14] 14. Hu Z, Ott PA, Wu CJ: Towards personalized, tumour-specific, therapeutic vaccines for cancer. Nat. Rev. Immunol. 2018 Mar; 18(3): 168–182. PubMed Abstract | Publisher Full Text | Free Full Text

[15] 15. Blass E, Ott PA: Advances in the development of personalized neoantigen-based therapeutic cancer vaccines. Nat. Rev. Clin. Oncol. 2021 Apr; 18(4): 215–229. PubMed Abstract | Publisher Full Text | Free Full Text

[16] 16. Nelde A, Maringer Y, Bilich T, et al.: Immunopeptidomics-Guided Warehouse Design for Peptide-Based Immunotherapy in Chronic Lymphocytic Leukemia. Front. Immunol. 2021 [cited 2022 Sep 30]; 12. PubMed Abstract | Publisher Full Text | Free Full Text

[17] 17. Terasaki M, Shibui S, Narita Y, et al.: Phase I trial of a personalized peptide vaccine for patients positive for human leukocyte antigen--A24 with recurrent or progressive glioblastoma multiforme. J. Clin. Oncol. Off. J. Am. Soc. Clin. Oncol. 2011 Jan 20; 29(3): 337–344. Publisher Full Text

[18] 18. Kibe S, Yutani S, Motoyama S, et al.: Phase II study of personalized peptide vaccination for previously treated advanced colorectal cancer. Cancer Immunol. Res. 2014 Dec; 2(12): 1154–1162. PubMed Abstract | Publisher Full Text

[19] 19. Bjerregaard AM, Nielsen M, Hadrup SR, et al.: MuPeXI: prediction of neo-epitopes from tumor sequencing data. Cancer Immunol. Immunother. CII. 2017 Sep; 66(9): 1123–1130. PubMed Abstract | Publisher Full Text

[20] 20. Wood MA, Nguyen A, Struck AJ, et al.: neoepiscope improves neoepitope prediction with multivariant phasing. Bioinformatics. 2020 Feb 1; 36(3): 713–720. PubMed Abstract | Publisher Full Text

[21] 21. Hundal J, Carreno BM, Petti AA, et al.: pVAC-Seq: A genome-guided in silico approach to identifying tumor neoantigens. Genome Med. 2016 Jan 29; 8(1): 11. PubMed Abstract | Publisher Full Text | Free Full Text

[22] 22. Bais P, Namburi S, Gatti DM, et al.: CloudNeo: a cloud pipeline for identifying patient-specific tumor neoantigens. Bioinforma. Oxf. Engl. 2017 Oct 1; 33(19): 3110–3112. PubMed Abstract | Publisher Full Text | Free Full Text

[23] 23. Reynisson B, Alvarez B, Paul S, et al.: NetMHCpan-4.1 and NetMHCIIpan-4.0: improved predictions of MHC antigen presentation by concurrent motif deconvolution and integration of MS MHC eluted ligand data. Nucleic Acids Res. 2020 Jul 2; 48(W1): W449–W454. PubMed Abstract | Publisher Full Text | Free Full Text

[24] 24. Jurtz V, Paul S, Andreatta M, et al.: NetMHCpan-4.0: Improved Peptide–MHC Class I Interaction Predictions Integrating Eluted Ligand and Peptide Binding Affinity Data. J. Immunol. 2017 Nov 1; 199(9): 3360–3368. PubMed Abstract | Publisher Full Text | Free Full Text

[25] 25. O’Donnell TJ, Rubinsteyn A, Laserson U: MHCflurry 2.0: Improved Pan-Allele Prediction of MHC Class I-Presented Peptides by Incorporating Antigen Processing. Cell Syst. 2020 Jul; 11(1): 42–48.e7. PubMed Abstract | Publisher Full Text

[26] 26. Shao XM, Bhattacharya R, Huang J, et al.: High-Throughput Prediction of MHC Class I and II Neoantigens with MHCnuggets. Cancer Immunol. Res. 2020; 8: 396–408. PubMed Abstract | Publisher Full Text | Free Full Text

[27] 27. Sarkizova S, Klaeger S, Le PM, et al.: A large peptidome dataset improves HLA class I epitope prediction across most of the human population. Nat. Biotechnol. 2020 Feb; 38(2): 199–209. PubMed Abstract | Publisher Full Text | Free Full Text

[28] 28. O’Leary NA, Wright MW, Brister JR, et al.: Reference sequence (RefSeq) database at NCBI: current status, taxonomic expansion, and functional annotation. Nucleic Acids Res. 2016 Jan 4; 44(D1): D733–D745. PubMed Abstract | Publisher Full Text | Free Full Text

[29] 29. Brister JR, Ako-Adjei D, Bao Y, et al.: NCBI viral genomes resource. Nucleic Acids Res. 2015 Jan; 43(Database issue): D571–D577. PubMed Abstract | Publisher Full Text | Free Full Text

[30] 30. Shao XM, Bhattacharya R, Huang J, et al.: High-throughput prediction of MHC class I and class II neoantigens with MHCnuggets. Cancer. Immunol. Res. 2019 Dec 23; canimm.0464.2019.

[31] 31. Sidney J, Peters B, Frahm N, et al.: HLA class I supertypes: a revised and updated classification. BMC Immunol. 2008 Jan 22; 9(1): 1. PubMed Abstract | Publisher Full Text | Free Full Text

[32] 32. Lide D: CRC handbook of chemistry and physics, 1992-1993: a ready-reference book of chemical and physical data.1992 [cited 2022 Sep 4]. Reference Source

[33] 33. Hackel M, Hinz HJ, Hedwig GR: A new set of peptide-based group heat capacities for use in protein stability calculations - ScienceDirect. [cited 2022 Sep 4]. Reference Source

[34] 34. Zhu C, Gao Y, Li H, et al.: Characterizing hydrophobicity of amino acid side chains in a protein environment via measuring contact angle of a water nanodroplet on planar peptide network. Proc. Natl. Acad. Sci. U. S. A. 2016 Nov 15; 113(46): 12946–12951. PubMed Abstract | Publisher Full Text | Free Full Text

[35] 35. Fogolari F, Corazza A, Fortuna S, et al.: Distance-Based Configurational Entropy of Proteins from Molecular Dynamics Simulations. PLoS One. 2015; 10(7): e0132356. PubMed Abstract | Publisher Full Text | Free Full Text

[36] 36. Kaluzny S original by DWSR port by AG adopted to recent SP by S. ash: David Scott’s ASH Routines.2015 [cited 2022 Jul 11]. Reference Source

[37] 37. Nguyen A: Underlying data for ‘Discordant results among MHC binding affinity prediction tools’. [Dataset]. Zenodo. 2023. Publisher Full Text

[38] 38. Shrout PE, Fleiss JL: Intraclass correlations: uses in assessing rater reliability. Psychol. Bull. 1979 Mar; 86(2): 420–428. Publisher Full Text

[39] 39. Nguyen A: Boeinco/peptide-MHCassess: F1000ReleaseV1 (Version F1000). [Code]. Zenodo. 2023; Publisher Full Text

[40] 40. Paul S, Weiskopf D, Angelo MA, et al.: HLA class I alleles are associated with peptide-binding repertoires of different size, affinity, and immunogenicity. J. Immunol. Baltim. Md. 1950. 2013 Dec 15; 191(12): 5831–5839. Publisher Full Text

[41] 41. Pavlos R, McKinnon EJ, Ostrov DA, et al.: Shared peptide binding of HLA Class I and II alleles associate with cutaneous nevirapine hypersensitivity and identify novel risk alleles. Sci. Rep. 2017 Aug 17; 7(1): 8653. PubMed Abstract | Publisher Full Text | Free Full Text

[42] 42. Abelin JG, Keskin DB, Sarkizova S, et al.: Mass Spectrometry Profiling of HLA-Associated Peptidomes in Mono-allelic Cells Enables More Accurate Epitope Prediction. Immunity. 2017 Feb; 46(2): 315–326. PubMed Abstract | Publisher Full Text | Free Full Text

[43] 43. Spurgin LG, Richardson DS: How pathogens drive genetic diversity: MHC, mechanisms and misunderstandings. Proc. R. Soc. B Biol. Sci. 2010 Apr 7; 277(1684): 979–988. PubMed Abstract | Publisher Full Text | Free Full Text

[44] 44. Manczinger M, Boross G, Kemény L, et al.: Pathogen diversity drives the evolution of generalist MHC-II alleles in human populations. PLoS Biol. 2019 Jan 31; 17(1): e3000131. PubMed Abstract | Publisher Full Text | Free Full Text

[45] 45. White CF, Pellis L, Keeling MJ, et al.: Detecting HLA-infectious disease associations for multi-strain pathogens. Infect. Genet. Evol. 2020 Sep 1; 83: 104344. PubMed Abstract | Publisher Full Text

[46] 46. Prugnolle F, Manica A, Charpentier M, et al.: Pathogen-Driven Selection and Worldwide HLA Class I Diversity. Curr. Biol. 2005 Jun 7; 15(11): 1022–1027. PubMed Abstract | Publisher Full Text

[47] 47. Roche PA, Furuta K: The ins and outs of MHC class II-mediated antigen processing and presentation. Nat. Rev. Immunol. 2015 Apr; 15(4): 203–216. PubMed Abstract | Publisher Full Text | Free Full Text

[48] 48. Weeder B, Wood MA, Li E, et al.: pepsickle rapidly and accurately predicts proteasomal cleavage sites for improved neoantigen identification. Bioinformatics|Oxford Academic. [cited 2022 Oct 2]; 37: 3723–3733. PubMed Abstract | Publisher Full Text Reference Source

[49] 49. Ritz U, Seliger B: The Transporter Associated With Antigen Processing (TAP): Structural Integrity, Expression, Function, and Its Clinical Relevance. Mol. Med. 2001 Mar; 7(3): 149–158. PubMed Abstract | Publisher Full Text | Free Full Text

[50] 50. López de Castro JA: How ERAP1 and ERAP2 Shape the Peptidomes of Disease-Associated MHC-I Proteins. Front. Immunol. 2018 [cited 2022 Oct 2]; 9. PubMed Abstract | Publisher Full Text | Free Full Text

[51] 51. Dincer AB, Lu Y, Schweppe DK, et al.: Reducing Peptide Sequence Bias in Quantitative Mass Spectrometry Data with Machine Learning. J. Proteome Res. 2022 Jul 1; 21(7): 1771–1782. PubMed Abstract | Publisher Full Text | Free Full Text

[52] 52. Edwards NJ: Novel peptide identification from tandem mass spectra using ESTs and sequence database compression. Mol. Syst. Biol. 2007 Jan; 3(1): 102. PubMed Abstract | Publisher Full Text | Free Full Text

[53] 53. Prakash A, Piening B, Whiteaker J, et al.: Assessing Bias in Experiment Design for Large Scale Mass Spectrometry-based Quantitative Proteomics. Mol. Cell. Proteomics. 2007 Oct 1; 6(10): 1741–1748. PubMed Abstract | Publisher Full Text

[54] 54. Timp W, Timp G: Beyond mass spectrometry, the next step in proteomics. Sci. Adv. 2020 Jan 10; 6(2): eaax8978. PubMed Abstract | Publisher Full Text | Free Full Text

[55] 55. Paul S, Croft NP, Purcell AW, et al.: Benchmarking predictions of MHC class I restricted T cell epitopes in a comprehensively studied model system. PLoS Comput. Biol. 2020 May 26; 16(5): e1007757. PubMed Abstract | Publisher Full Text | Free Full Text

[56] 56. Trolle T, Metushi IG, Greenbaum JA, et al.: Automated benchmarking of peptide-MHC class I binding predictions. Bioinformatics. 2015 Jul 1; 31(13): 2174–2181. PubMed Abstract | Publisher Full Text | Free Full Text

[57] 57. Zhao W, Sher X: Systematically benchmarking peptide-MHC binding predictors: From synthetic to naturally processed epitopes. PLoS Comput. Biol. 2018 Nov 8; 14(11): e1006457. PubMed Abstract | Publisher Full Text | Free Full Text

[58] 58. Venkatesh G, Grover A, Srinivasaraghavan G, et al.: MHCAttnNet: predicting MHC-peptide bindings for MHC alleles classes I and II using an attention-based deep neural model. Bioinformatics. 2020 Jul; 36(Suppl 1): i399–i406. PubMed Abstract | Publisher Full Text | Free Full Text

[59] 59. Bhattacharya R, Sivakumar A, Tokheim C, et al.: Evaluation of machine learning methods to predict peptide binding to MHC Class I proteins. bioRxiv. 2017 Jul 27; 154757.

Discordant results among major histocompatibility complex binding affinity prediction tools

Abstract

Keywords

Introduction

Methods

Sequence retrieval, peptide filtering, and kmerization

Peptide-MHC class I binding affinity predictions

Dimensional reduction and binning analysis

Figure 1. Schematic of peptide binned density metric.

Allele ordering similarity

Interrater reliability

Results

Peptide predictions are inconsistent across tools

Figure 2. Inconsistency of peptide predictions across tools.

Figure 3. The correlation of HLA allelic presentation of 8-11mers from the random proteome between tools.

Amount of training data does not explain inconsistencies between tools

Figure 4. The relationship between training data and consistency of predictions.

Predicted binding quantities are similar between human and viral proteomes

Figure 5. The correlation between peptide sources of predicted allelic promiscuity across alleles.

Peptide physical properties are associated with allele-specific binding predictions

Figure 6. The relationship between physical property similarity vs peptide binding similarity.

Figure 7. Differential distributions of physical properties for 9-mer peptides predicted to bind to HLA alleles.

Discussion

Conclusions

Data availability

Underlying data

Software availability

References

Comments on this article Comments (0)

Open Peer Review

Comments on this article Comments (0)

Open Peer Review

Reviewer Status

Reviewer Reports

Comments on this article

Browse by related subjects

Competing Interests Policy

Stay Updated