Antibody performance in ChIP-sequencing assays: From quality scores of public data sets to quantitative certification

NGS-QC database content

All datasets presented in the retrospective analysis were originally retrieved from the GEO database and processed with the NGS-QC Generator algorithm. Quality indicators (QCis) were computed as previously reported². Briefly, QCis were generated by comparing the original read intensity profile with those observed in a fraction of the total mapped reads. For this, total mapped reads (TMRs) were first randomly sub-sampled at three defined subsets (90%, 70% and 50% respectively), then the read counts in 500 nt bins of the genome were computed for each of the random sub-sampled as well as in the original dataset. In the ideal theoretical case the read counts in all genomic bins are expected to decrease proportionally to the random subsampling (e.g., a 50% decrease of the read count intensity (RCI) when 50% of the original TMRs were sub-sampled). Genomic regions presenting the lowest variations from this theoretically expected value are considered “robust” to the random sampling and thus of high quality. For quantitation we calculated the fraction of genomic windows with RCI dispersions within defined levels (2.5, 5 and 10%).

Quality scores for all analysed publicly available datasets are available at www.ngs-qc.org. The antibody references associated with the ChIP-seq datasets analysed in this study are given in Table 1.

Antibody certification procedure

We propose the following certification procedure for antibodies to be used in ChIP-seq and related enrichment-based technologies. Clearly, this certification will not replace but rather complement the molecular biology assays currently used for validating antibody specificity. Indeed, it provides a certificate of antibody performance specifically in ChIP-seq assays. The following experimental conditions were used for the certification of antibodies described in this study:

Cell culture. HeLa cells were grown in DMEM 1g/L glucose, 5% Fetal Calf Serum and 40µg Gentamicin to a density of 15–20 millions cells/15cm plates. Cells were fixed for 30min with paraformaldehyde (1% in PBS). Fixation was quenched with 0.2M glycine in PBS, then cells were washed three times with PBS, collected and stored at -80°C.

Chromatin immunoprecipitation. Sonication: 40 million cells were sonicated in 500µL of Lysis Buffer (1% Na-deoxychlorate, 50mM TrisHCl pH8, 140mM NaCl, 1mM EDTA, 1% Triton X-100) containing 5-times diluted Protease Inhibitor Cocktail (PIC; Roche Diagnostic; 1 tablet solubilized in 10ml Lysis Buffer). Sonication was performed with a Bioblock Scientific instrument (Vibra Cell 75043; 40 cycles, 30s ON end 59s OFF; 38% power). Chromatin fragmentation was evaluated by agarose gel electrophoresis as follows: 20µL of sonicated chromatin was diluted with 20µL TE (10mM Tris-HCl pH 8, 1mM EDTA) and 5µL 5M NaCl was added. Diluted chromatin was incubated at 100°C for 30 min, centrifuged at 12,000 rpm at room temperature and the supernatant was loaded onto a 2% agarose gel.

Chromatin Immunoprecipitation: 25µL of ChIP-IT Protein G Magnetic Beads (ActiveMotif) were incubated with the antibody under evaluation (the amounts of antibody in use corresponded to that indicated by the supplier’s information) in presence of 100µL PIC-containing Lysis Buffer. After two hours at 4°C on a rotating shaker, chromatin from 3 million cells was added and the final volume was adjusted to 500µL with PIC-containing Lysis Buffer and incubation on a rotating shaker was continued overnight at 4°C. The immunoprecipitated chromatin was recovered by magnetic bead separation, followed by multiple washing steps on a custom liquid handling platform (TECAN EVO75). Specifically, the washing is performed as follows: (1) Low salt washing (0.1% SDS, 1% Triton X-100, 2mM EDTA, 20mM TrisHCl pH 8, 150mM NaCl); (2) High salt washing (0.1% SDS, 1% Triton X-100, 2mM EDTA, 20mM TrisHCl pH 8, 500mM NaCl); (3) LiCl-washing (0.25M LiCl, 1% IGEPAL CA630, 1% Na-deoxycholate, 1mM EDTA, 10mM Tris pH 8) and (4) 1×TE washing. The immunoprecipitated chromatin was eluted and de-crosslinked in 100µL of elution buffer (1% SDS, 100mM NaHCO₃, 250mM NaCl, 0.2mg/ml Proteinase K) and incubated for 4 hours at 65°C. The eluted chromatin was supplemented with 200µL H₂O and 300µL phenol/chloroform/isoamyl alcohol (25/24/1) mix was added. After two extraction steps, the aqueous phase was subjected to ethanol precipitation in presence of 1µL GlycoBlue (Invitrogen; 15mg/ml). The precipitated material was re-suspended in 45µL H₂O; 5µL was used for validation by quantitative PCR, the remaining 40µL was used for DNA library preparation.

DNA library preparation and massive parallel sequencing. The DNA library preparation for massive parallel sequencing was performed according to standard procedures (NEXTFlex ChIP-Seq Kit (Biooscientific)) adapted to automation by our custom liquid handling platform (TECAN EVO75). Prior to DNA sequencing library preparation was monitored using a Tapestation (Agilent). Samples were sequenced on an Illumina HiSeq2500 platform following manufacturer’s standard procedures.

NGS-QC certification. The antibody certification is based on the quality control system previously described². The certification is based on two biological replicate ChIP-seq assays performed at high sequencing depths (~50 million mapped reads per dataset). For each replicate the global quality grades were computed. In addition the following issues were part of the certification:

Optimal sequencing depth: This is performed by re-computing quality grades at decreasing fractions of the initial total mapped reads (TMRs). Briefly, defined TMR subsets were generated by random sampling (20%, 40%, 60%, 80% and 100% of TMRs) and quality scores were assessed. The sequencing depth at which the quality grades transit from A to B is extrapolated and designated as optimal sequencing depth.

Local QC Irreproducibility Discovery Rate (local QC-IDR): Concordance among biological ChIP-seq replicate assays were previously assessed by the Irreproducibility Discovery Rate assay⁴. Similarly, we have established an IDR-type assay based on the comparison of the location of genomic regions (500nt length) presenting the lowest read count intensity dispersion (dRCI). Briefly, genomic regions displaying a dRCI <10% in the two biological replicates were paired and ranked on the basis of their lower absolute difference between paired dRCI values (genomic regions present in only one dataset are kept and paired with a “penalty genomic window” for which a dRCI=15% was allocated). Finally, the local QC-IDR was defined as the fraction of the top 5,000 windows using a sliding window of 500nt.

Comparing QC scores with those assessed for publicly available data. Antibody certification scores were compared with quality scores computed from publicly available data in the context of their TMRs. For it, a scatter-plot displaying quality scores (y-axis) relative to the TMRs (x-axis) was generated for all datasets associated to the particular target molecule, as well as for that related to the certified antibodies.

Contents of the NGS-QC database

The NGS-QC Generator database hosts currently quality scores for >28,000 datasets (www.ngs-qc.org; December 2015), covering a variety of ChIP-sequencing and related assays (e.g. Dnase-seq, FAIRE-seq, MBD-seq, GRO-seq, etc) performed from 9 species (Homo sapiens: 54%; Mus musculus: 34%; D. melanogaster: 6%; etc) (Figure 1A). As it is based on datasets available from GEO, we retrieved antibody sources for 12,036 datasets from the complementary information; 96% of these corresponded to human or mouse-related assays (Figure 1B). Furthermore, these datasets concern 680 different target molecules, with several histone modifications being highly represented (Figure 1C). Finally, thanks to the information provided by the authors, we managed to trace the commercial sources of the antibodies (Figure 1D). Overall, this analysis reflects the interest of the scientific community in studying the role of epigenetic factors in human and mouse systems. The data reveal also that three companies dominate the market, as they provided the antibodies for 75% of the evaluated datasets.

Figure 1. Current content of the NGS-QC Generator database.

(A). The NGS-QC Generator database hosts currently quality scores for 28,108 datasets from a variety of organisms. For about 50% of the datasets the commercial source of the used antibodies could be retrieved from the information present in GEO. (B). Classification per organism of the qualified datasets for which the antibody source is known. Note that nearly 90% correspond to Homo sapiens or Mus musculus datasets. (C). Top ranked target molecules for which the antibody source is known. (D). Ten most-used commercial antibodies retrieved from the NGS-QC database.

ChIP-seq quality relative to sequencing-depth and antibody source

As ChIP-seq datasets for the histone modifications H3K4me3, H3K27ac, H3K27me3 and H3K4me1 are the most represented in the collection (Figure 1C), we focused our analyses on these datasets.

We first evaluated quality grades in the context of sequencing depths. Importantly, the expected increase in quality relative upon increased sequencing-depth is not uniform but is rather target-dependent, as is supported by the different slopes of the datasets (Figure 2A). Indeed, while H3K4me3 datasets reach high quality even with relatively low sequencing depth (~25M TMR), the profiles for H3K27me3, which display broad signals, require >60M TMRs for reaching an “AAA” qualification (Figure 2A). Figure 2B illustrates the differences in quality that correspond to the different quality grades (Figure 2B); note that these data have been obtained with human embryonic stem (ES) cells using the same antibody^5–7.

Figure 2. Influence of the sequencing-depth on the quality of ChIP-seq assays.

(A) Scatter-plot illustrating global quality scores relative to sequencing depth for ChIP-seq assays concerning the indicated histone modification marks. All illustrated mouse or human datasets were retrieved from the public domain and qualified with the NGS-QC Generator. Quality scores were computed for a read count intensity dispersion (dRCI) of 2.5%. Broken lines depict to the transition borders between discretized quality grades (“A”, “B”, “C” or “D”) defined as the quartiles of the quality distribution computed from the entire NGS-QC collection (>28,000 datasets). The continuous lines correspond to the locally weighted scatter-plot smoothened (LOWESS) regression curves (displayed confidence interval p-value: 0.995). (B) Local genomic view (HoxD cluster) displaying read count intensity patterns for the histone modification mark H3K27me3 derived from three different datasets. In all cases the same cell type (human ES cells) and antibody source (Millipore: # 07–449) has been used, while different DNA sequencing coverage was applied. Note that the associated quality grades correlate with the sequencing depth.

Another aspect to highlight is the fact that there is a high variability of quality scores for profiles from <50M TMRs. In contrast, all datasets with higher sequencing depths tend to have quality grades between “B” and “A” but the number of such datasets is low compared to those with less sequencing coverage. These observations clearly support the notion that in addition to the sequencing depth, other experimental factors - including the commercial source of the antibody - directly influence the global quality of ChIP-sequencing assays.

To further explore the role of the antibody source, we have classified the different datasets for antibody vendors and TMR intervals, such that the quality grades per datasets are displayed in context of these two parameters. As illustrated in Figure 3, this classification recapitulates the influence of the sequence depth irrespective of the antibody source and reveals the target–specific quality differences. For example at <50 million TMRs most of the evaluated H3K27me3 datasets (Figure 3A) have quality grades lower than “A”. However, while datasets generated with an antibody from a particular vendor would rapidly gain in quality grades with increased TMRs, those from other sources improve only weakly and require much more TMRs to reach the highest quality grades; examples are the H3K27me3 antibodies of Millipore and Abcam with the Millipore one exhibiting the higher robustness. Similarly, H3K4me1 datasets require >20M TMRs to reach high quality grades but a few antibodies yielded datasets at 20–30M TMRs that got grade “A”, while others needed much higher coverage to reach “A” grades; examples are again the Abcam and Millipore antibodies, which show the inverse robustness of their H3K27me3 counterparts (Figure 3B). Notably, datasets related to the histone modification marks H3K27ac and H3K4me3 present high quality grades even for lower TMRs. In fact a minimum of 10–20 million reads was sufficient to attribute quality grade “A” to some H3K27ac datasets (Figure 3C) and even lower DNA sequencing coverage was required to reach such quality scores for H3K4me3 datasets (Figure 3D).

Figure 3. Bar graphs illustrating the frequency of datasets related to a defined antibody vendor per quality grade and total mapped reads (TMRs) interval.

Datasets for each target were categorized on the basis of their sequencing coverage (X-axis: from 1 to 100 total mapped million reads; intervals of 10 million and a last category from 100–500 million), as well as on their quality grade (Y-axis: from “A” to “D”, defined on the basis of a read count intensity dispersion (dRCI) threshold criterion of 2.5%). The illustrated bar graph correspond to the fraction of datasets related to a given vendor per TMRs interval. A to D. Frequency bar graphs corresponding to H3K27me3, H3K4me1, H3K27ac and H3K4me3, respectively.

Taken together this analysis reveals that while there is a general direct correlation between the sequencing depth and the quality scores of the corresponding profiles, there are three important considerations: (i) a minimal sequencing depth is required to ensure an acceptable quality of the assay; (ii) the nature of the histone mark has a profound impact on the sequencing depth needed to reach high quality grades, as the increase in quality with increasing coverage is not genuine and differs between marks; and (iii) the source of the antibody has a significant impact on the quality of the profile that can be obtained at a certain sequencing depth.

Overall, this meta analysis provides not only numerical values to assess the effect of the sequencing depth on ChIP-seq quality, but it offers in addition the possibility to rate the performance of new antibodies and antibody batches. Note, however, that a multitude of additional factors affects the quality of ChIP-seq datasets independently of the antibody source; these factors include cell types/tissues in use, the chromatin fixation conditions, experimenter variability, and many more. Indeed, the scatter of QCis observed at defined TMR intervals (Figure 2A) results most likely from such effects. As a consequence antibody certification for ChIP-seq assays should be based on standardized experimental conditions together with quality assessment derived from biological duplicates at high sequencing depth, such that optimal sequencing depths can be recommended to the users.

Improved certification for “ChIP-seq grade” antibodies

To respond to the needs of the scientific community for highly reliable, target specific, low background polyclonal (and monoclonal) antibody sources for epigenome or any other ChIP-based studies commercial antibody suppliers have incorporated “ChIP-seq grade” antibodies in their portfolio and based this grade on visual inspection of generally a single selected genomic region from a ChIP-seq profile. However, there are several problems associated with this procedure: First, the screenshot of a rather short genomic region says next to nothing about the quality of the entire genome-wide profile. Second, the assay conditions and sequencing statistics are generally not provided, and third, information on replicate and batch variation is generally missing.

To improve this situation we have set up a certification procedure which is based on (i) the use of a highly reproducible pipeline for performing ChIP-seq samples preparation and (ii) the use of the NGS-QC Generator tool for assessing quality grades from biological replicates that can this time reflect the performance of the antibody (Figure 4A). The procedure involves an extensive documentation including experimental details, full sequencing statistics and quality analysis of the profiles obtained from biological replicates together with recommendations for how to use this antibody to obtain optimal quality grades.

Figure 4. A quantitative approach for the certification of ChIP-seq grade antibodies.

(A). Scheme of the antibody certification procedure comprising the standardization of the experimental steps involved in sample preparation and computational treatment of ChIP-seq datasets. (B) to (F). Examples of some of the key analytical data generated from biological replicates during the certification process using the Active Motif anti-H3K27ac antibody # 39685. (B). Quantitative RT-PCR validation of the chromatin enrichment efficacy performed at the end of the standardized experimental procedure. DPP10 and MB (Myoglobin) refer to reference regions devoid of H3K27ac, while GAPDH and CCNA2 promoter regions are used for the evaluation of the enrichment levels (Fold occupancy relative to DPP10). (C). Extrapolation of the optimal sequencing depth by sub-sampling of the initial TMRs. The inferred quality scores are displayed relative to the mapped reads. The vertical green line defines the reads at which the transition of quality grade from “A” to “B” is observed. (D). Irreproducibility discovery rate (IDR) of biological replicates. The local QC IDR is defined as the fraction of genomic regions (500nt window size) which exhibit reproducibly read count intensity dispersion levels (dRCI) below 10%. For computation, all genomic regions were subdivided in groups of 5,000 windows (sliding window with a span of 500nt) followed by ranking according to their RCI dispersion. The X-axis displays the ranked genomic regions for which the local QC IDRs (Y-axis) were computed. The broken horizontal line delimits a local QC IDR threshold of 0.1, which is used to define the number of genomic windows with sub-threshold IDR levels. (E). Scatter-plot displaying the quality scores computed during the antibody certification (triangles correspond to the two biological replicates used in the certification process) relative to those of publicly available H3K27ac ChIP-seq datasets. (F). Genomic region (chromosome 12), illustrating the local enrichment of H3K27ac mark generated by the certified antibody. Below each read count intensity profile, a heatmap illustrates the robustness of genomic bins (500nt) to random sampling as measured by their dRCI levels (yellow to black: 0–10%). The certification grade attributed to this antibody is quality grade “AAA” as indicated.

This certification procedure, which is based on the use of an automated liquid handling platform for ChIP assays and subsequent preparation of the corresponding sequencing library, has been set up with a panel of six antibodies kindly provided by Brian Egan of Active Motif. Specifically, antibodies targeting the histone modification marks H3K4me3, H3K27ac and H3K9ac were enrolled in the certification process; for each of them monoclonal and polyclonal preparations were assessed. ChIP assays were performed with Hela cells cultured and formaldehyde-fixed (see Materials and methods). The efficacy of the ChIP was verified by quantitative RT-PCR (Figure 4B). The conditions for the automated preparation of the sequencing library were also standardized.

Massive parallel sequencing was performed at high depth such that coverage was not a limiting factor for quality assessment and that the minimal sequencing depth to obtain “A” quality scores could be predicted (Figure 4C). An important component is the evaluation of the irreproducibility discovery rate of biological replicates as an integral part of the certification process (Figure 4C). Finally, we also performed a comparison with the quality grades assessed relative to the publicly available datasets (Figure 4D) and we provide screenshots of several local genomic regions to illustrate the read count enrichment for the biological replicates and to provide a local quality score (500nt window, heatmap; Figure 4E).

Overall, this certification procedure provides quantitative means for assessing antibody performance in ChIP-sequencing assays, thus their "ChIP-seq grade" would not represent a questionable marketing argument but rather a solid certification label.