Patterns of database citation in articles and patents indicate long-term scientific and industry value of biological data resources

Sources of full-text and data accession citations used in this study

The full-text research articles used in this study were accessed from EPMC²². The content scope of EPMC covers over 25 million PubMed abstracts and 3.5 million full-text articles (see https://europepmc.org/About), each article is identified by a unique identifier (a “PMID” for abstracts and a “PMCID” for full-text). Data accession references were extracted using EPMC’s text-mining pipeline based on a combination of rule-based knowledge about possible accession number structures and an empirically-determined set of contextual cues^14,20,23. The pipeline is integrated into the EPMC infrastructure (http://europepmc.org/) and is used to identify instances of data citation in full-text articles on a daily basis. The data citations are made publically available via EPMC’s APIs. When comparing research articles with patents we focused on 2014 as the most recent year available. However, due to the fact that embargoed articles were still being added at the time of our study, we repeated our analysis using material from 2012 and 2013 to ensure that our comparisons were robust.

The Protein Data Bank (PDB) is the global archive of 3D structures of proteins, nucleic acids and complex assemblies. This large corpus of data (94,117 holdings in 2014) and related citations provide an extensive test set for developing and understanding data citation and access metrics (http://www.wwpdb.org/stats/deposition). We used the European site PDBe as the definitive source of deposition data, i.e. accession identifiers, deposition dates and associated PMID publication details.

The European Nucleotide Archive (ENA; http://www.ebi.ac.uk/ena) is Europe's primary resource for nucleotide sequence information. The current size of the ENA is in excess of 2.5 petabytes, with a doubling time of approximately 20 months (see http://www.ebi.ac.uk/ena/about/statistics). We used ENA as an additional definitive source of deposition data, i.e. accession identifiers, deposition dates and associated PMID publication details.

SureChEMBL (https://www.surechembl.org/) is a publicly available, large-scale resource containing chemical annotations found in the full-text, images and attachments of patent documents²¹. Its data content at 28 October 2015 included more than 14 million chemically annotated full-text patent documents. In addition, it contains 130 million patent abstracts from DOCDB, the European Patent Office master documentation database with worldwide coverage containing bibliographic data, abstracts, and citations (but no full-text or images).

SureChEMBL provides full-text searching of the patent literature using a keyword-based querying functionality, complemented by a chemistry-based query engine. Our queries retrieved full-text patent documents (both applications and granted patents) written in the English language, published in 2014 by the three main patent authorities, namely the European Patent Office (EPO), the US Patent and Trademark Office (USPTO) and the World Intellectual Property Organisation (WIPO). To ensure the relevance of the retrieved patent documents to biological and life sciences, the appropriate international patent classification (IPC http://www.wipo.int/classifications/ipc/en/) codes (predominantly from categories A (human necessities) and C (chemistry), full query: “(ic:(A01 OR A23 OR A24 OR A61 OR A62B OR C05 OR C06 OR C07 OR C08 OR C09 OR C10 OR C11 OR C12 OR C13 OR C14 OR G01N) OR cpc:(A01 OR A23 OR A24 OR A61 OR A62B OR C05 OR C06 OR C07 OR C08 OR C09 OR C10 OR C11 OR C12 OR C13 OR C14 OR G01N) OR ecla_ec:(A01 OR A23 OR A24 OR A61 OR A62B OR C05 OR C06 OR C07 OR C08 OR C09 OR C10 OR C11 OR C12 OR C13 OR C14 OR G01N)) AND desc:the AND pdyear:[2010 TO 2014] AND pnlang:EN AND pnctry:(WO OR EP OR US)” ) were used to filter the results²⁴. No further selection was carried out on the basis of patent kind (an indication of where the patent is in the review process, e.g. application stage, or granted). Patent families were identified using the simple patent family definition provided by the European Patent Office (EPO)²⁵ and a single example selected at random to be sole representative of the group in subsequent analysis. In total, 188,589 documents published in 2014 were retrieved and used as input for the identifier extraction process. The XML content generated by these patent selections was then mined for accession numbers using the EPMC text-mining pipeline.

Text-mining performance characteristics

The performance assessment characteristics of the text-mining pipeline have been previously reported as 97.45% precision/59.6% recall for ENA and 94.63% precision/91.36% recall for PDB accession references when calibrated against an open access full-text corpus from EPMC¹⁴. No large-scale validation of the pipeline has been performed on the patent literature. However, manual inspection on a subset of 110 entries indicated that the approximate precision of the system was 99% and recall was 93%. Overall then the accuracy of the system appears to be higher when working with patents. This is possibly due to that fact that most citation-positive patents contain multiple exemplars whereas many research articles only include one. This would reduce the incidence of false negatives.

Metadata acquisition

The EPMC metadata and text-mining results used in this study can be accessed or generated via Europe PMC’s RESTful API which gives access to search tools with citation-count sort order and data citation features. For example, to get all the PDB citations text-mined in the articles published in EPMC in 2014 go to http://www.ebi.ac.uk/europepmc/webservices/rest/search?query=PUB_YEAR:2014 and then for each of those get the accessions identifiers (e.g. for PMID 22517515 the query is http://www.ebi.ac.uk/europepmc/webservices/rest/MED/22517515/textMinedTerms/ACCESSION). The ENA accession data used here was obtained from EMBL release 124 (described in detail here ftp.ebi.ac.uk/pub/databases/ena/sequence/release/doc/relnotes.txt). The data are public and available at: ftp.ebi.ac.uk/pub/databases/embl/release/ or through the ENA Browser and REST API. We used the primary accession identifiers and deposition article PMIDs found in the flat XML files for each entry, and included all ENA data classes with the exception of the WGS (whole genome shotgun) depositions because these are lower level assemblies with sparse or no annotation information and so less likely to be cited in publications.

The PDB data was obtained from the 2 September 2015 release of the PDB. PDB has a weekly release cycle that is loaded and processed by the PDBe team. The PDBe database also contains information extracted from EPMC about additional PMID that reference or mention any given PDB accession identifier. This information is updated once a month. Citation data was extracted from the PDBe database and included information on PMID that mention the PDB identifier code or cite the primary citation describing the given PDB entry. Citation data is available via the PDBe API (See related publication call at http://www.ebi.ac.uk/pdbe/api/doc/) as well as on the individual PDBe entry pages (e.g. http://pdbe.org/3p8c and http://pdbe.org/3p8c/citations).

Data analysis

Each record in a database has a unique accession number, a release or publication date, a series of revision dates, the bibliographic details of the deposition article, subsequent references associated with the generation of the data set and a list of references citing the source. By combining the metadata within the data resource with the citation information from EPMC we could identify the citation linked to the deposition article and hence distinguish between the initial citation event associated with the deposition article or the release of the data to the public, and track the secondary citations of a data entry (or annual cohorts of data entries) over time.

The data sets associated with the generation of Figure 1, Table 1, Table 2, Table 3 and Table 4, and Supplementary material Table 1 and Supplementary material Table 2 are provided (see Data availability). More specifically, data sets containing accession identifiers, deposition_PMID, deposition year, year of first_publication, and publication year of PMID were extracted from the data resources, and corresponding accession identifiers, citation year and number of data citations in that year were extracted from EPMC.

The merged data set contained the variables: [accession_id], [deposition_pmid], [deposition_year], [first_public_year], [pmid_publication_year], [citation_year], [citations]. For records that had a [pmid_publication_year] equal to a [citation_year], we reduced the corresponding [citations] count by one to remove the impact of the deposition citation. We then tabulated total citations for [first_public_year] (or [pmid_publication_year]) against accession/source article [citation_year].

Our data analysis was carried out using the STATA 12 package (http://www.stata.com/products/).

Secondary citation of data from biomolecular resources

To establish a baseline, we used citations of accession identifiers captured by the EPMC text-mining pipeline to provide a comprehensive picture of the annual data citation characteristics for ENA²⁶, UniProt⁴, PDBe³, OMIM⁶, RefSNP, RefSeq⁵, Pfam²⁷, InterPro²⁸, Ensembl²⁹, and ArrayExpress³⁰.

In 2014, the ENA, PDB, and RefSNP accounted for 42.6%, 21.9% and 21.7% respectively of the total text-mined citations (Table 1). These proportions remained approximately constant throughout the sampled periods and hence provide a reference for comparison with the patent corpus below. In the Kafkas et al. study¹⁴ the corresponding percentages for a cohort of 486,472 articles published between 1990 and 2012 were 56.5%, 19.9% and 13.8%. We believe that the differences in these percentages can be attributed to the age structures of the two data corpuses, with the Kafkas set providing a more longitudinal view hence favouring well-established repositories such as ENA.

Unsurprisingly, given the breadth of the biomedical literature, data citations of individual biomolecular resources are relatively infrequent in EPMC: for ENA, the proportion of citing papers in 2014 are 7,016/319,815, or 2.2%, and for PDB, 5,913/319,815 or 1.8%. Collectively the investigated databases are referenced in 6.5% of our EPMC sample (the EPMC search: “pub_year:2014 in_epmc:y”, conducted 20 Oct 2015, retrieved 319,815 articles).

Estimates of secondary data citation in the scientific literature, whether measured via citation of an accession identifier in the article text or mentioned in the reference list (e.g. “1fho” or “doi: 0.2210/pdb1fho/pdb”) or via citation of the corresponding deposition article (e.g. “Blomberg et al.³¹”), need to make a distinction between citations that arise from the original act of data deposition and those that arise from the secondary citation of data. A further distinction, not investigated systematically here, could also be made according to whether the article citations come from one or more of the original author group – as above - or from an independent research group. The former practise would appear to be quite common for ENA depositions (D. Bousfield, unpublished observations). While this distinction seems straightforward in principle, different policies and deposition practices, as well as ambiguity of author names, make it difficult to distinguish these alternatives in a large-scale analysis. We note that the adoption of ORCID within publication workflows will support future disambiguation.

Combining metadata stored in EPMC and the data resources allowed us to build up a picture, based on the summation of individual data elements, of how annual cohorts of accessions and deposition articles are cited over time (see Table 2). For example the PDB accession 2jhr that refers to the crystal structure of myosin-2 motor domain in complex with ADP-metavanadate and pentabromopseudilin was made public on 13 January 2009. The corresponding article for this deposition is PMID:19122661, entitled “The mechanism of pentabromopseudilin inhibition of myosin motor activity”, published later in 2009³². At the time of our study, the deposition article had been cited a total of 9 times during the period 2009–2014 (the current list of citing articles can be found in EPMC using the query: cites:19122661_med). None of these papers cite the data accession identifier 2jhr. However, the database record was cited once by its accession identifier in PMID:21841195 (PMCID:3186370), “Structural basis for the allosteric interference of myosin function by reactive thiol region mutations G680A and G680V”. The actual statement from this paper provides a good example of how data citation occurs in narrative text: “This is very unusual, as the meta-vanadate is clearly visible in known wild-type myosin-2 structures that were obtained in the presence of ADP-VO3, like e.g. PDB IDs 2JJ9, 2JHR, and 2XO8”³³. The text components recognised by the text-mining pipeline as being an accession citation of 2JHR are highlighted in bold. The text-mining pipeline also found 2JJ9 and 2XO8.

Note that whereas each data resource by definition contains references to the complete set of deposition articles, EPMC is incomplete in its full-text literature coverage and therefore will contain only a partial set of cited accession identifiers. In addition, the text-mining process will miss some citations (false negatives) and potentially create a small number of false positives (see Methods). These factors need to be kept in mind when interpreting the results shown in Table 2 and Table 3.

Table 2 displays the secondary citation of PDB accession identifiers, subject to above-mentioned caveats, published between 2005 and 2014. In theory, elements below the diagonal should all be zero as the non-zero numbers imply that the accession identifier has been cited before it has been made public. Some of these “below the diagonal” observations may be true false positives created by the text-mining process but also occur when the primary reference article in the database has been updated to a more recent publication. Non-zero “below diagonal” citations can also arise when authors embargo the publication of the data until after the publication of their own additional work citing the data set.

Table 3 shows the corresponding picture for the continuing citation of the PDB deposition articles. Some similar “below diagonal” patterns were found and attributed to use of updated primary reference articles or occasionally genuine misalignments of the underlying archives.

As can be seen from Table 2 and Table 3 the citation of PDB data accession identifiers and PDB deposition articles remain high as the annual cohorts age. The average annual citation-rate for each deposition article in PDB is 6.7 and the annual average number of citations per accession identifier is 0.2. For ENA the corresponding statistics were 2.1 and 0.1 (Supplementary material Table 1 and Supplementary material Table 2 show the corresponding two data sets for ENA). In all four cases these citation rates are stable over time. It is worth noting that most ENA data depositions are not accompanied by a deposition article: 32,188,662 ENA entries in 2014 were not associated with a deposition article as compared to the 26,384,613 entries that were associated with 9,375 source articles. It is also worth noting that the text-mining of accession numbers in EPMC only occurs in the subset of the scientific literature where full-text is available in open access resources, hence these numbers represent a lower bound on direct data citations.

Biological data resources are extensively used within patents

Patents are frequently used as an indicator of broader societal value of research^34–37. Importantly, it is estimated that only a small fraction of the scientific and technological innovation first reported in patents is subsequently disclosed in scientific literature sources³⁸. During the creation of a patent it is essential to unambiguously identify the components of the invention and to provide extensive reviews of any prior art³⁹. Thus we sought to address the question of how these requirements translate into data citation practices within patents.

Our SureCHEMBL corpus of 188,589 full-text patents contained 7,923 patents with data citations (4.2% of the corpus). Data citations were most common in the description section – which usually constitutes by far the largest section of the document text. The breakdown by patent office shows that the majority of patents with data citations were from the US (see Supplementary material Table 3). The proportion of accessions found for the different repositories (Table 4) differed considerably from that of EPMC articles (see Table 1 for comparison) with RefSeq, ENA, RefSNP and UniProt dominating. The average number of cited accession identifiers per repository and per document (13.9) and the variance of these figures across the resources was also much higher than found for the full-text scientific literature corpus. Since the international patent code (IPC, see Methods) is a hierarchical patent classification system we can use its additional levels to probe the subject matter of accession-positive patents further. Figure 1 shows that patents with references to ENA and UniProt were extensively used to define biological entities in the IPC subclasses A61 (“Preparations for medical, dental or toilet purposes”), C07 (“Organic chemistry”) and C12 (“Microorganisms or enzymes”). The content profiles and scientific topics covered in the two corpuses – open access scientific publications and patent documents - are different and further work is needed to understand how this influences data citation rates.

Figure 1. The prevalence of the top 5 four character IPC categories for the data set as a whole, those patents containing a data citation, and those patents having a data citation to UniProt or ENA.

Note individual patents can have several IPC annotations – these percentages are based on summing all instances, i.e. “one code, one vote”. For example, 17% of the IPC codes annotating UniProt-positive patents were A61K. Key to coding: A61K preparations for medical, dental, or toilet purposes; C12N micro-organisms or enzymes; A61B diagnosis, surgery, intervention; C07K peptides; G01N investigating or analysing materials by determining their chemical or physical properties; C12P fermentation or enzyme-using processes to synthesise a desired chemical compound; C12Q measuring or testing processes involving enzymes or micro-organisms; C07D heterocyclic compounds. Note absence of A61B from the more biological data sets, compared to the presence of C07K.