The impact of the National Heart, Lung, and Blood Institute data: analyzing published articles that used BioLINCC open access data

I went through the manuscript and amended and responded to all comments. Here are the responses.

Reviewer Colby Vorland and Andrew Brown



It is an honor to receive a feedback from Drs Colby and Brown from Indiana university, we performed almost all the changes suggested, and we hope the current version satisfy the quality required. Here are the detailed responses.



Summary:
The authors ask an interesting question as to what the impact of BioLINCC has been on the use of open data. However, the assessments of impact do not seem to appropriately contextualize the use of BioLINCC datasets as compared to growth of scientific publishing overall. Further, the authors include data in their analyses before the existence of BioLINCC, and the methods used to sample are unclear.
Response: Thank you. The study is mostly descriptive of the studies published using datasets stored at the BioLINCC repository, with a bibliometric analysis of these studies. We believe that such analysis will show the impact of open data and will encourage authors to further share their data publicly. So we agree with the reviewer that the current analysis lacks the comparison with the growth of overall scientific literature, but we will consider such analysis in the near future.
The BioLINCC is basically a repository to store and facilitate the share of data collected by National Heart, Lung, and Blood Institute funded studies, where these studies and their data might have been done before the existence of the BioLINCC, but were stored in the BioLINCC repository afterward.

--------------
Abstract 
It is unclear why WoS indexing is in the conclusions, unless it is being used as a proxy for ‘impact.’ If so, the article does not make clear that WoS indexing is being used as a sign of impact.
Response: We used the WoS database as they have strict and high bar criteria to index publication, so it was used as a proxy for impact as the authors stated. We clarified this point in the abstract as suggested.

 
The choice of ‘citations’ as a metric for ‘impact’ is questionable. Citations may best be considered a metric of ‘attention’.
Response: we agree with the author that the issue of considering citation as the sole metric for impact is debatable, so we made this point clear in the limitation section.
 
Introduction
BioLINCC was not initiated in 2000 as stated- it was 2008: https://www.nhlbi.nih.gov/science/biologic-specimen-and-data-repository-information-coordinating-center-biolincc
The NHLBI had a different repository since 2000. However, since the authors focus on BioLINCC, this raises the question why the authors start their survey in 2002, and how this data came to be included in their sampling.
Response: Whereas the BioLINCC repository itself was made in 2008, the datasets deposited in it were developed before that, since 2000*, so the publications may date back to as early as 2002.
* Coady SA, Wagner E. Sharing individual level data from observational studies and clinical trials: a perspective from NHLBI. Trials. 2013 Dec;14(1):1-3.

Methods
The methods as currently stated are not reproducible. For example, on what date were BioLINCC and PubMed sampled? What was the search string for PubMed?
It is not clear whether PubMed entries necessarily indicate that they use BioLINCC datasets. The authors mention “any study that reported the use of the searched dataset as part of its results was included”. Does that mean full texts were reviewed? If so, by what process and by which of the authors?
Response: The original dataset was extracted by contacting the BioLINCC personnel and asking them for the up to date list of publications that used repository’s datasets. Our supplementary PubMed search was carried out as follows, which we further elaborated in the methods section:
“A manual search of PubMed was also carried out to confirm an updated full list of publications as follows:

• We used the basic search of PubMed by inputting the title of the dataset in the search field (e.g., Cooperative Study of Sickle Cell Disease or CSSCD), in order to retrieve results that mention the dataset in the title, abstract, or keywords.
• The searched articles were manually screened by one of the authors (SAA) to check if the dataset was used in the study to generate results, where authors either detail the name and acronym of dataset used in the methods section, usually with specific citation to relevant study, or in the acknowledgment section in their articles. The included articles either used data stored in the BioLINCC repository alone or used these datasets along with other datasets from other repositories
• We added the searched articles to the original dataset provided by the BioLINCC.
•  We analyzed the number of studies published using each dataset (supplementary material).”

It is not clear what is meant by “title of the dataset in the search field”. Does this mean the study name? The study acronym? It seems likely that many publications would not use the exact dataset name in the title or abstract and this approach would therefore potentially miss papers. Were any new papers found beyond the BioLINCC list from the PubMed search? How many articles from the BioLINCC list were not confirmed in the PubMed search? This information is missing from the methods.
Response: Inputting the title and the acronym of the dataset in the PubMed search will retrieve all articles that mentioned the dataset in the title, abstract, keywords. The guidelines for reporting secondary analysis articles require the mention of the dataset used in the title or abstract*. Despite that, we agree with the reviewers that our search might miss few articles that did not mention the dataset there. We tried to limit the words for the methods and results, which is why these details are not provided the full manuscript. We directly added the results searched by the supplementary search directly on the original dataset provided by the BioLINCC, which is provided as supplementary material.

* Swart E, Schmitt J. STandardized Reporting Of Secondary data Analyses (STROSA)-Vorschlag für ein Berichtsformat für Sekundärdatenanalysen. Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen. 2014 Jan 1;108(8-9):511-6.

 
Results
It is indicated that over 9% of the articles using data from BioLINCC are not WoS indexed. If these articles are not evenly distributed over time, then it will skew the results of the trends. For example – were the papers not indexed more recent papers that WoS has not yet picked up? At minimum, the authors can manually extract the year, journal, and country of publication from these papers to include the in assessments.
Response: We analyzed the non-indexed articles manually to check if they were published in 2019, which if so might reflect a delay in the indexing. We found that they were distributed over the years with the majority were published in the year 2018. We could not perform detailed analysis as they could not be analyzed using the WoS database, so clarified this in the results: “For the 99 (9.12%) articles that were not indexed, they were distributed over the years with the majority (i.e. 42 articles) were published in 2018.”
 
The utility of the analyses as currently presented seem questionable. How does the increase in articles published and citations by year compare to trends in overall metrics of these measures? i.e. do these trends outpace or just reflect the growth of scientific publishing overall? The authors may also consider limiting such comparison to the specific fields that use BioLINCC data.
Response: Thank you for the important point. While we did not compare with the overall publishing trend in the field, we tried to show the increase in the number of publication using open access data in each field. The use of specific dataset might not be restricted to the field of the dataset itself, as a dataset that was originally a cardiovascular dataset might be used by researchers from other fields for other ideas. As an example, Radiological images in ACCESS datasets were used several times for Radiology publications.

The authors note the values ‘peaked in 2018’, but that was the most recent year of full data, given their partial year in 2019. Thus, 2019 is likely artificially small by virtue of it being a partial year.
Response: We agree with the reviewer, so we made this point clear in the methods. 

The University of Alabama at Birmingham is part of the University of Alabama System, and thus counting them separately does not seem to make sense.
Response: We corrected according to reviewer suggestion, the WoS database have both as separate affiliation, which led to this confusion.

It is unclear how fields of study were determined. Were these just extracted from WoS (is this “topic of publication” per methods or a separate extraction), or did the authors classify them? Regardless, the finding that cardiology is the top field is not surprising, given that BioLINCC is from the National Heart, Lung, and Blood Institute. This should be made clear.
Response: These are WoS based classification, we changed in the text accordingly.
 
One key point that may undermine the idea of ‘impact’ of the open datasets is that the study investigators appear to be included in these counts. For example, the University of Alabama at Birmingham is a key site for some studies (e.g., CARDIA), and thus they would be publishing from their datasets whether they were open in BioLINCC or not. So, what is the incremental contribution to investigators who are not part of the cohort? What difference is it making for how many papers would be published if the data were open or not?
Response: We thank the reviewers for the important remarks, as it is difficult to performed such discrimination in the current study, we made this point clear in the limitation part, so that readers would consider this point upon interpreting the results.

Discussion
In general, the discussion does not seem to flow logically. For example, in one paragraph, the authors discuss the percent of publications after data release, the top countries from which BioLINCC data are used and top journals, and then a single example of clinical impact from using BioLINCC data. The points in the discussion should be separated and connected to the purpose of the study. New results (e.g., impact factor) should not be introduced in the discussion. Further, have there been other studies that have examined these or related questions about BioLINCC or other repositories?
Response: We made several changes on the discussion to improve its flow. We removed some of the unrelated discussion part. We also removed the part related to impact factor.


“The impact of these publications can be measured in terms of citations received, where citations of publications using BioLINCC data have exponentially increased”
Exponential growth is a specific mathematical term whereas the growth in the figures appears to be roughly linear.
Response: We changed accordingly, thank you.

“Researchers new to open data might be skeptical about the publishing opportunity of studies performed using open data.”
This statement does not seem relevant to the analysis nor supported by any citations.
Response: We removed it through our effort to improve the discussion part, thank you.


Finally, a limitations section is needed noting the sole focus on WoS and whether the inclusion of other indexes might alter conclusions. For example – to our knowledge, F1000Research is not indexed in WoS; would relevant studies published here be included in a different index?
Response: We agree with the reviewer, we made clear that WoS database might not include all studies published using open access data from BioLINCC repository.
 
Data
We downloaded and inspected the data:
There is no data dictionary to interpret the dataset.
Response: We added a description at the dataset website:
 https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi%3A10.7910%2FDVN%2F1TXA3C&version=DRAFT

‘Recid’ starts at 4 and not 1. Some ‘Recid’s are missing (for example, #5, #7). Were these entries those that were not indexed by WoS? Those DOIs would still be useful to include in the dataset so future researchers can use them.
Response: The entries did not use the BioLINCC data, so were not included in the dataset.
 
Were theses and other article types included in all analyses (include this information in the methods)?
Response: Thesis were not included, we added this to the methods. 

There are missing data (e.g., funding; MESH terms; article types; study type; one publication was missing ‘study list’). The authors state that they searched WoS by DOI, and yet DOIs are missing from some entries. How was this accounted for in the analysis? Are the missing DOIs counted as part of ‘not indexed in WoS’?
Response: Missing doi were added to the manually to the WoS search for data analysis. They were not counted as part of the “not indexed in WoS”. After inputting doi to the databse, information about the study will automatically be retrieved from the WoS database, so missing data in the excel sheet won’t affect the analyzed data.


Thank you for your consideration!

Sincerely,
Saif Aldeen AlRyalat, M.D.
Corresponding author.

We would like to thank Dr. Frederer, who is an expert in the field of data science, for the insight and thoughts she shared through her revision. While we agree with her comments on the gap between our study and the big aim of “studying the impact of open data”. Our study tried to answer a certain aspect of this aim, which we further specified in the current version, and added more data that will provide better insight on BioLINCC datasets, publications, and their impact. Here are our detailed responses:
 
Comment: The authors have addressed an interesting question - what are the impacts of open data, specifically considering citations received by publications using open data sets. This question is very timely given the increasing number of funder and journal requirements that datasets be made open. However, my primary concern with this paper is that it doesn't really provide much context for understanding impact. 
 
Response: We totally agree with Dr. Federer, that the analysis of open data is most relevant during this time. One of the most important data repository in the biomedical field containing high quality datasets for well conducted studies is the BioLINCC repository. The interest to study the characteristics of this data repository is not new, as we pointed in the publications by Coady et al., Ross et al., and Giffen et al. While none of these papers alone provide the full picture of the impact of the BioLINCC repository and open data, they each provide knowledge on certain aspects. We chose to study the number of datasets that were used out of the total datasets in the BioLINCC repository, the number of publications generated (considering that this is the main reason for data requests as found by Ross et al), and number of citations these publications received. While this aim will cover a small aspect of the big question of “the impact of open data”, it will provide an important insight for better understanding of the characteristics of open data at the BioLINCC repository, what are the main datasets used, their fields, and the assurance that using an open dataset won’t compromise publishing potential of studies, if important findings found. We tried to stress further on this point.
 

Comment: The authors have tracked citations to articles that reuse datasets over time. However, with nothing to compare these counts to, it's hard to contextualize what these citations really mean. Are these papers being cited more/less than similar articles that aren't reusing datasets? How do citations to these articles describing secondary reuse compare to the number of citations received by the articles describing the dataset originally and its primary use? It's evident that citations to these articles are going up over time, but that's to be expected to an extent. So I'm not really sure what to make of these numbers and how to use them to understand impact. While this article provides an overview of the state of citations to articles reusing BioLINCC data, it is unclear to me what conclusions can be reasonably drawn from this analysis about impact.
Response: We agree with the author that comparing the number of citations received by publications that used open data with primary data articles would provide a better insight into the impact of open data compared to other articles. We amended the aim to narrow its scope so that it accommodates the aspects covered by our study:
“The impact of open access data, in terms of the number of datasets used from a repository, publications generated from these datasets, and citations received by these publications are still unknown. In this study, we aim to study the BioLINCC datasets, the number of publications that used BioLINCC open access data, and the impact of these publications through the citations they received.”
 
We added the point mentioned by the esteemed reviewer to the study limitations and as a suggestion for future studies. We further discussed the aspects regarding the BioLINCC use of datasets and the characteristics of its publications.