Amendments from Version 1

F1000Research

2046-1402

F1000 Research Limited

London, UK

10.12688/f1000research.36395.2

Method Article

Articles

Organizing training workshops on gene literature retrieval, profiling, and visualization for early career researchers

[version 2; peer review: 2 approved]

Al Ali

Fatima

Data Curation Formal Analysis Methodology Visualization Writing – Review & Editing https://orcid.org/0000-0001-6451-7357 1 Marr

Alexandra K

Data Curation Formal Analysis Methodology Visualization Writing – Review & Editing 1 Tatari-Calderone

Zohreh

Methodology Writing – Review & Editing 1 Alfaki

Mohamed

Methodology Writing – Review & Editing https://orcid.org/0000-0003-4913-2357 1 Toufiq

Mohammed

Methodology Writing – Review & Editing https://orcid.org/0000-0002-6368-6746 1 Roelands

Jessica

Methodology Writing – Review & Editing https://orcid.org/0000-0003-3631-2041 1 Syed Ahamed Kabeer

Basirudeen

Methodology Writing – Review & Editing https://orcid.org/0000-0002-7287-1636 1 Bedognetti

Davide

Funding Acquisition Methodology Writing – Review & Editing 1 2 3 Marr

Nico

Funding Acquisition Methodology Writing – Review & Editing 1 2 Garand

Mathieu

Methodology Writing – Review & Editing 1 Rinchai

Darawan

Conceptualization Methodology Writing – Review & Editing https://orcid.org/0000-0001-8851-7730 1 Chaussabel

Damien

Conceptualization Data Curation Formal Analysis Funding Acquisition Methodology Visualization Writing – Original Draft Preparation Writing – Review & Editing https://orcid.org/0000-0002-7287-1636 a 1 1Research Branch, Sidra Medicine, Doha, Qatar 2College of Health and Life Sciences, Hamad Bin Khalifa University, Doha, Qatar 3Department of Internal Medicine and Medical Specialties, University of Genoa, Genoa, 16126, Italy

a Damien.chaussabel@jax.org

No competing interests were disclosed.

11 5 2023

2021

275

20 3 2023

2023

This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Developing the skills needed to effectively search and extract information from biomedical literature is essential for early-career researchers. It is, for instance, on this basis that the novelty of experimental results, and therefore publishing opportunities, can be evaluated. Given the unprecedented volume of publications in the field of biomedical research, new systematic approaches need to be devised and adopted for the retrieval and curation of literature relevant to a specific theme. Here we describe a hands-on training curriculum aimed at retrieval, profiling, and visualization of literature associated with a given topic. This curriculum was implemented in a workshop in January 2021. We provide supporting material and step-by-step implementation guidelines with the ISG15 gene literature serving as an illustrative use case. Through participation in such a workshop, trainees can learn: 1) to build and troubleshoot PubMed queries in order to retrieve the literature associated with a gene of interest; 2) to identify key concepts relevant to given themes (such as cell types, diseases, and biological processes); 3) to measure the prevalence of these concepts in the gene literature; 4) to extract key information from relevant articles, and 5) to develop a background section or summary on the basis of this information. Finally, trainees can learn to consolidate the structured information captured through this process for presentation via an interactive web application.

Literature profiling Science education Concept extraction Data visualization

Qatar National Research Fund

NPRPgrant#10-0205-170348

The work presented here was supported in part by NPRP grant # 10-0205-170348 from the Qatar National Research Fund (a member of Qatar Foundation). The work reported herein is solely the responsibility of the authors.

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Revised Amendments from Version 1

In response to one of the reviewer's comments we removed the step where information captured from research articles by workshop participants and structured in a spreadsheet format was used to automatically generate "standard sentences", to be used later to form a coherent narrative. We incorporated the edits and corrections suggested by the reviewer throughout the document, and we revised the abstract and title to improve clarity and conciseness.

Introduction

Peer-reviewed publications constitute the body of knowledge upon which biomedical research is based. This resource is essential for the generation of novel hypotheses and the design of studies, experiments, and trials and is thus key to the discovery process itself. However, the volume of literature on any given research topic has grown dramatically in recent years, making it increasingly challenging for a single person to manually survey the relevant literature in its entirety, and consequently to acquire the foundational knowledge needed for discovery research. Hence, developing a solid foundation of skills for literature retrieval and profiling is of critical importance for early career biomedical scientists. These competencies will, for instance, be needed: 1) to acquire a sound knowledge base through developing the ability to compile and summarize large volumes of literature. 2) to develop data interpretation skills as well as become able to assess novelty and potential impact of a given finding; and 3) to develop scientific writing skills and become able to write background material on specific topics.

Publicly available omics data provides ideal material for training new generations of biomedical researchers. ^{1–
3} One of the “collective omics data” training modules that we have developed follows a reductionist analysis and interpretation workflow using publicly available transcriptome data as the source material. ⁴ The first activities that form part of this training module involve literature retrieval and profiling for a given candidate gene.

In this article, we describe a detailed stepwise approach that we have developed running literature profiling training workshops - literature profiling here meaning information extraction from titles and determination of keyword frequencies in titles and abstracts. The training program that we have devised takes advantage of publicly available omics data. ⁴ The training module presented here focuses on retrieval of gene-centric literature. Supporting material, such as sets of slides, templates, and a handout, is also provided along with an illustrative use case. Notably, although the training activity that is presented focuses on gene-centric literature retrieval and profiling, the skills and approaches can be adapted to any other application (e.g., focusing on disease-centric, or pathway/process-centric literature).

Methods

The hands-on training exercise described here is suitable for implementation as part of a broader course on research methodology, or as a stand-alone workshop. The training is appropriate for undergraduate, graduate and post-doctoral trainees and no prior bioinformatics experience is required in order to participate. The time commitment depends on the level of experience of the attendees, the overall organization of the workflow and the volume of literature associated with the candidate gene(s) selected. For instance, for a gene with about 1,000 associated articles, five participants could work together on this same gene, each focusing on a different theme. Generally, the training could be covered in one introductory session (40 min), and three two-hour hands-on sessions. The format and content of these sessions are described in more detail below. These would cover: •

Introductory session

•

Step 1 – Retrieving the relevant literature

•

Step 2 – Extracting concepts

•

Step 3 – Generating literature profiles

•

Step 4 – Developing interactive data visualizations

•

Step 5 – Writing a narrative

In addition to covering workshop pre-requisites and time commitment, announcements for such workshops can also list the sets of skills trainees are expected to develop including: •

Literature retrieval (development of advanced PubMed queries).

•

Literature profiling (information extraction, determination of keyword frequencies).

•

Data visualization (structuring and presenting information in an interactive format).

•

Extraction of biomedical information (capturing information in a structured format).

Trainees can be asked to perform tasks between each of the sessions. Alternative formats are possible depending on the needs of the participants and a range of biomedical themes (e.g., diseases, pathways, or cell types) could be selected as the focus of the literature workshop. The endpoint for such workshops may include interactive literature profiling representations generated as part of the hands-on training activities and/or a peer-reviewed publication that, for instance, could build on such a resource.

Introductory session

The introductory session is designed to provide participants with an understanding of basic concepts and present an outline of the training curriculum. In addition, this session presents the overall rationale and teaching objectives of the training program. The introduction also defines the endpoint of the workshop activity as the development of a web resource that permits the visualization and exploration of structured literature profiles for a gene of interest (described in detail below, taking ISG15 as an illustrative case). This is followed by an overview of each of the steps that will be undertaken during the hands-on session. These are outlined above and described in more details below. It also provides participants with an opportunity to ask questions before the start of the hands-on sessions. Ideally, the presentation should last no longer than 40 minutes and an illustrative case could serve to support the presenter’s narrative. A generic presentation is provided ( Extended data File 1). ⁵

Step 1: Retrieving the relevant literature

As a first step, all the literature that is relevant to a given gene of interest must be identified. This forms the body of literature that will be subjected to literature profiling in subsequent steps. Most researchers will already be familiar with PubMed, the search engine hosted by the US National Center for Biotechnology Information (NCBI) ( https://pubmed.ncbi.nlm.nih.gov/). While this tool is straightforward to use, developing queries that will permit the comprehensive retrieval of literature associated with a given theme can be more complicated.

It is important to design a PubMed query that will permit the retrieval of all the literature that is available for the gene of interest. Challenges when retrieving literature associated with a given gene include capturing all aliases and variations that may exist in addition to the official gene names and symbols. For instance, 14 aliases were used to develop a query for the gene officially known as “ ISG15 Ubiquitin Like Modifier”. Failure to include all the aliases can lead to under-representation of the literature and erroneous judgments that may have major consequences (e.g., when gaging the novelty of a given finding and its suitability for publication). The use of synonyms among gene names and aliases can also result in high proportions of false-positive results, particularly when aliases are common language terms (e.g., CAMEL genes [official symbol: CTAG2] or WARS genes [official symbol: WARS1]). If the literature retrieved in this step misses relevant records for the candidate gene (= false negatives), the resulting literature profiles will be incomplete. If the literature retrieved contains records that are irrelevant (= false positives), the resulting literature profiles will be inaccurate. For these reasons, it is important to optimize PubMed queries used for gene literature retrieval. Notably, the same principle applies when generating PubMed queries for any given topic. For instance, when querying PubMed to identify literature associated with inflammation, terms such as “inflammatory” or “inflamed” is also captured.

The endpoint of this step is the development of an optimized PubMed query for a given gene. Any criterion can be used for the selection of a candidate gene to be assigned to a trainee or group of trainees. However, a significant body of literature is associated with the gene in question. For the purposes of illustration in this article, we have selected ISG15.

Practical activities for this step:

The official gene name, symbol and all aliases are retrieved from the GeneCard website (e.g., for ISG15: https://www.genecards.org/cgi-bin/carddisp.pl?gene=ISG15). Although gene names may also be retrieved from different sources, the advantage of using GeneCards is that information has already been compiled from reference databases such as NCBI Entrez Genes, Uniprot or genenames.org.

PubMed queries are built using the official gene name, official symbol, and all aliases for the gene of interest as search terms, and by using the appropriate Boolean operators (AND, OR, NOT), field restriction tags, and suitable syntax. For instance, in a query using multiple search terms, the Boolean operator OR must be capitalized in between each keyword. The field restriction tag [tw] can be used after each term to search “text words”, included, for instance, in title, abstract, MeSH terms and subheadings, publication types, and substance names. Or alternatively the more restrictive [tiab], can be used in order to limit the search to titles and abstracts only. A query using [tw] for multiple search terms could for example be noted as follows: ISG15 [tw] OR IFI-15K [tw] OR IFI15 [tw]. In addition, the field restriction tag [pt], can be used to restrict/exclude the search of a particular publication type (e.g., NOT review [pt], would exclude review articles from the search results). Quotation marks are employed when compound words appear as an exact phrase in the search (e.g., “ ISG15 Ubiquitin Like Modifier” [tw]). Notably, Pubmed’s advanced search builder can also be used to design such complex queries ( https://pubmed.ncbi.nlm.nih.gov/advanced/) And for reference, a wide range of training material can also be found on the NCBI website:

https://pubmed.ncbi.nlm.nih.gov/help/

https://learn.nlm.nih.gov/documentation/training-packets/T0042010P/

The PubMed query is run, and quality checks are performed among the publications that were returned. This is to identify search terms that may be too permissive and return false-positive results. For instance, short three-character acronyms tend to be more problematic, as these are terms that are otherwise used as part of a common language (e.g. CAMEL, WARS).

If necessary, the query is optimized by addressing problematic terms. Removing the ambiguous or problematic term altogether may be a solution, but this could also lead to false-negative results (missing literature that is actually relevant to the gene of interest). As a compromise, the search term could be retained but optimized by adding a keyword that would restrict the search (e.g., provided below for HUCRP and UCRP). One may also find in some instances that the list of aliases known for a given gene is incomplete and needs to be amended (e.g., below).

Illustrative case:

The optimized query (searching all words and numbers in the title, abstract, other abstract, MeSH terms, MeSH subheadings, publication types, substance names, personal name as subject, corporate author, secondary source, comment/correction notes, and other terms) for ISG15 is as follows:

ISG15 [tw] OR “ISG15 Ubiquitin Like Modifier” [tw] OR "Interferon-stimulated gene 15" [tw] OR "IFN-induced 15-kDa protein" [tw] OR IFI-15K [tw] OR “Ubiquitin Cross-Reactive Protein” [tw] OR "Ubiquitin-Like Protein ISG15" [tw] OR (HUCRP [tw] AND "Cross-Reactive Protein" [tw]) OR G1P2 [tw] OR IP17 [tw] OR (UCRP [tw] AND "Cross-Reactive Protein" [tw]) OR "Interferon-Induced 17-KDa/15-KDa Protein" [tw] OR "Interferon-Stimulated Protein, 15 Kda" [tw] OR "Interferon-Induced 17 Kda Protein" [tw] OR IFI15 [tw] OR IMD38 [tw] NOT review [pt].

This query returned 1,186 results (as of September 1 ^st, 2020). Optimization included the addition of aliases for this gene that were not captured in the GeneCard database but identified by reviewing some of the results (e.g., notably, the search argument Interferon-stimulated gene 15" [tw], which alone retrieves over 230 entries in PubMed). The acronyms HUCRP or UCRP also proved a source of false-positive results (CPR standing for cross-reactive protein, rather than C-reactive Protein). This was rectified by adding the ‘AND “cross reactive protein [tw]”’ argument to each of the ambiguous acronym.

Step 2: Extracting concepts

A large body of literature can be associated with a given gene. It may be useful then to employ a systematic “cataloguing” or indexing approach. For this step, workshop participants may first define themes under which concepts, and keywords associated with these concepts, will be categorized (e.g., themes could be ‘Human diseases and pathogens’, ‘Tissues’, ‘Cell types’, or ‘Cellular processes’). Participants would in turn scan titles of articles associated with ISG15, looking for concepts linked to the theme of interest. For example, under ‘Human diseases and pathogens’, a concept could be ‘liver cancer’ and associated keywords could be ‘liver cancer’, ‘hepatic carcinoma’, ‘hepatocellular carcinoma’, ‘HCC’, or ‘liver carcinoma’. The output for this activity would be lists of concepts and keywords associated with the gene literature organized under different themes. It is the prevalence of these concepts that will be measured in the subsequent step (generation of literature profiles).

Notably, this process could be repeated for other themes (or different themes could be assigned to each of the participants). If the literature is extensive and time is limited, it would also be possible to divide the literature into subsets (e.g., by batches of 100 articles, with participants assigned different batches to work on for the same theme). If the literature is sparse, all available articles for the gene may be used rather than just focusing on those articles including search terms in titles (i.e., using [tw], instead of [ti] in Step 2a). However, it may be generally preferable for a workshop to be based on selected genes with a relatively abundant literature (e.g., >100 articles returned when restricting the search to titles).

Practical activities for this step:

A subset of the literature is retrieved, restricting the search to titles (using the optimized query from Step 1, and substituting the field restricting argument [ti] for [tw]).

A given theme is assigned to, or selected by, participants (e.g., ‘Human diseases and pathogens’, ‘Tissues’, ‘Cell types’, ‘Biomolecules’, ‘Pathways’, ‘Biological processes’).

Concepts relevant to the theme in question are identified in the titles of articles retrieved by the query designed in a) (e.g., liver cancer, HIV).

The concepts and associated keywords (e.g., ‘hepatocellular carcinoma’, ‘liver carcinoma’, ‘hepatic cancer’, for the concept ‘liver cancer’ or ‘virus’, ‘viral’, ‘human immunodeficiency virus’ for the concept HIV) are recorded in a spreadsheet (example and templates are available in Extended data File 2). ⁶

Illustrative case:

The subset of the ISG15 literature in which the official gene name, symbol or aliases are present in titles is retrieved. The query below is adapted to retrieve only the literature for which gene name and aliases are present in titles (using [ti]).

ISG15 [ti] OR “ISG15 Ubiquitin Like Modifier” [ti] OR "Interferon-stimulated gene 15" [ti] OR "IFN-induced 15-kDa protein" [ti] OR IFI-15K [ti] OR “Ubiquitin Cross-Reactive Protein” [ti] OR "Ubiquitin-Like Protein ISG15" [ti] OR (HUCRP [ti] AND "Cross-Reactive Protein" [tw]) OR G1P2 [ti] OR IP17 [ti] OR (UCRP [ti] AND "Cross-Reactive Protein" [tw]) OR "Interferon-Induced 17-KDa/15-KDa Protein" [ti] OR "Interferon-Stimulated Protein, 15 Kda" [ti] OR "Interferon-Induced 17 Kda Protein" [ti] OR IFI15 [ti] OR IMD38 [ti] NOT review [pt]. This query returned 312 results (as of September 1 ^st, 2020).

Titles are then parsed from the ISG15 literature for concepts corresponding to the theme: “human diseases and pathogens”. The concepts and associated keywords retrieved for the “human diseases and pathogens” category are listed in Extended data File 2 ⁶ (Excel File: Extraction of concepts Tab).

Step 3: Generating literature profiles

Determining the relative prevalence of concepts among the literature associated with a given gene can be useful. This is the case when assessing the novelty of a finding, for instance (e.g., change in transcript or protein abundance associated with pathogenesis). Moreover, such an exercise and the information being derived would also be useful in other instances when writing general background/summary about the gene for a report or a manuscript.

With the two previous steps completed, determining the prevalence of concepts in the literature associated with a given gene can be achieved quite simply. For this, the literature query developed in Step 1 is modified to narrow the search and retrieve literature associated with the concepts identified in Step 2. The endpoint for this activity is a table showing the frequency of articles for these concepts in the literature associated with a given gene (e.g., Extended data File 2 ⁶ [Excel File: Literature Profiling Tab]).

Notes: Participants may also be encouraged to explore different types of visual representations for this type of data. Treemaps or word clouds can for instance show relative prevalence, while other types of graphs, such as 2D bubble graphs, may also show article frequencies (e.g., Figure 1). Another exercise could involve visualization of changes in the abundance of the literature associated with the selected concepts over the years. For this purpose, queries can be amended to add a range of publication dates with the field restriction tag [dp] (e.g., adding AND 2000:2010 [dp] to the query).

Figure 1. Visualizing <italic toggle="yes">ISG15</italic> literature profiles.

Treemap representation of the relative prevalence of concepts associated with the “human diseases and pathogens” theme among the ISG15 literature.

Practical activities for this step:

The literature query from Step 1 is employed (using the field search restriction [tw]).

The Boolean operator AND is added, followed by search terms corresponding to keywords related to one of the concepts identified in Step 2 (e.g., “Liver cancer”, “Liver carcinoma”, “Hepatic carcinoma”).

Quotation marks, field search restriction and the Boolean OR are added, so that the notation would read as follows: … AND (“Liver cancer” [tw] OR “Liver carcinoma” [tw] OR “Hepatic carcinoma” [tw]).

The query thus constructed is run, and the number of articles retrieved recorded (for instance in a spreadsheet: Extended data File 2). ⁶ Steps b through d are repeated for the rest of the concepts identified in Step 2.

Visual representations of the literature profiles are generated.

Illustrative case:

The prevalence of concepts among the ISG15 literature associated with the theme “Human diseases and pathogens” is given in Extended data File 2 ⁶ (Excel File: Literature Profiling Tab) and represented visually in Figure 1 (also in Extended data File 2 ⁶ [Excel File: Generating plots Tab]).

Step 4: Developing interactive data visualizations

Valuable insights can be gained from visual representation of information. And this could be further facilitated when the information underlying these visual representations can be accessed interactively by the end user.

To produce such interactive visual representations in the context of training workshops, we recommend using the Prezi web application ( https://prezi.com). This tool has been developed to create presentations in which it is possible to zoom in and out between levels of information. This gives users the opportunity to visualize the prevalence of concepts while at the same time allowing them to “drill down” into each individual concept in order to access relevant underlying information. The endpoint for this activity is the production of a “hierarchical circle packing chart” representing the relative abundance of concepts in the literature for a given gene and theme ( Figure 2A, https://prezi.com/view/zCedrcYaAEUAON1VeEUi). Such a resource gives users access to underlying reference information for each of the concepts ( Figure 2B) and can be made available publicly.

Figure 2. <italic toggle="yes">ISG15</italic> Human diseases and pathogens circle packing chart. A.

At the highest level, the representation permits visualization of the relative abundance of concepts associated with the “human diseases and pathogens” theme for the ISG15 gene. The color and size of the circles is a representation of the number of articles retrieved for a given concept (as indicated in the figure key). B. Zooming into each of the circles permits access to another level of information, such as links to PubMed results as well as screen captures of articles relevant to a specific topic (e.g., the relevance of ISG15 as a biomarker). An interactive version of this presentation can be accessed via: https://prezi.com/view/zCedrcYaAEUAON1VeEUi/

For the practical activity designed for this step, participants will need to register with Prezi and create an account to access and edit the interactive presentations. They can do so free of charge by selecting the basic account at: https://prezi.com/pricing/basic/. Ideally, this is completed ahead of the training session. If instructors have access to a paid subscription, they can create an unlimited number of presentations and invite individual participants to collaborate with editing rights. Otherwise, the participants can create their own presentation and invite the instructors as collaborators. They would also need to copy and paste material from a master template in their own Prezi account. Thus, this alternative solution would be workable, but probably not ideal.

Practical activities for this step:

Participants are given access to a Prezi presentation that contains a template and illustrative example (e.g. https://prezi.com/view/u65ZqHn9ZBJx1VKHqfZA/). As indicated above, one such presentation could be created and made available for each participant. It could serve as their own “Sandbox”, to familiarize themselves with the application and use as a starting point to develop an interactive resource. Multiple users can also work simultaneously on the same shared presentation. It would thus also be possible to create one presentation for a group of participants to work on cooperatively.

Starting with the template and following the illustrative example, participants can build a circle packing chart for concepts identified in Step 3.

The size and color of the circles is determined by the frequency of articles in the gene-associated literature related to a given concept (as shown in Figure 2A).

The circles can be arranged manually to create a visually attractive representation.

Underlying information is then added to each of the circles and accessed by zooming in to different levels ( Figure 2B). This information could include, for instance:

•

The gene symbol and concept;

•

The query link;

•

Number of articles retrieved on a specified date;

•

Result link to those articles;

•

Screen capture and links pointing to articles relevant to a specific topic (see Step 5 below).

Illustrative case:

The circle packing chart representation for ISG15, focusing on the “Human diseases and pathogens” topic is shown in Figure 2 . The Prezi presentation can be accessed for interactive exploration via: https://prezi.com/view/zCedrcYaAEUAON1VeEUi/

A screen recording demonstrating how to add underlying information to a Prezi circle packing chart can be accessed via: https://soapbox.wistia.com/videos/2tmC1VeQyr

Step 5: Writing a narrative

Writing material for a report or manuscript can be one of the motivations for profiling the literature associated with a given topic. To become successful in their future academic endeavors early career scientists need to become proficient independent writers. The last hands-on activity of the workshop consists in capturing key information from published work and using it as a basis for writing up a narrative about the gene of interest.

A specific topic would be selected as the focus of the activity to limit the amount of literature that would need to be covered. One such topic could, for example, be the relevance of a gene (in this case ISG15) as a biomarker for diagnostic applications within the overall theme of “Human diseases and pathogens” defined in Step 2. Participants could then focus on the most prevalent disease (e.g. in the case of ISG15 literature, three diseases have >50 articles and ten diseases have >20 articles). Literature queries corresponding to the diseases that have been selected would then be modified to retrieve only those most likely to contain information about ISG15 and its relevance as a biomarker.

Practical activities for this step:

Concepts most prevalent in the gene-associated literature for the theme of interest are selected (e.g. concept = Hepatitis C infection, within the theme = “Human diseases and pathogens”).

The query developed to retrieve literature for a given concept is modified to restrict the search in order to retrieve articles relevant to a given topic (e.g. relevance of the gene as a biomarker). The Boolean “AND” is appended at the end of the query along with relevant keywords in parenthesis separated by the Boolean “OR” (e.g. “AND (biomarker OR biomarkers OR diagnostic OR diagnosis OR prognostic OR prognosis)”.

Articles are reviewed and those deemed relevant to the topic are selected (e.g., those in which changes in abundance of the gene product in clinical specimens are reported).

Relevant information is captured from the abstract and/or full text and recorded in a spreadsheet (e.g. analyte name, species, data generation methods, comparator groups, etc.). In some cases, the full text of articles may not be accessible (e.g., paywall), and the abstract has only incomplete information. This can lead to the absence of important information in the “capture” spreadsheet, rendering the findings unusable. Participants can then be reminded of the importance of using best practices when reporting findings (e.g., mentioning the comparator group, species or the specific factor being measured – being protein or RNA). It may also be an opportunity to discuss the merits of publication in open access journals.

The information captured in the spreadsheet will serve as a synopsis that participants will be able to rely upon for developing a written narrative, which could be used in an introduction or review article.

Illustrative case:

Gene of interest: ISG15

Theme: “Human diseases and pathogens”.

Topic: Biomarker/diagnostic relevance.

Focusing on ISG15 and “human diseases and pathogens” as a theme, with biomarker/diagnostic/prognostic as a topic, the main diseases identified in step 3 were selected (>20 articles; Figure 2A, red circles and orange circles). Starting with the query developed for retrieving ISG15 literature relevant to Hepatitis C virus, the search is restricted by adding the expression “AND (biomarker OR biomarkers OR diagnostic OR diagnosis OR prognostic OR prognosis)”. Three articles were returned. From each article, relevant information available in the abstract, and in the full text where possible/necessary, is recorded in a spreadsheet ( Extended data File 3). ⁷ The information in question relates specifically to changes in abundance of ISG15 measured in clinical specimens (since it is directly relevant to the topic that was selected).

Narrative, based on the information captured in the spreadsheet:

“Interferon-stimulated gene 15 (ISG15) is a member of the ubiquitin family, which includes ubiquitin and ubiquitin-like modifiers (Ubls). Ubiquitin and Ubls are involved in the regulation of a variety of cellular activities, including protein stability, intracellular trafficking, cell cycle control and immune modulation. ISG15 has been implicated as a biomarker with diagnostic relevance for a number of human disorders, including cancer (melanoma and breast cancer) and autoimmune diseases (SLE), as well as infection with pathogens such as HBC, HCV and HIV.

The expression of ISG15 has been implicated in a wide range of human cancers, although the roles of ISG15 in tumorigenesis and responses to anticancer treatments remain largely unknown. In patients with breast cancer, ISG15 is overexpressed at both the mRNA and protein levels in mammary tumor tissue compared to that in normal mammary tissue. ^{8,
9} Increased ISG15 protein expression has also been detected in human melanoma cell lines. ¹⁰ These findings indicate the potential of ISG15 as a tumor biomarker. ISG15 protein expression is upregulated in immune cell markers from patients with invasive breast cancer ¹¹ as well as dendritic cells from melanoma patients. ¹⁰ ISG15 is also highly expressed in breast cancer patients undergoing immunotherapy. ¹² Short-hairpin RNA-mediated silencing of ISG15 expression has been shown to inhibits breast tumor growth and increase NK cell infiltration a nude mouse model of tumorigenesis. ¹³ Furthermore, shRNA-mediated knockdown of ISG15 increased the resistance of human tumor cells to CPT. ¹⁴ These findings indicate that ISG15 is also a candidate biomarker of the responsiveness to immunotherapy among patients with cancer.

ISG15 plays a key role in the host antiviral response. As such, ISG15 is implicated as a diagnostic biomarker of viral infection. Studies in patients with HCC have shown high levels of ISG15 protein in human HBV-related HCC tissues compared to the levels detected in non-tumor tissues. ¹⁵ Furthermore, upregulated expression of ISG15 was observed at both the mRNA and protein levels in liver tissue samples from patients with HCV infection compared to the levels detected in uninfected controls. ¹⁶ Similarly, the abundance of ISG15 transcripts was found to be increased in human PBMCs and liver cells in patients with HCV infection who were unresponsive to IFN treatment compared to the levels in corresponding IFN-responsive patients. ¹⁷ In addition, high levels of ISG15 in the liver of patients with HCV infection were associated with an unfavourable HCV genotype 1, a high hepatic HCV load and a low antiviral response to IFN compared to patients who did not present such characteristics. ¹⁶ These findings indicate the potential of ISG15 as a biomarker of IFN treatment response in patients with HCV infection.

Increased ISG15 expression has also been reported in mouse models related to HIV infection. ISG15 is upregulated in transgenic mice overexpressing the HIV gp120 protein. ¹⁸ Moreover, in mouse models of acute and chronic neuronal injuries with HIV infection, ISG15 protein in the brain was increased compared to the levels detected in mice with global ischemia and traumatic brain injuries. ¹⁸ The importance of ISG15 as a biomarker of HIV infection and treatment response has also been revealed in human studies. Compared to healthy donors, TaqMan assays revealed high ISG15 expression (mRNA and protein) as well as an increased frequency of ISG15-SNPs in human PBMC from untreated patients with HIV infection. ¹⁹ Moreover, ISG15 expression decreased in human PBMC in patients with HIV infection after long-term antiretroviral therapy. ¹⁹

The key importance of ISG15 in the human immune system is also reflected in its role in autoimmune diseases, such as SLE. High levels of ISG15 transcripts were detected in human whole blood cell samples from 28 patients newly diagnosed with SLE compared to with 10 patients with undifferentiated connective tissue disease, and 22 healthy volunteers, ²⁰ indicating the potential of ISG15 as a specific biomarker of SLE. Moreover, the upregulation of ISG15 in human plasmablasts/plasma cells from patients with active SLE indicate that ISG15 can be used as a marker of disease activation in patients in remission. ²¹

Thus, an increasing body of evidence supports a role of ISG15 as a biomarker with diagnostic relevance for a number of human disorders.”

Implementation

As proof of principle, a workshop was implemented using the step-by-step guide and supporting information provided with this paper. It was led by FAA and AKM, who participated in the development of the training curriculum, but had no prior experience running such training activities. The workshop took place on January 26, 2021. In total, 29 graduate students from Hamad Bin Khalifa University took part. It was offered as one three-hour class as part of an “Introduction to Data Science” course. Due to Covid-19 restrictions the workshop was run remotely using Webex. As no information was collected from workshop participants (i.e., no surveys, or questionnaires), the activity was not considered to constitute human research and therefore no ethical approval was required. The generic introductory presentation in Extended data File 1 ⁵ was adapted to provide more specific context, and notably explain how, CCR1, the target gene for this workshop was selected (for illustrative purposes this introductory presentation is also made available here: Extended data File 4). ²² FAA and AKM, the two co-instructors, also prepared a handout to guide participants through the different steps ( Extended data File 5), ²³ specifically for the CCR1 use case. Several days prior to the workshop, participants had been asked to create a Prezi “Basic” account and a template was created and shared with each one of them ahead of time. Following the introductory presentation (10 minutes) participants carried out hands-on activities following each one of the tasks described in the handout, with each step corresponding to one working session. Given time constraints, material prepared ahead of time by FAA and AKM was provided to the participants at the end of each working session so that they could move on to the next. Ideally sessions should span multiple days in order for participants to complete the assignments. However, the more intensive schedule implemented here had the advantage of introducing literature profiling concepts and approaches to a large group of trainees in a time-effective manner. The opportunity to continue work later on, on their own time being also available to them.

Conclusions

Effectively harnessing biomedical literature is one of the most fundamental skills required by biomedical researchers. Thus, early career researchers need to develop the ability to read the scientific literature at different levels: from “scanning” a large number of articles to reading the full text version for a more in depth understanding. Given the current rate at which articles are published, developing more systematic approaches to literature profiling based on defined principles may prove especially beneficial to early-career biomedical researchers.

Here, we present a training workflow as well as supporting material that may be re-used/adapted for the organization of ‘Gene literature retrieval, profiling and visualization’ training workshops. Hands-on activities range from literature retrieval and optimization of PubMed queries, to the development of interactive resources and authoring original material focusing on a specific topic.

A number of the steps described could easily be automated. For instance, the concatenation of official gene names and aliases for retrieving gene-specific literature using PubMed (Step 1). Indeed, such tools are under development by our group and will be made available in the future. These tools would not only save time, but also minimize user error. Nevertheless, some of the steps, such as optimization of search queries or extraction of information from abstracts will always require some critical evaluation and decision making that cannot be automated. Refraining from the use of such automated tools in a training workshop may also be a deliberate choice on the part of the instructor if the emphasis is initially on the development of competency in completing the steps involved throughout the process.

Here, we focused on gene-associated literature, simply because the training curriculum that is currently under development is based on the reuse of publicly available gene expression data. A similar approach could be employed to profile the literature associated with any given disease, pathway, molecular process, or drug, for instance.

We aim to further develop an illustrative case that would lead to the publication of a review of the ISG15 literature employing the literature profiling and visualization approaches described herein. We also plan in the future to develop and make available guides and supporting material for the implementation of hands-on training workshops focusing on other topics covered by our Collective Omics Data (COD) training curriculum, which includes three modules: COD 1 focuses on retrieving, visualizing and interpretating gene-specific transcriptional profiles ^{24–
26}; COD2 focuses on constituting and curating of themed public dataset collections ^{27–
35}; COD3 focuses on global analysis of large-scale omics data. ^{36–
38} These hands-on training activities would build in part on the skills developed through the implementation of the literature profiling workshop that we have presented here. Progress to date towards this goal includes the publication of a study guide for COD1, ³⁹ along with a use case that examines the relevance of CEACAM6 as a blood transcriptional marker. ⁴⁰

Data availability Underlying data

No newly generated data are associated with this article. Information was retrieved via the literature search engine PubMed ( https://pubmed.ncbi.nlm.nih.gov/).

Extended data

This project contains the following extended data:

Figshare: Literature profiling workshop, introduction session slides (S File 1), https://doi.org/10.6084/m9.figshare.13669070.v2. ⁵

Figshare: Literature profiling workshop: steps 1-3 (S File 2), https://doi.org/10.6084/m9.figshare.14160329.v3. ⁶

Figshare: Literature Profiling Workshop: Step 5 (S File 3), https://doi.org/10.6084/m9.figshare.14161484.v1. ⁷

Figshare: Literature profiling workshop: HBKU handout (S File 4), https://doi.org/10.6084/m9.figshare.14166395.v1. ²²

Figshare: Literature profiling workshop: HBKU intro presentation (S File 5), https://doi.org/10.6084/m9.figshare.14166500.v1. ²³

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC BY 4.0).

Acknowledgements

We wish to thank all the students from Hamad Bin Khalifa University (HBKU) for participating in the workshop that served as a proof-of-principle for the use of the implementation guide described in this paper. We also thank the Division of Genomics and Translational Biomedicine at the College of Health and Life Sciences at HBKU for hosting this workshop.

References 1

Margolis

Derr

Dunn

: The National Institutes of Health’s Big Data to Knowledge (BD2K) initiative: capitalizing on biomedical big data. J Am Med Inform Assoc. 2014 Dec;21(6):957–8. 25008006

10.1136/amiajnl-2014-002974

PMC4215061

Van Horn

Fierro

Kamdar

: Democratizing data science through data science training. Pac Symp Biocomput. 2018;23:292–303. 29218890

PMC5731238

Garmire

Gliske

Nguyen

: The training of next generation data scientists in biomedicine. Pac Symp Biocomput. 2017;22:640–5. 27897014

10.1142/9789813207813_0059

PMC5425257

Chaussabel

Rinchai

: Using “collective omics data” for biomedical research training. Immunology. 2018;155(1):18–23. 29705995

10.1111/imm.12944

PMC6099165

Chaussabel

: Literature profiling workshop, introduction session slides (S File 1). figshare. Journal contribution. 2021. 10.6084/m9.figshare.13669070.v2

Chaussabel

: Literature profiling workshop: steps 1-3 (S File 2). figshare. Journal contribution. 2021. 10.6084/m9.figshare.14160329.v3

Chaussabel

: Literature Profiling Workshop: Step 5 (S File 3). figshare. Journal contribution. 2021. 10.6084/m9.figshare.14161484.v1

Bektas

Noetzel

Veeck

: The ubiquitin-like molecule interferon-stimulated gene 15 (ISG15) is a potential prognostic marker in human breast cancer. Breast Cancer Res. 2008;10(4):R58. 18627608

10.1186/bcr2117

PMC2575531

Tecalco-Cruz

Cortés-González

Cruz-Ramos

: Interplay between interferon-stimulated gene 15/ISGylation and interferon gamma signaling in breast cancer cells. Cell Signal. 2019 Feb;54:91–101. 30500379

10.1016/j.cellsig.2018.11.021

Padovan

Terracciano

Certa

: Interferon stimulated gene 15 constitutively produced by melanoma cells induces e-cadherin expression on human dendritic cells. Cancer Res. 2002 Jun 15;62(12):3453–8. 12067988

Kariri

Alsaleem

Joseph

: The prognostic significance of interferon-stimulated gene 15 (ISG15) in invasive breast cancer. Breast Cancer Res Treat. 2020 Oct;19. 33073304

10.1007/s10549-020-05955-1

PMC7867506

Wood

Pan

Z-K

Seavey

: The ubiquitin-like protein, ISG15, is a novel tumor-associated antigen for cancer immunotherapy. Cancer Immunol Immunother. 2012 May;61(5):689–700. 22057675

10.1007/s00262-011-1129-9

PMC4561532

Burks

Reed

Desai

: Free ISG15 triggers an antitumor immune response against breast cancer: a new perspective. Oncotarget. 2015 Mar 30;6(9):7221–31. 25749047

10.18632/oncotarget.3372

PMC4466680

Desai

Wood

Tsai

Y-C

: ISG15 as a novel tumor biomarker for drug sensitivity. Mol Cancer Ther. 2008 Jun;7(6):1430–9. 18566215

10.1158/1535-7163.MCT-07-2345

PMC2561335

Qiu

Hong

Yang

: ISG15 as a novel prognostic biomarker for hepatitis B virus-related hepatocellular carcinoma. Int J Clin Exp Med. 2015;8(10):17140–50. 26770308

PMC4694208

Broering

Trippler

Werner

: Hepatic expression of proteasome subunit alpha type-6 is upregulated during viral hepatitis and putatively regulates the expression of ISG15 ubiquitin-like modifier, a proviral host gene in hepatitis C virus infection. J Viral Hepat. 2016 May;23(5):375–86. 26833585

10.1111/jvh.12508

Katsounas

Hubbard

Wang

: High interferon-stimulated gene ISG-15 expression affects HCV treatment outcome in patients co-infected with HIV and HCV. J Med Virol. 2013 Jun;85(6):959–63. 23588721

10.1002/jmv.23576

Wang

R-G

Kaul

Zhang

D-X

: Interferon-stimulated gene 15 as a general marker for acute and chronic neuronal injuries. Sheng Li Xue Bao. 2012 Oct 25;64(5):577–83. 23090498

PMC3587786

Scagnolari

Monteleone

Selvaggi

: ISG15 expression correlates with HIV-1 viral load and with factors regulating T cell response. Immunobiology. 2016 Feb;221(2):282–90. 26563749

10.1016/j.imbio.2015.10.007

Yuan

: Interferon-stimulated gene 15 expression in systemic lupus erythematosus: Diagnostic value and association with lymphocytopenia. Z Rheumatol. 2018 Apr;77(3):256–62. 28204879

10.1007/s00393-017-0274-8

Care

Stephenson

Barnes

: Network Analysis Identifies Proinflammatory Plasma Cell Polarization for Secretion of ISG15 in Human Autoimmunity. J Immunol. 2016 Aug 15;197(4):1447–59. 27357150

10.4049/jimmunol.1600624

PMC4974491

Chaussabel

: Literature profiling workshop: HBKU handout (S File 4). figshare. Journal contribution. 2021. 10.6084/m9.figshare.14166395.v1

Chaussabel

: Literature profiling workshop: HBKU intro presentation (S File 5). figshare. Journal contribution. 2021. 10.6084/m9.figshare.14166500.v1

Toufiq

Roelands

Alfaki

: Annexin A3 in sepsis: novel perspectives from an exploration of public transcriptome data. Immunology. 2020 Jul;18. 32682335

10.1111/imm.13239

PMC7692248

Rinchai

Kewcharoenwong

Kessler

: Increased abundance of ADAM9 transcripts in the blood is associated with tissue damage. F1000Res. 2015;4:89. 27990250

10.12688/f1000research.6241.2

PMC5130078

Roelands

Garand

Hinchcliff

: Long-Chain Acyl-CoA Synthetase 1 Role in Sepsis and Immunity: Perspectives From a Parallel Review of Public Transcriptome Datasets and of the Literature. Front Immunol. 2019;10:2410. 31681299

10.3389/fimmu.2019.02410

PMC6813721

Rinchai

Boughorbel

Presnell

: A curated compendium of monocyte transcriptome datasets of relevance to human monocyte immunobiology research. F1000Res. 2016;5:291. 27158452

10.12688/f1000research.8182.2

PMC4856112

Roelands

Decock

Boughorbel

: A collection of annotated and harmonized human breast cancer transcriptome datasets, including immunologic classification. F1000Res. 2017;6:296. 29527288

10.12688/f1000research.10960.2

PMC5820610

Huang

SSY

Al Ali

Boughorbel

: A curated collection of transcriptome datasets to investigate the molecular mechanisms of immunoglobulin E-mediated atopic diseases. Database (Oxford). 2019 01;2019. 31290545

10.1093/database/baz066

PMC6616200

Bougarn

Boughorbel

Chaussabel

: A curated transcriptome dataset collection to investigate the blood transcriptional response to viral respiratory tract infection and vaccination. F1000Res. 2019;8:284. 31231515

10.12688/f1000research.18533.1

PMC6567289

Marr

Boughorbel

Presnell

: A curated transcriptome dataset collection to investigate the development and differentiation of the human placenta and its associated pathologies. F1000Res. 2016;5:305. 27303626

10.12688/f1000research.8210.2

PMC4897750

Mackeh

Boughorbel

Chaussabel

: -A curated transcriptomic dataset collection relevant to embryonic development associated with in vitro fertilization in healthy individuals and patients with polycystic ovary syndrome. F1000Res. 2017;6:181. 28413616

10.12688/f1000research.10877.1

PMC5365227

Blazkova

Boughorbel

Presnell

: A curated transcriptome dataset collection to investigate the immunobiology of HIV infection. F1000Res. 2016;5:327. 27134731

10.12688/f1000research.8204.1

PMC4838008

Bougarn

Boughorbel

Chaussabel

: A curated transcriptome dataset collection to investigate inborn errors of immunity. F1000Res. 2019;8:188. 31559014

10.12688/f1000research.18048.2

PMC6749933

Rahman

Boughorbel

Presnell

: A curated transcriptome dataset collection to investigate the functional programming of human hematopoietic cells in early life. F1000Res. 2016;5:414. 27347375

10.12688/f1000research.8375.1

PMC4916988

Rinchai

Presnell

Vidal

: Blood Interferon Signatures Putatively Link Lack of Protection Conferred by the RTS,S Recombinant Malaria Vaccine to an Antigen-specific IgE Response. F1000Res. 2015;4:919. 28883910

10.12688/f1000research.7093.2

PMC5580375

Rawat

Rinchai

Toufiq

: A Neutrophil-Driven Inflammatory Signature Characterizes the Blood Transcriptome Fingerprint of Psoriasis. Front Immunol. 2020 [cited 2020 Dec 7];11. 33329570

10.3389/fimmu.2020.587946

PMC7732684

Reference Source

Rinchai

Altman

Konza

: Definition of erythroid cell-positive blood transcriptome phenotypes associated with severe respiratory syncytial virus infection. Clin Transl Med. 2020;10(8):e244. 33377660

10.1002/ctm2.244

PMC7733317

Rinchai

Chaussabel

: A training curriculum for retrieving, structuring, and aggregating information derived from the biomedical literature and large-scale data repositories. [version 1; peer review: 2 approved with reservations]. F1000Research. 2022;11:994. 10.12688/f1000research.122811.1

Rinchai

Chaussabel

: Assessing the potential relevance of CEACAM6 as a blood transcriptional biomarker [version 1; peer review: 1 approved with reservations]. F1000Research. 2022;11:1294. 10.12688/f1000research.126721.1

10.5256/f1000research.145671.r173013

Reviewer response for version 2

Matarese

Valerie

1 Referee https://orcid.org/0000-0003-2687-194X 1Authors' editor and editorial consultant, Self-employed, Vidor TV, Italy

Competing interests: No competing interests were disclosed.

25 5 2023

2023

This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

recommendation

approve

I have read the revised manuscript and am pleased to see that the authors acted on most of my comments. As a result, the manuscript has improved on many levels. The inclusion of this manuscript in the literature on scientific writing workshops will be useful to the scientific community, as it illustrates some unique approaches and viewpoints. I therefore give my approval to the manuscript and look forward to seeing it indexed in bibliographic databases.

Is the rationale for developing the new method (or application) clearly explained?

Partly

Is the description of the method technically sound?

Yes

Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Partly

If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Yes

Are sufficient details provided to allow replication of the method development and its use by others?

Yes

Reviewer Expertise:

I am a former biomedical researcher with 25 years' experience as a biomedical editor in different capacities and 15 years' experience as instructor of research writing at the PhD level. I am also a scholar of academic editing.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

10.5256/f1000research.39457.r96508

Reviewer response for version 1

Matarese

Valerie

1 Referee https://orcid.org/0000-0003-2687-194X 1Authors' editor and editorial consultant, Self-employed, Vidor TV, Italy

Competing interests: No competing interests were disclosed.

20 10 2021

2021

recommendation

reject

Overview

This method article describes a short workshop (3 hours) with five stated teaching objectives: bibliographic research in PubMed on gene-related topics (“retrieval”); extraction of relevant keywords (“concepts”) from article titles; “profiling” (counting the number of articles per concept); using the data and online tools to create interactive graphics (“visualization”); and “writing a narrative review” using a semi-automated method in a spreadsheet. I welcome a workshop description for teaching bibliographic research, and I am sure that workshop trainees will enjoy learning to make treemaps and circle packing charts. However, I have substantial reservations about the last objective and thus, as peer reviewer, I cannot endorse the method as a whole.

Regarding F1000Research questions

The rationale for the workshop is clearly given, but no valid justification is given for using a spreadsheet to write a narrative review (fifth objective) for the Introduction section of a research article or for a review article. The conclusions about the method and its performance are supported in part by the findings presented in the article, but the generated nonsense text is clear evidence that the proposed method for automated text generation should not be published.

Major comment

The fifth objective, “Writing a narrative”, is associated with two skills that workshop trainees can expect to learn: “Assimilation of biomedical knowledge (synthesis and presentation)” and “Scientific publication (optional)”. However, as described in the manuscript, the workshop is unable to fulfill these promises at all. In this fifth part of the workshop, participants are instructed to read the “abstract and/or full text and record in a spreadsheet” a series of data. They then use the spreadsheet’s concatenate function to generate “standard sentences” and collate these into a “cohesive paragraph or section”. Finally, the text is “polished” by the students, an instructor or a “professional scientific copyeditor”. There are enormous problems with all of this.

First and foremost, the method is proposing the automated generation of text for publication based primarily on abstracts (“full text where possible/necessary”) and without critical appraisal. Only in the conclusions of the manuscript is there any minimal mention of this problem: “Nevertheless, … extraction of information from abstracts will always require some critical evaluation and decision making that cannot be automated” (emphasis added). I cannot endorse a method for the computer-assisted writing of research articles and reviews, as it is unscholarly, wasteful to the reviewers and (if published) readers who will have to sift through substantial nonsense text, and damaging to knowledge advancement.

That this method can generate nonsense text is seen in the two pages of “Illustrative case”. Part A (page 11) shows the computer-generated “standard sentences” which are close to gibberish, for example:

The abundance of ISG15 Proteins measured by immunohistochemistry is increased in vivo in human immune cell markers “in patients with” invasive BC patients with long-term follow-up compared to … … ………… … .. [33073304].

Part B (page 12) shows the “edited narrative” which “was prepared by a professional scientific copyeditor”. Given both the poor science and poor English grammar and style of this narrative, I doubt that a professional editor worked on it. Furthermore, the use of an editor to create meaningful text from the “standard sentences” in Part A would amount to ghost writing. Here is just one example from the supposedly edited text:

Compared to healthy donors, TaqMan assays revealed high ISG15 expression (mRNA and protein) as well as an increased frequency of ISG15-SNPs in human PBMC from untreated patients with HIV infection.

TaqMan assays do not inform on protein levels. The frequency of a SNP cannot change unless there is gene duplication (what changes is the allele frequency). There is no need to use “human” to describe cells from HIV patients. There is a misplaced modifier before “TaqMan” and an unwanted hyphen.

I therefore feel that, for a revised version of this manuscript to be approved, the fifth objective should be deleted along with its two associated skills (impossible to achieve in a short workshop). In addition, a paragraph should be added with comments on the need for early career researchers to learn to read the scientific literature at different levels (from scanning to critical appraisal of the full text) and to appreciate that scientific findings must be replicated and validated before they can be considered “true”.

Minor comments

The title is a hard read, because there are six words between “Organizing” and what gets organized. I suggest: Organizing training workshops on gene literature …. for early career researchers”.

The word “profiling” is used to the point of being meaningless. It is used six times in the 20-line Introduction, in the context of literature, omics and transcriptome. It should be deleted after “omics” and “transcriptome”, as these words are best used as nouns, not adjectives describing the vague “profiling”. Regarding “literature profiling”, this non-standard expression should be defined early in the text, not in the Methods section (bottom page 3). Here, profiling means “information extraction” from titles and “determination of keyword frequencies”. This definition should be given in the Introduction as soon as “literature profiling” is mentioned.

Throughout the text: The use of conditional mood (“this session should present”, “the presentation should last” and so on) is unnecessary, burdens the text and slows reading. The simple present tense can be used in most cases. A method is not a recommendation (“should”) but a description (“is”).

Throughout the text: The use of italics in the “illustrative cases” makes reading difficult, especially because of the extensive use of quotation marks and parentheses. Italics should be reserved for emphasis on small pieces of text. Here, instead, a block quote (indented text) would be better. Text boxes can also help.

Page 2: “some independent preparation”. Does this mean independent study, exercises, getting organized technically?

Page 2: Ref. 34 is cited right after Ref. 4. It should be renumbered 5. I confused it with Refs. 3,4 (missing a comma). Similarly Ref. 35 is cited too early, on page 6.

Page 3, line 3: “on any given specific research topics” should be changed to “on any given research topic”. Topic should be singular. Specific is unneeded and redundant with “given topic”.

Page 3: “for a gene with ± 1000 associated articles and five participants”. This phrase is too concise to have meaning, as a gene cannot have participants.

Page 4, Step 1. No mention is made of NCBI’s Advanced Search Builder; MeSH terms; or truncation by asterisk. The manuscript should explain why these advanced features are not taught.

Page 4, bottom: “The field restriction tag [tw] (or alternatively the more restrictive [tiab]) should be added after each term… to limit the search to titles and abstracts”. Actually, it is [tiab] that limits the search to titles and abstracts. The broad [tw] searches almost everything (see https://pubmed.ncbi.nlm.nih.gov/help/#tw).

Page 5: “examples provided above”. Where? Vague cross-references like this make reading difficult.

Page 6: “For instance, when assessing the novelty …”. This incomplete sentence is lacking a subject-verb pair. Similar problem on page 13, bottom.

Page 6, bottom: “The Boolean argument AND is added”. AND is a Boolean operator, not argument.

Page 6, bottom: ‘… AND (“Liver cancer”[tw] OR “Liver carcinoma”[tw] OR “Hepatic carcinoma”[tw], etc.)’. A query cannot contain “, etc.”. It should be removed, and a final period added to the sentence.

Page 9, Step 5. This part is excessively wordy and suggests an uneasiness with the concepts. We read “is the measure by which productivity is measured”, “it can be at first important to be able to identify information that is worth capturing” and “A specific topic would be selected as the focus … to limit the amount of literature that would need to be covered”.

Page 13: Webex. Not WebEx.

Punctuation

Comma after “Supporting material” in the third paragraph of Introduction. Comma after “This is a skill that is worth developing” (page 9).

Replace ± with “about” (first paragraph of Methods).

Hyphens: no hyphen after -ly adverbs, e.g. “early career researcher”. Use hyphens in “gene-associated literature”, “false-positive result” and “false-negative result” (but not in “false positives” or “false negatives”).

There are many instances where the required space between words is missing, especially when a quotation mark is used, e.g. “For example, under’Human diseases and … associated keywords could be’liver cancer’,’hepatic carcinoma’, …” (page 5). The entire manuscript should be checked and spaces added where needed. A hint is that the curly quotation mark is in the wrong direction before the word when a space is missing.

Double quotation marks should be used in all cases except when nesting is needed. Nesting should be used on page 5 where the sentence:

This was rectified by adding the AND “cross-reactive protein [tw] argument to each of the ambiguous acronym.

should be changed to:

This was rectified by adding the ‘AND “cross-reactive protein [tw]”’ argument to each of the ambiguous acronyms.

or more simply:

This was rectified by adding ‘AND “cross-reactive protein [tw]”’ to each of the ambiguous acronyms.

Is the rationale for developing the new method (or application) clearly explained?

Partly

Is the description of the method technically sound?

Yes

Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Partly

If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Yes

Are sufficient details provided to allow replication of the method development and its use by others?

Yes

Reviewer Expertise:

I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.

Chaussabel

Damien

The Jackson Laboratory for Genomic Medicine, USA

Competing interests: The authors have no competing interests to declare.

17 3 2023

Overview

We appreciate the time spent reviewing this work and the constructive feedback given. The reason we are coming back to revising this article a little bit late is because we wanted first to complete the development of a more rounded training curriculum. As a result, a study guide and use case have recently been published in F1000Research ( https://f1000research.com/articles/11-994, https://f1000research.com/articles/11-1294). Literature profiling remains nonetheless a major component of this training curriculum and could be run as stand-alone training/workshop. Also, we also wanted to circle back to this manuscript at this time and address all outstanding comments and add references to the more recent work.

We have addressed the main comment raised in the review by eliminating the step in the workflow that involved automatically generating text from the information provided by workshop participants. We did not intend for this step to cause controversy and were simply trying to practically train early-career scientists in countries where English literacy may not be as high as in more privileged countries. The question then becomes, how can these brilliant scientists overcome the language barrier and fulfil their potential? The approach we attempted, as an experiment, was only meant to be a first step towards trainees becoming independent writers, eventually able to produce literary work without artificial crutches. However, we admit that the approach may have been flawed or misguided (or both!), and we acknowledge your point. We are happy to remove this step from the manuscript.

Regarding F1000Research questions

Thanks for raising these concerns. We have now removed the steps that pertains to automated text generation (5e and 5f). We feel that capturing relevant information and recording it in a spreadsheet is still a worthwhile exercise. Indeed, participants can learn in the process what information they should be looking for in the work of others and make sure to report in their own (e.g. abstracts often fail to mention in what species the work is performed or if abundance of protein or transcripts is measured). We now write instead as step 5e:

“The information captured in the spreadsheet will serve as a synopsis that participants will be able to rely upon for developing a written narrative, which could be used in an introduction or review article.”

The spreadsheet has also been amended accordingly, removing the functions used to generate text. And the text that was automatically generated has been removed from the manuscript. We retained the written narrative, simply prefacing the paragraph with the following statement:

“Narrative, based on the information captured in the spreadsheet: “

Major comment

Compared to healthy donors, TaqMan assays revealed high ISG15 expression (mRNA and protein) as well as an increased frequency of ISG15-SNPs in human PBMC from untreated patients with HIV infection.

We amended Step 5 as described above, removing altogether any notion of automated text generation, while keeping the step that consists in capturing and recording key information in a spreadsheet. If necessary, even this could be removed, but again as stated above we feel it is a worthwhile exercise.

We amended our learning objectives, removing the two points mentioned in your critique and writing instead:

Extraction of biomedical information (capturing information in a structured format).

We also added wording highlighting the need for participants to go beyond scanning titles and abstracts and read research papers:

“Thus, early career researchers need to develop the ability to read the scientific literature at different levels: from “scanning” a large number of articles to reading the full text version for a more in depth understanding.”

And for the record – since integrity is no small matter in science - we maintain that a professional and reputable science editor did work on this text, as we had stated earlier; even though as pointed out the service rendered was maybe not of the highest quality in this case (we have been otherwise been quite happy with them).

Minor comments

Thank you for the helpful edits, which have been incorporated throughout the text and to the best of our abilities. We also took your advice and changed the title of the article to improve legibility.

We have deleted profiling when associated with omics data and defined literature profiling early on in the introduction.

That makes sense and we have edited the text throughout.

Text boxes sound ideal. We will reach out to the editor to see if this is feasible.

Page 2: “some independent preparation”. Does this mean independent study, exercises, getting organized technically?

We clarified this point by rewording this sentence, that now reads: “Trainees can be asked to perform tasks between each of the sessions.”

Page 2: Ref. 34 is cited right after Ref. 4. It should be renumbered 5. I confused it with Refs. 3,4 (missing a comma). Similarly Ref. 35 is cited too early, on page 6.

Indeed, we will communicate this to the editor so it can be addressed in the formatted document.

This sentence has been edited accordingly.

Page 3: “for a gene with ± 1000 associated articles and five participants”. This phrase is too concise to have meaning, as a gene cannot have participants.

We have edited this sentence to read:

“For instance, for a gene with about 1,000 associated articles, five participants could work together on this same gene, each focusing on a different theme.”

Page 4, Step 1. No mention is made of NCBI’s Advanced Search Builder; MeSH terms; or truncation by asterisk. The manuscript should explain why these advanced features are not taught.

Indeed, thanks for the suggestion. We have re-written this section and mention the availability of the advanced search builder and provided the corresponding URL. We also pointed to training material available on the NCBI website. In our experience truncation by asterisk can produce unwanted results and increase false positives, also did not want to recommend using it in this case.

Yes, this is correct, and this entire section has been rewritten for clarity. We found the [tw] field to be effective in preventing PubMed from trying best guesses when few entries are returned. [tiab] can be used when users wish to further restrict the search. But both should provide good results for our purpose.

Page 5: “examples provided above”. Where? Vague cross-references like this make reading difficult.

Thanks for pointing this out. We are now providing those examples instead.

Page 6: “For instance, when assessing the novelty …”. This incomplete sentence is lacking a subject-verb pair. Similar problem on page 13, bottom.

This has been corrected.

Page 6, bottom: “The Boolean argument AND is added”. AND is a Boolean operator, not argument.

Indeed, this has been corrected.

This has been corrected as well.

We rewrote and condensed this paragraph.

Page 13: Webex. Not WebEx.

This has been corrected.

Punctuation

Thank you, that was very useful, and we have now made the necessary edits throughout.

Comma after “Supporting material” in the third paragraph of Introduction. Comma after “This is a skill that is worth developing” (page 9).

Replace ± with “about” (first paragraph of Methods).

Double quotation marks should be used in all cases except when nesting is needed. Nesting should be used on page 5 where the sentence: ' This was rectified by adding the AND “cross-reactive protein [tw] argument to each of the ambiguous acronym.' should be changed to: ' This was rectified by adding the ‘AND “cross-reactive protein [tw]”’ argument to each of the ambiguous acronyms.' or more simply: ' This was rectified by adding ‘AND “cross-reactive protein [tw]”’ to each of the ambiguous acronyms.'

10.5256/f1000research.39457.r86220

Reviewer response for version 1

McDowell

Gary S.

1 Referee https://orcid.org/0000-0002-9470-3799 1Lightoller LLC, Chicago, IL, USA

Competing interests: No competing interests were disclosed.

15 6 2021

2021

recommendation

approve

The authors discuss the need and provide a curriculum for early career researchers to develop the ability to survey and synthesize the background literature to a field in an appropriate manner. This is demonstrated with publicly available “omics” data as an example.

This article gives an excellent description of a training workflow that is suitable and adaptable for training in systematic literature profiling. The article provides clear illustrative use case examples that are very helpful in demonstrating the results of the training workflow, and provide excellent materials for others to work from in developing similar materials.

Is the rationale for developing the new method (or application) clearly explained?

Yes

Is the description of the method technically sound?

Yes

Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Yes

If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Yes

Are sufficient details provided to allow replication of the method development and its use by others?

Yes

Reviewer Expertise:

Early career researcher experiences, training and professional development.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.