Advancing the international data science workforce through shared training and education

5.1 Identifying resources for indexing

Training portals, such as ELIXIR TeSS and the BD2K TCC Portal, powered by ERuDIte, aim to provide access to high-quality training resources that are available in a wide variety of online sources. Identifying such resources is a challenging problem, given that they are widely distributed and continuously evolving. We are following a three-pronged approach to resource identification:

5.2 Describing resources

Having rich, structured metadata annotations benefits multiple audiences that interact with resource aggregators, such as ELIXIR’s TeSS and the BD2K TCC’s ERuDIte. First, with richer metadata, learners have more information to use when evaluating whether or not a resource suits their needs. Secondly, search engines can more easily find and index resources, thereby increasing discoverability by learners, resource creation organizations, and any resource aggregator. Finally, providing structured metadata facilitates data interchange between resource indexes. However, to maximize these benefits, common metadata standards are desirable.

In this workshop, we focused on aligning the metadata standards used by TeSS and ERuDIte. TeSS (Beard et al., 2016) collects training events and materials related to the life sciences, with a special interest in bioinformatics, whilst ERuDIte focuses on materials related to data science (Ambite et al., 2017). Both portals are interested in creating a common representation of online training materials, the organizations that create and host them, and the people who create and teach them.

Currently, Bioschemas.org has defined specifications on how to represent life science training materials, people, organizations and events, based on the CreativeWork, Person, Organization and Event classes of the prevalent and popular Schema.org standard. Given ELIXIR’s membership in Bioschemas.org, in order to facilitate data exchange between TeSS, ERuDIte and other training sites and indexes, ERuDIte has aligned its learning resource descriptions to a common standard built from Schema.org. TeSS and ERuDIte have determined a core set of fields for our respective usage of the CreativeWork, Person and Organization classes, and we have documented how we use the properties of each class. These are the core properties and classes in ERuDIte, with their respective namespaces. Properties are also inherited from superclasses:

ERuDIte is aligned with Bioschemas.org and Schema.org. ERuDIte includes three additional properties not available in Schema.org.

TeSS and ERuDIte provide concept tags to every material, but the two indexes use different tagging ontologies. TeSS uses EDAM (Ison, et al., 2013), which is expanded to suit the specific coverage of life science, healthcare and bioinformatics events and materials. ERuDIte uses the custom Data Science Education Ontology (DSEO), which was designed using both top-down and bottom-up approaches (Ambite et al., 2017). In a top-down fashion, the DSEO was first manually designed as an initial ontology based on expert knowledge. In the bottom-up approach, the ERuDIte team automatically computed topic models from materials’ text to identify additional concepts. Finally, the team curated and organized all these concepts. We are creating a mapping between DSEO and EDAM, which will allow us to see which materials we are interested in and then tag the materials with the terms in our respective ontologies. While this tagging ontology mapping will emerge from the TeSS and ERuDIte collaboration, it also opens up collaborations with any other institutions that use EDAM, such as GOBLET, H3ABioNet and EMBL-ABR, who also provide a rich set of training content suitable for indexing.

5.3 Helping users find resources

Currently, TeSS and ERuDIte use similar approaches to enable resource discovery. Both portals have simple, intuitive search interfaces that allow free text searching, and also provide resource tags that facilitate search filtering. Although search interfaces handle the majority of resource-discovery needs, both TeSS and ERuDIte hope to provide other meaningful ways to explore resources.

First, both portals are actively working on initiatives that allow learners to create personalized resource collections. While these collections are formed to serve the specific learning needs of their creators, they also serve the needs of learners who are interested in similar topics and learning paths. In addition to these learner-created collections, both TeSS and the TCC portal aim to introduce expert-level collections, which will contain recommended resources that learners can use as templates for their own learning. Given the growth and rapid evolution of the learning domains of both ERuDIte and TeSS, having expert validated collections will help learners find a starting place for their learning journey through a topic or concept.

The BD2K TCC is also interested in providing interactive resource browsing interfaces. Currently, the TCC provides resource summarizations that guide learners to resources based on topics and providers of interest. In the future, the TCC aims to provide a meaningful resource-to-resource browsing interface where learners can see and select groups of similar resources. This visualization paradigm can then be shared with TeSS, if a browsing visualization is requested by its learning community.

Lastly, both the TCC and TeSS are interested in creating personalized recommendations for learners. Both sites are currently constrained by the size of their user bases, but as more user data are available, both portals will be able to provide resource-level suggestions that can be used on the portal directly or through community outreach methods (e.g., email campaigns or lists).

5.4 Reusing training resources

Ultimately, one clear way to assess the success of the TeSS and TCC web portals is the volume of resource reuse and re-indexing by other organizations. However, this relies heavily on the relevance and quality of the resources presented on TeSS’s and TCC’s respective websites. Both organizations hope that quality and relevance assessment can be accomplished by two groups: online learners and portal developers.

Resource quality evaluations from portal developers would need to consist of procedures that could semi-automatically check for resource status (e.g., active link or dead link), and duplication and accuracy of collected metadata. Using appropriate standards and schemas, the version history of training resources, and their author and contributor lists, can also be tracked in a transparent fashion, giving the content developers due credit and visibility for their materials. Resource quality and relevance assessments from online learners will come from user input, which could be implicit (e.g., the frequency of a resource in learners’ collections), or explicit (e.g., ratings and surveys). Both implicit and explicit feedback from learners will be essential for improving user experiences and resource reuse on both portals.

Once the reputation of the high quality of resources in the respective portals has been established, others will be encouraged to reuse and share them. Ideally, reuse will be accomplished by extracting our metadata markups, which will then spread the learning resource standards we have specified, allowing greater visibility and community use of our representations.

6.1 Learner profiles, learner modality and recommendations

The more information a user provides to the platform, the more personalized their learning experience can be. Initially, users may be asked to fill out a learner profile survey, which should ask them to rate their current level of expertise with various concepts, their interest in learning more about those concepts or achieving broader training goals. Users who would prefer to explore resources for themselves rather than take a survey can still receive personalized experiences. They can incrementally complete a learner profile through explicit or implicit indicators: e.g., rating resources they are familiar with, or simply browsing resources related to particular topics. As learners use the platform, personalization is most obvious through resource recommendations: when looking at a resource, they may be recommended to follow it with another that continues an appropriate progression for their goals. They may also be presented with an entirely automatically generated learning plan, where the topics and the resources reflect their expertise and interests. In the following sections, we explore learner modeling and recommendations in more depth.

Mastery Rubric and Bloom’s Taxonomy. The Bloom Taxonomy describes six levels of cognitive skills, providing a description over a coded continuum of expertise that would be useful for resource developers. The levels are: 1) Remember/Reiterate, 2) Understand/Summarize, 3) Apply/Illustrate, 4) Analyze/Predict, 5) Evaluate/Synthesize, and (6) Create/Apply/Judge; these increase in cognitive complexity with each level (Bloom 1956). Mastery Rubric (MR) development provides a powerful methodology for developing curricula and assessments designed to teach key Knowledge, Skills and Abilities (KSAs) for specific disciplines (Tractenberg et al., 2010; Tractenberg et al., 2017). The MR formulation involves outlining how KSAs align with performance levels to provide a trajectory for students as they advance in a subject. MRs have been developed for Clinical Research (Tractenberg et al., 2017), Ethical Reasoning (Tractenberg et al., 2015), and Statistical Literacy and Evidence-Based Medicine (Tractenberg et al., 2016).

6.2 Modeling learners

6.2a High-level structure. There are many ways to model learners, but we will focus on the aspects that are important to enabling the automated personalization of learning experiences. The aspects we have identified are a) the learner, b) the concept-skill combinations comprising the domain of interest, and c) the available learning resources. We will describe these in this section.

6.2b A four-dimensional learner model. Perhaps the key question when considering how we might develop user-specific capabilities for an individual within a learning portal is to think carefully about how we model that individual. We propose modeling users with a matrix that links subject domains and skills to users’ expertise and interests. This is shown diagrammatically below.

The example in the figure depicts the situation of a learner who wants to find out about the field of genetics but does not know where to start. It shows that the same learner is also an R programmer with existing expertise, but wants to learn Python to fulfill the requirements of a project. The system should recommend courses that meet these interests, whilst leveraging elements in which the learner already has expertise.

The key aspect of this underlying model is that the table shown above would need a large number of rows that codify the various different domains and skills possessed by a trained individual. Clearly, how we go about modeling each of these dimensions is crucial to the utility or effectiveness of this approach. We begin here to outline a first pass over the process of characterizing these dimensions, based on the various ontologies and conceptual schema being developed within the GTE community to date.

6.2c Domains. To model people’s learning goals and their progress toward them, it is important to have a model of the domain of interest. Domain models take various forms, ranging from ontologies that domain experts and knowledge engineers construct by hand at great cost, to methods that are less reliable but fast and scalable, such as automatic topic modeling using Latent Dirichl location (LDA; Blei et al., 2003).

We do not prescribe a method for creating a domain model or a precise form, but there are several requirements for supporting personalized learning. The domain model must provide a list of discrete, named concepts. These allow the user model to represent the state of the learner’s knowledge about particular aspects of the domain, and those aspects that they have expressed interest in learning more about, either explicitly or implicitly: e.g., by browsing resources related to a particular concept. The model of the domain must also connect the identified concepts to the available resources with weighted links, giving the degree to which each resource pertains to a particular concept. Concepts can be connected to one another by various relations, including prerequisite, similarity and part-of. These relations are used to recommend concepts and their associated learning resources, described in Section 4.

6.2d Skills and abilities. Following the methodology defined in the development of Mastery Rubrics³, we encode KSAs as a key element. To our knowledge, the development of taxonomies or ontologies for skills and abilities in biomedical data science is still in its infancy, but existing resources provide informal representations that support this notion. For example, Welch et al. (2016) provide a preliminary encoding of “core competencies” that broadly outline various general skill-sets required for bioinformatics.

Going beyond these general categories into more detail may require a finer-grained representation of specific skills and abilities. Potential starting points are provided by the EDAM ontology⁴. This resource provides a taxonomy of operations that could act as a proxy in the specific area of bioinformatics-based data analysis. Similarly, the Data Science Education Ontology (DSEO)⁵ provides a taxonomy of data science processes that could be elaborated into a more complete description of skills and abilities that we could use here.

Beyond these resources, a starting set of high-level skill categories could include: simple comprehension, concept mastery, basic procedural execution, procedural troubleshooting, and procedural implementation. It may then be necessary to further specialize these abilities within domains, programming languages, data science processes, etc.

6.2e Knowledge/expertise. Here, we follow the encoding provided by the Bloom Taxonomy/Mastery Rubric methodology (Bloom, 1956; Tractenberg et al., 2010), which specifies expertise as an ordinal scale with the following categories (in ascending order of expertise): a) beginning, b) novice, c) competent, and d) proficient. Subsequent incarnations of the Mastery Rubric specify a trajectory from a) Novice, to b) Beginner, to c) Apprentice, to d) Journeyman.

6.2f Interest. Learners may have only a general idea of what they are interested in (e.g., “data science” or “bioinformatics”), or they may have specific interests (e.g., “using R to analyze genomic data”). Part of a functional learner model is to record which concepts – at different levels of granularity – a learner would like to learn more about. Wherever possible, this interest should be assigned to the particular skill: e.g., conceptual vs. procedural. Our model of learners’ interests is a mixture of what they tell us (e.g., by defining their goal), and what we infer (e.g., from identifying themes in learning resources they access).

6.2g Leveraging existing resources and implementation. ERuDIte (Ambite et al., 2017; Van Horn et al., 2018) consists of an index of both manually and automatically identified data-science resources – predominantly online courses and videos – which are tagged with relevant concepts from DSEO. Ongoing work seeks to identify natural sequences of resources based on prerequisites, building on the architecture to do so from the TechKnAcq project (Gordon et al., 2016).

The EDAM ontology (Ison et al., 2013) provides an extensive representation of four branches: a) Data (with Data Identifiers) denoting types of information that may be processed computationally; b) Operations that may be applied to those data; c) the many Formats in which the data may be expressed; and d) the Topic to which the data are relevant. This provides an excellent, exhaustive collection of element classes that may be combined to describe data-science processes and structures.

Currently in development on the TeSS portal are “Training Workflows”⁶. These are visual tools to help users navigate sets of diverse training resources. Broadly, three distinct types of training workflow are encapsulated in TeSS:

• abstract snapshots of typical data-analysis pipelines;

• hands-on, step-by-step guides through specific data-analysis tasks; and

• sign-posted routes charting developmental trajectories through training resources.

Training Workflows are constructed manually by domain experts and training professionals. A graphical WYSIWYG editor is provided to help trainers design their workflows, using standard components and headers. Within the editor, users may specify which training resources trainees should review in order to gain the requisite understanding of a topic before moving on; they may also select tools from the bio.tools⁷ registry, and/or databases, standards and policies from the FAIRSharing⁸ registry.

7.1 Making recommendations

When learners first use this learning portal, it is proposed that they would be required to complete a “user profile”, where they can state their prior experience/expertise, career level, their interests and, if known, explain what their end goal, career-wise, will be. It is envisaged that this information could then be used in one of two ways:

or

The theory behind the second case is that learners, sometimes, do not know what courses would be complementary or useful for their work, and do not always know what other courses are available. The courses would be recommended informally, in a similar manner to shopping for items on Amazon, using phrases such as “other learners who attended this course, also attended this course…”. Just as with Amazon ‘recommendations’, users would be free to choose which of the offerings resonated most with their personal training needs. These recommendations are not intended to be required for the accomplishment of a learning goal—they are a gentle guide for people who may be new to the topic or resource or for people who are uncertain about what resources they should view next. It is important to note that interest does not equate to expertise, and before making a course recommendation, the system would check that any prerequisite courses have been completed by the learner. Additionally, by filling out a comprehensive user profile when first using the platform, it is hoped that the recommendations would be accurate, and would ultimately allow more learning paths to be created.

To create personalized learning recommendations, we must also capture the relations between concepts in the domain. For example, before recommending a resource about the concept Machine Learning, we would like to ensure that the learner has familiarity with the concepts of Probability and Algorithms, as these concepts are joined by prerequisite relations. If a learner indicates interest in Machine Learning, we should suggest resources about Supervised Learning, Unsupervised Learning, EM Algorithm, etc., as these are all connected to Machine Learning by hierarchical part-of relations.

8.1 What are the commonalities across international data science training efforts?

The huge volumes of data now being routinely generated from high-throughput instrumentation (imaging data, data from sequencing platforms, etc.) have generated an insatiable demand for bioinformatics and data science training among bioscientists. There are many online training resources, including videoed lectures from conference talks, tutorials, short courses and degree programs, and materials on GitHub and other online repositories. However, while much effort has been placed in creating these materials, it can still be very difficult both for students, who wish to improve their skills, and for trainers, who wish to share and re-use other trainers’ materials, to find what they need. By following the FAIR Data Principles, international initiatives like the ERuDIte and TeSS portals seek to make it much easier to find and re-use training materials by identifying, indexing and collating them into central resources. Complementing these largely automated approaches are dedicated, manually maintained training portals, such as the one provided by the GOBLET Foundation. By working together, international training efforts can both identify commonalities and complementarities, and pinpoint synergistic opportunities to drive the field forward.

To make training materials FAIR, they must be adequately annotated and described so that they can be indexed appropriately. Currently, nascent annotation standards are not yet widely adopted, and there are no widely accepted quality metrics to indicate whether courses are delivering the requisite core competencies to trainees (Welch et al., 2016). Furthermore, to make a course truly reusable, all the information required to re-use the associated materials (including data-sets and tools used for practical exercises, and the exercises themselves) needs to be made available both to students and to fellow trainers. Jupyter notebooks, for example, containing code snippets and notes describing the code, are readily reusable, and there are many Jupyter notebooks on GitHub. But how can these be described so that they can be indexed and readily found? International collaboration could help to define agreed guidelines and standards for course and material description to alleviate this problem.

Alongside the challenge of making training materials FAIR is the growing demand for training, which is outstripping our ability to readily satisfy the changing skill requirements of bioscientists. To meet this demand, more instructors are needed, and new trainers must be adequately prepared and trained to train others. The Software Carpentry (SWC) Foundation has developed a model to ensure that trainers who deliver Software Carpentry- or Data Carpentry-branded workshops are trained to do so by undertaking an online or on-site (for SWC members) course, where they learn about pedagogy. Similarly, ELIXIR and GOBLET have developed a Train-the-Trainer (TtT) program (Via et al., 2017), which was piloted across Europe in 2016, to help build trainer capacity. ELIXIR has also extended these TtT course to Australia via BPA-CSIRO and EBI collaborations set up in 2012. Since then, more than 20 trainers have been trained to deliver bioinformatics training in Australia (Watson-Haigh et al., 2013). Against this background, the international data science and bioinformatics communities could together develop common guidelines for trainer training that would help to raise the quality of training delivery and increase the pool of available trainers.

Having participated in a course, a common theme among students is the desire for some form of certification. Without any aspect of assessment in the course, this will most commonly be a certificate of participation. However, many students are interested in a certificate of accomplishment; this is particularly common with MOOCs, such as those delivered by Coursera, EdX or Udacity. Here, on completion of end-of-course assessments to a set standard – and payment of a fee – students receive a certificate of accomplishment; those who opt to take the course without paying the fee are not certificated. There are practical limitations to the length of a course that determine whether and how it is meaningfully assessable. Nevertheless, for short courses, no assessment guidelines are yet available for universal adoption. Combining our joint experience and expertise, an opportunity therefore exists for the international community to make a set of recommendations for trainers who are developing course assessments, in order for some sort of universal certification to be created.

When reusing online materials, it is important to both cite the sources and give credit to the authors. Materials can be released under several different Creative Commons licences, each of which has different features. Understanding the differences between each licence, and when a particular one should be used, would be beneficial to course developers. Across the international bioinformatics and data science communities, offering guidance to trainers on what licence to use, and allowing end-users to understand their responsibilities when using licensed materials, would be very valuable, and may encourage more trainers to make their resources available online.

There are different types of training courses, including those that i) focus on skills development, ii) tackle data-analysis problems, and iii) develop and encourage creative thinking. Skills-development workshops are the most common, where students follow lectures and tutorial exercises, and perform tasks for which tools and data have already been prescribed: e.g., how to analyze RNASeq data. In data-analysis workshops, which are outcome-driven (and sometimes also referred to as ‘hackathons’ or ‘data jamborees’), rather than follow pre-set, generic examples, participants actually create something meaningful to them using freely available data. H3ABioNet has used the hackathon concept for different scenarios, including the development of Docker containers for selected data-analysis workflows and for collaborative analysis of specific data-sets, bringing together scientists from diverse backgrounds. Extending this idea, ‘creative thinking’ workshops bring together people with different expertise (e.g., bioinformaticians and mathematicians, or bioscientists and data scientists), but without a fixed agenda. Here, the idea is that by discussing issues that one group may have with their data, the other group may propose solutions. Ultimately, the outcome may be, for example, a research proposal for a grant application, leveraging the interests and expertise of both groups.

Data science in biology and medicine can learn from training approaches in other disciplines: e.g., astronomy and its enormous data-sets bring familiar challenges in data analysis and stewardship, and the same basic principles and underlying skill-sets apply. Bioscientists, however, generally have little exposure to the computing and data science skills now necessary for their work, and are likely to seek training much later in the data lifecycle (Attwood et al., 2017). IT professionals play a fundamental role in supporting the life sciences, as their skills are required to install bioinformatics tools on HPC platforms, and to troubleshoot the many problems that are likely to arise. However, rapid advances in technology mean that the current best-practice software for a given task changes rapidly, making it difficult to keep pace. Understanding what bioscientists do with these tools, how quickly they change, why they may need so many tools to analyze a given data-set, the demands of storing and analyzing their data, and so on, would be hugely informative to the IT professionals who support their work.

Internships offer another route via which bioscientists may augment their data science skills. The BD2K TCC supports – most commonly for junior faculty staff – the opportunity to partner with senior data scientists (for 2–3 weeks, or up to multiple months) in order to learn how best to analyze their data. At the end of these “road trips”, the bioscientists return to their own institutions and deliver training courses on what they have learned during their internship. H3ABioNet offers internships to acquire skills form experts at the host institution while working on their own data. The EU offers COST Action grants¹⁰, which exploit a similar training paradigm – so-called “short term scientific missions”. Finding out about such professional development opportunities can be very difficult, so providing a resource that indexes them would be very useful; this could also expand the program to become a globalized “road trip”, facilitating exchange of expertise internationally if sufficient funding were available. A combined international index could also offer a “matchmaking” service, connecting those who wish to gain a particular skill-set with those who are willing to host them and help them do so (rather like the Knowledge Transfer Program¹¹ advocated by the Centre for Proteomic and Genomic Research in Cape Town). Typically, such programs yield research collaborations, funding proposals and publications that last long after the road trips have ended. Consequently, junior researchers are not just supported in developing their skills, but may also benefit from new international collaborations that support them along their career paths.

Increasing the visibility of data science training will facilitate an increase in diversity and enable under-represented groups to better exploit these opportunities. Indeed, some initiatives are already underway to represent and encourage more participation by women in the data sciences. WiDS (Women in Data Science)¹² is a global initiative to inspire and educate data scientists worldwide that specifically supports women. Their 2017 conference was hosted at Stanford University, and was webcast live or had a delayed broadcast to 80 locations, in 30 countries, worldwide. Similarly, RLadies¹³ is an organisation that promotes gender diversity in the R community, and has multiple city-based chapters across the world.

Aside from connecting international training communities and efforts, a clear value of making training material more visible and discoverable is in sharing experience and expertise, reducing redundancy, and combining efforts to make individual contributions greater than the sum of their parts. Frequent meetings among the groups that represent these communities offer further opportunities for communication that will help each group to understand the other’s work, and to identify skill or knowledge gaps. However, these activities cannot continue without funding. Training bioscientists in data science is an essential investment for 21^st-century bioscience. Continued investment in data science training may be possible through large funding initiatives, such as NIH’s, or through large-scale multinational projects like ELIXIR. However, data science skills are also sought after by industry, with companies like Google, Microsoft and Amazon becoming active in biomedical and bioscience research. It may be possible to leverage this interest and partner with them to increase the pool of bioscientists skilled in data science. Furthermore, charitable organizations like the Bill and Melinda Gates or Gordon and Betty Moore Foundations, which fund grand challenges in bioscience, may also be interested in partnering to address the global skills shortage in data science.

Figure 1. Diagram of the high-level structure of relations among learners, concepts and skills, and resources.

Figure 2. Example model of a learner.