Approaches for containerized scientific workflows in cloud environments with applications in life science

Containers

Software container technology has emerged as a complement to VMs. While they offer a bit less isolation between processes, they are on the other hand more lightweight in terms of storage and memory requirements and are thus easier to share. By sharing the operating system kernel with the host computer, they are also much faster to launch and terminate.⁸ Containers encompass all dependencies of the provisioned tools and greatly simplify software installations for end users. The most widely used containerization solution is Docker (www.docker.com), but Singularity,⁹ uDocker¹⁰ and Shifter,¹¹ are recent alternatives that prevent users from running containers with root privileges, thus addressing the most common security issue when deploying containers in multi-tenant computing clusters such as on HPC clusters. Docker containers are commonly shared via Docker Hub.¹² There are also initiatives for standardizing containers in the life sciences such as BioContainers.¹³ Containers have seen an increased uptake in the life sciences, both for delivering software tools and for facilitating data analysis in various ways.^14-18

When running more than just a few containers, an orchestration system is needed to coordinate and manage their execution and handle issues related to e.g. load balancing, health checks and scaling. Kubernetes¹⁹ has over the last couple of years become the de facto standard container orchestration system, but Docker Swarm²⁰ and Apache Mesos²¹ are other well-known systems. A key objective for these systems is to allow the users to treat a cluster of compute nodes as a single deployment target and handle the packaging of containers on compute nodes behind the scenes.

Scientific workflows

While orchestration tools such as Kubernetes enable scientists to run many containers in a distributed environment (such as a virtual infrastructure on a cloud provider), the problem remains how to orchestrate the scheduling and dependencies between containers and being able to chain (define data dependencies between) them into an analysis pipeline. This is where scientific workflow management systems (WMSs) can be very useful; in fact, it can be difficult to carry out more complex analyses without such a system. Traditionally in data-intensive bioinformatics, an important task for WMSs has been to support analyses consisting of several components by running a set of command-line tools in a sequential fashion, commonly on a computer cluster. The main benefits of using a WMS include: i) making automation of multi-step computations easier to create, more robust and easy to change; ii) providing more transparency as to what the pipeline does through its more high-level workflow description and better reporting and visualization facilities; and iii) providing more reliable policies to handle transient or persistent error conditions, including strategies to recover from interrupted computations while re-using any partially finished data.

Using containerized components as the nodes in a scientific workflow has the advantage of adding isolation between processes and the complete encapsulation of tool dependencies within a container, and reduces the burden to administrate the host system where the workflow is scheduled to run. This means the system can be tested on a local computer and executed on remote servers and clusters without modification, using the exact same containers. Naturally, this opens up for execution on virtual infrastructures, even those provisioned on-demand with features such as auto-scaling. Apart from greatly simplifying access to the needed distributed compute resources, a positive side-effect is that the entire analysis becomes portable and reproducible.

Data ingestion and access to reference data constitute a key step when using cloud-based resources. When spawning a virtual infrastructure, data used in the analysis either needs to be available on storage connected to the compute nodes, or ingested into a locally provisioned file system. At the end of the analysis, the resulting data needs to either be put into persistent storage on the cloud or downloaded to local storage. However, the overhead with this can also be turned around. If data resides on storage connected to a cloud infrastructure, it is possible to simply move the workflow and containers there and “bring compute to the data”.

One key difference with running workflows locally on a desktop computer or server, or on a single HPC node, is that when running containerized workflows, the workflow engine often sits on another compute node than the executing containers. Also, the storage system is also typically separated, and accessible from any container, regardless of the exact compute node in a cluster where it is executed. Figure 1 illustrates this.

Figure 1. Overview of a containerized workflow with a workflow engine managing the dependency graph and scheduling of the containers encompassing the tools to run.

The tools pass data between each other by reading and writing to a shared storage system.

There are also multiple possibilities on how to run a workflow engine in relation to the containers, as the workflow engine can either be run on a separate system from the container cluster, such as a researcher’s local computer, or the workflow engine can itself run in a container on the cluster. Figure 2 illustrates this.

Figure 2. Comparison between two approaches for how to run the workflow engine.

It can run either a) on the user’s computer, or b) in a pod in the Kubernetes cluster.

Nextflow

Nextflow²² is a workflow framework that aims to ease the implementation of scientific workflows in a portable and reproducible manner across heterogeneous computing platforms and to enable the smooth migration of these applications to the cloud. The framework is based on the dataflow paradigm, a functional/reactive programming model in which tasks are isolated from each other and are expected to be executed in a stateless manner. This paradigm simplifies the deployment of complex workflows in distributed computing environments and matches the immutable execution model of containers.

Nextflow provides built-in support for the most used container runtimes i.e. Docker, Singularity, Shifter and uDocker (though the last two are undocumented because of still being in an experimental state). The use of containers is defined in a declarative manner, i.e. by specifying the container image that a workflow needs to use for the task executions. Once annotated in this way, the Nextflow runtime takes care of transparently running each task in its own container instance while mounting the required input and output files as needed. This approach allows a user that needs to target the requirements of a specific execution platform to quickly switch from one containerization technology to another with just a few changes in the workflow configuration file (e.g. the same application can be deployed in an HPC cluster using Singularity or in the cloud with Docker).

Great flexibility is given for container image configuration. Nextflow supports both the image-per-workflow pattern (the same image for all executed tasks) and the image-per-task pattern (each task uses a different container image). The container image used by a task can even be specified at runtime by a given input configuration.

Nextflow allows the deployment of containerized workloads in a cloud-native manner using different strategies, as briefly discussed in the following paragraphs.

Cloud-managed computing service: in this approach workflow tasks are submitted to a cloud provider batch execution service. The containerization and auto-scaling are delegated to the cloud service. The workflow inputs/outputs and intermediate result files need to be stored using the provider object storage service. Nextflow adapts the task definition, resource requests and container image to the cloud provider format, submitting the corresponding API requests and properly staging the task data. Currently, Nextflow has built-in support for the Amazon Web Services (AWS) Batch service and the Google Genomics Pipelines service.

Cloud unmanaged computing service: when using this deployment strategy, Nextflow takes care of the provisioning of a set of cloud VM instances in which it automatically sets up its own clustering engine for the distributed execution of the workflow. The VM instances only require the availability of a Docker runtime and the Java virtual machine. Workflow tasks exchange data via a shared file system or by using a cloud object storage service. The workflow execution is managed by Nextflow which also handles the cluster auto-scaling, i.e. adds or removes VM instances on-demand to adapt the actual needs of a workload at any point in time and optimizes the overall computing cost when applicable. At the time of writing, this feature supports the use of the AWS EC2 cloud service and Google Compute Engine service.

Figure 3 provides an overview of the components involved when executing a containerized Nextflow workflow.

Figure 3. Overview of a containerized workflow with a workflow engine managing the dependency graph and scheduling of the containers encompassing the tools to run.

The tools pass data between each other by reading and writing to a shared storage system.

Galaxy

Galaxy (Galaxy, RRID:SCR_006281) is a data-analysis workflow platform that has probably the most active development community in the field of workflow environments in Bioinformatics. This has enabled it to persist as an active project and to continue to evolve since 2005, enabling reproducible research through active channels for sharing tools, workflows and datasets.²³ Galaxy is currently used by tens of thousands of scientists across the globe.²⁴ Besides offering a modern user interface (UI) for end users, Galaxy is accessible through its REST API or through the BioBlend Python binding²⁵ (that talks to this REST API).

For execution versatility, Galaxy has adapters to offload jobs in a wide variety of systems, from local Docker containers to different batch scheduler systems. In Galaxy, there is a complete separation of concerns between tools, workflows and execution environments, meaning that a workflow definition is not bound to a particular execution environment. Within the last seven years Galaxy gained the ability to launch on AWS through the CloudMan Launcher Galaxy sub-project.²⁵ This initial setup followed a relative VM centric approach, which has natural limitations for scaling in the number of tools included as they all need to be provisioned inside the VM used for the provisioned cluster.

More recently, through efforts within the PhenoMeNal H2020 Project,²⁶ and later contributions by the Genomics Virtual Lab (GVL) initiative, Galaxy gained the ability to deploy inside Kubernetes in an automated fashion through the use of Helm Charts.²⁷ Since then, the Galaxy Helm chart has been adopted as a project by the Galaxy community (https://github.com/galaxyproject/galaxy-helm), and two major versions have been released, making v3.x of the Galaxy Helm Chart a mature product relying on a minimal Galaxy container for fast deployment. Galaxy Helm Chart v3 allows the deployment of multiple web and jobs handlers, making the setup scalable from any perspective (number of user and number of jobs). Newer versions of the CloudMan Galaxy launcher now rely on this setup instead of VM deployments and recently a Single Cell RNA-Seq Galaxy setup²⁸ was released based on the Galaxy Helm chart.

The Kubernetes job runner, also contributed to the Galaxy community by the PhenoMeNal H2020 Project and then further maintained by the Galaxy community, allows Galaxy to offload jobs to Kubernetes, either when Galaxy runs inside the container orchestrator or from the outside (provided that the shared file system is accessible to both the container orchestrator and Galaxy). The Kubernetes runner for Galaxy will take care of resource requirement settings, recovery of jobs in Kubernetes if there is a restart, auto-upgrading of resource utilization, Persistent Volume Claim mounting both for Galaxy and in each jobs pod, as well as adequate supplementary groups handling for network filesystem and other shared file systems. This integration requires the use of a shared file system, which needs to be mountable in Kubernetes in a read-write-many configuration, as both the main Galaxy job handler pod and the job pods need to mount it at the same time (through the PV/PVC abstraction). The Kubernetes runner for Galaxy adheres to the principles of resilience and redundancy commonly considered in cloud computing, providing ways to recover from jobs being lost by cluster nodes going down in the middle of an execution.

The Kubernetes runner for Galaxy can benefit from explicit tool-to-containers mapping through Galaxy dynamic destinations, or take advantage of dynamic bioconda-to-containers mapping thanks to built-in functionality in Galaxy for this purpose. This means that if a Galaxy tool declares a Bioconda package as the dependency to be resolved, it will bring an automatically built container for that Bioconda package²⁹ from BioContainers.¹³ For tools with multiple conda packages as dependencies, the Galaxy/Biocontainers community builds automatically multi-tool “mulled” containers (https://github.com/BioContainers/multi-package-containers) through Planemo-Monitor (https://github.com/galaxyproject/planemo), and these are as well resolved automatically when running on the Kubernetes setup.

The Galaxy-Kubernetes integration, which includes all the aspects related to Kubernetes mentioned in the previous paragraphs, has been battle tested by the PhenoMeNal H2020 Project, having its public instance receiving in the order of tens of thousands of jobs during the past 4 years, having tens of deployments in production environments and possibly hundreds of deployments in development settings. It has also been used as part of GVL’s CloudMan for the past year and for running training courses with more than 30 concurrent users in the context of Single Cell RNA-Seq. This integration has been tested locally on Minikube, on OpenStack deployments, on Google Cloud Platform and AWS turn-key Kubernetes services. In these two last cases, the setup can behave elastically, deploying more Kubernetes worker nodes automatically for increased capacity on demand, as the number of jobs or users increment, and then automatically reducing the number of nodes and containers deployed when load diminishes. It is expected that in the next few years the Galaxy-Kubernetes integration is embraced in the areas of proteomics and transcriptomics, and becomes one of the main routes of deploying Galaxy installations.

Pachyderm

Pachyderm³⁰ is a large-scale data processing tool built natively on top of Kubernetes. This platform offers: i) a data management system based on Git semantics and ii) a workflow system for creating distributed and reproducible pipelines based on application containers. In order to create workflows with Pachyderm, users supply a JSON pipeline specification including a Docker image, an entrypoint command to execute in the user containers and one or more data input(s). Afterward, Pachyderm will ensure that the corresponding pods are created in Kubernetes, and will share the input data across them and collect the corresponding outputs. Thanks to using Kubernetes for container orchestration, Pachyderm is able to, among other things: i) optimize cluster resource utilization, ii) run seamlessly on multiple cloud and on-premise environments, iii) self-heal pipeline jobs and related resources.

This workflow tool provides an easy mechanism to distribute computations over a collection of containers. In other words, it offers similar capabilities to frameworks such as Apache Spark, but replaces MapReduce syntax with legacy code. More specifically, Pachyderm is able to distribute workloads over collections of containers by partitioning the data into minimal data units of computation called “datums”. The contents of these datums are defined by glob patterns such as “/” or “/*”, which instruct Pachyderm how to process the input data: all together as a single datum, or in isolation (as separate datums). Users can define the number of pipeline workers to process these datums, which will be processed one at a time in isolation. When the datums have been successfully processed the Pachyderm daemon gathers the results corresponding to each datum, combines them and versions the complete output of the pipeline. The scheduling of pipeline workers is determined by the resource utilization status of the workers’ nodes and the resource requests that are made for each pipeline, allowing for efficiently distributing workloads. Recently, work to integrate and demonstrate Pachyderm in bioinformatics was published.³¹

Luigi

Luigi³² is an open-source Python module for defining and running batch-like workflows. The system is strongly oriented towards Big Data, and it comes as a convenience tool to pipe jobs from well-established analytics frameworks (e.g. Hadoop, Spark, Pig and Hive). Hence, instead of trying to substitute any of the available analytics systems, Luigi provides the means to stitch together jobs from different frameworks in a single batch-oriented workflow. In doing so, Luigi seamlessly handles many boilerplate operations such as: dependency resolution, visualization, failure tolerance and atomic file system operations.

The Kubernetes integration was developed by the PhenoMeNal consortium, and it is now part of the official Luigi project and thus being supported and maintained by the community. The integration enables to include pipeline processing steps as Kubernetes Jobs, which represent batch-like application containers that run until command completion. Hence, Luigi mainly uses Kubernetes as a cluster-resource manager where multiple parallel jobs can be run concurrently. To this extent, it is important to point out that Luigi is not meant to be used as a substitute for parallel processing frameworks (such as Spark and Hadoop), as it has limited support for concurrent tasks. In fact, in our experience Luigi can handle up to 300 parallel tasks before it starts to break down, or 64 parallel tasks if running on a single node. A project to adapt Luigi for scientific applications in general and bioinformatics in particular resulted in a tool named SciLuigi.³³

We benchmarked Luigi on Kubernetes by reproducing a large-scale metabolomics study.³⁴ The analysis was carried out on a cloud-based Kubernetes cluster, provisioned via KubeNow and hosted by EMBL-EBI, showing good scalability up to 40 concurrent jobs.³⁵ Workflow definition, and instructions to reproduce the experiment, are publicly available on GitHub.³⁶

SciPipe

SciPipe³⁷ (SciPipe, RRID:SCR_017086) is a workflow library based on Flow-based programming principles, which enables building workflows from component libraries of predefined workflow components. SciPipe is implemented as a programming library in Go, which enables using the full power of the Go programming language to define workflows, as well as to compile workflows to executable files, for maximum ease of deployment in cloud scenarios.

SciPipe has been used to orchestrate machine learning pipelines to build predictive models for off-target binding in pharmacology.³⁸ SciPipe currently provides experimental support for container based workloads via Kubernetes, implemented through the official Kubernetes Go client library.³⁹ The Go client library enables transparent access to a Kubernetes cluster regardless of whether the SciPipe workflow runs outside the cluster or inside, in a pod. When running inside the cluster, it automatically detects cluster settings. When not inside a cluster, the connection to a cluster can be configured by setting the appropriate environment variables. The Kubernetes integration in SciPipe is demonstrated through the implementation of a use case workflow consisting of a set of data preparation steps for mass-spectrometry data, using the OpenMS software suite.⁴⁰ It contains a Go file (with the .go extension) with the workflow, and a Kubernetes job-spec file in YAML format (with the .yml extension) for starting SciPipe inside a Kubernetes cluster is together with the workflow.⁴¹

Singularity containers⁹ can already today be used from SciPipe on a simple level, by configuring the appropriate shell commands when creating new processes through the shell-command process constructor (the scipipe.NewProc() function in SciPipe). Development of more integrated support for Singularity containers is planned.

Argo workflow

Argo workflow⁴² originates from a corporate application with the need of a competent continuous integration/continuous deployment solution but matured into a generalized workflow engine. It was open sourced by Intuit and is now adopted across several industries and companies among them life science and biopharma. A big differentiator of Argo is that no domain specific language knowledge is required to build workflows. The workflows are instead constructed by a clearly defined declarative approach manifested as Kubernetes custom resource definitions (CRDs), which is a templated approach to enable composability and extendability to the original Kubernetes resource orchestration. Argo is often referred to as a cloud native workflow orchestration tool and is designed for container native execution without the overhead of other non-native container solutions. Argo provides a robust way to declare and run workflows as arguments and artifact objects can be declared as input and output and containers run as state transitions merely passing objects and arguments through the pipeline. In each workflow step the state is stored in redundant etcd⁴³ memory for resilience and reproducibility. Argo is complemented by an artifacts concept that enables data and parameter versioning through complementing Argo with an S3 compatible storage backend. Furthermore Argo events enable signals and events for triggering workflows on webhooks, resource changes, GitHub or Docker image changes etc enabling additional automation scenarios.

Kubeflow

Kubeflow⁴⁴ is another open-source platform for Kubernetes, specifically aimed at building and running machine learning workflows and is developed by Google. As a framework, Kubeflow is intended to provide a set of tools for various tasks in the process of machine learning workflows, from data preparation and training to model serving and maintenance. Many of the tools are existing tools with a user base in the industry, such as Jupyter Notebook and Seldon, while others are customized wrappers or endpoints to frameworks like TensorFlow, PyTorch.

In particular, Kubeflow provides a workflow tool called Kubeflow Pipelines, which allows users to create workflows of varying complexity, chaining together series of container operations. The tool is based on the workflow engine Argo, which has been integrated with the Kubeflow ecosystem with a different UI than the original Argo UI. In essence, a user declares a workflow in the Kubeflow Pipelines software development kit, defining the different steps in terms of container operations, i.e. what container to use for the step, Kubernetes resource allocations, commands to execute in the container and so on. The user also declares the order of these container operations and any desired input parameters to the workflow, and from this definition a pipeline can be compiled. This pipeline is a portable configuration which in theory can be uploaded and used in any Kubeflow environment. With the pipeline deployed in Kubeflow, a user can start a run where available pipeline parameters can be provided and the progress of the run tracked in the Kubeflow Pipelines UI. Here, logs of the container’s standard output, output artifacts and various Kubernetes information can be tracked for each step, along with a visualization of the workflow graph.

As an evaluation of the Kubeflow Pipelines system, a simple pipeline was developed, based on previous work⁴⁵ where a convolutional neural network is trained to “… predict cell mechanisms of action in response to chemical perturbations...”, based on a dataset of cell microscopy images from the Broad Bioimage Benchmark Collection. The pipeline covers the following steps of the machine learning process: data pre-processing, model training, model evaluation and model serving preparation. The result is a workflow that handles the entire process end-to-end, building a servable machine learning model and publishing this as a Docker container that can be deployed for serving predictions. The pipeline source code, while partly tailored to a specific environment, is available in.⁴⁶