The Flux Operator

2.1 Implementation

This section details the architecture of the Flux Operator, first describing the design and needs of Flux, and how those are satisfied in Kubernetes. From the standpoint of a software architect, the task of designing the Flux Operator could be approached as a problem of matching software design patterns. Knowing that Kubernetes provides a set of components and application programming interfaces, a key challenge was to assemble the components in a way that would deploy the full Flux Framework stack running inside of Kubernetes. An ideal design might aim to achieve the keystone properties of Kubernetes applications, including but not limited to fault tolerance, load balancing, and elasticity. Abstractions for storage, networking, and workloads could be selected for this design, and with a mindset of portability, meaning that the software components would be in containers themselves. The following sections refer to two roles – an operator developer, or someone that designs and implements the Flux Operator itself, and an operator user, an individual that installs the operator and uses it. These architecture sections start with an overview of Kubernetes Operators, and then describe each component of Flux, and a mapping from traditional bare metal solutions to abstractions in Kubernetes.

2.1.1 Kubernetes operators

While individual components such as pods or services could be individually implemented and created in the Kubernetes cluster, the advent of programmatic operators in 2016 has hugely simplified this process for the developer user. A Kubernetes operator serves as a controller for one or more Kubernetes objects, meaning that a developer can express all of the custom logic needed for an application of interest in code that is compiled and run in Kubernetes. The operator implements a custom resource whose behavior is dictated by a custom resource definition (CRD), a YAML file with a specification of variables for the controller to use. For the Flux Operator, this custom resource is called a “MiniCluster”. The basic design of a controller is a loop, running a reconciliation process until a cluster reaches a desired state requested by a user via this custom resource definition YAML specification. This is a declarative model in that the operator user can specify high level constructs such as the cluster size and application to run, and they don’t need to know the details of setting up a Flux cluster, nor do they need to consider orchestration or update of components. This approach is advantageous in that it exposes only the amount of detail that is needed for some number of jobs, and the complexity that would require niche expertise is hidden.

2.1.2 Flux Framework

The Flux Framework is a novel, graph-based resource manager that is typically deployed on-premises at HPC centers.¹⁶ It won an R&D 100 award in 2021,²¹ and is slated to be the primary scheduler for the upcoming exascale-class system “El Capitan” at Lawrence Livermore National Laboratory. It is called a framework because several projects combine together to form the resource manager known as Flux. A summary of these projects is described in Table 1, and the interested reader is directed to the learning guide for an in-depth overview.

While Flux can be described in terms of its modules or components, for the work here it will described as it is seen in the space of Kubernetes abstractions.

2.1.3 A Flux MiniCluster

The node Flux is typically deployed across nodes on an HPC cluster, where each node can be thought of as an addressable compute or storage unit, and as having a unique network address. Moving into the space of cloud, the physicality of the server goes away, and instead the basis of a node is a virtual machine. However, while Kubernetes itself is deployed on nodes, a fundamental object is the pod – an abstract slice of a node that is tasked with some unit of work to run one or more containers, and allocated a particular set of resources. Since the Kubernetes scheduler is designed to assign more than one pod to a single node, the first task in defining this cluster was to ensure that there was a mapping of one pod per actual physical node. The reason for this mapping is due to Flux not being able to detect running on a partial node, which is a result of its use of the portable hardware locality (hwloc) library to discover resources. Pod isolation in Kubernetes is based on a kernel technology, cgroups, which allows for resource management on the level of the kernel. However, this abstraction does not take hardware into account, and so regardless of a resource specification, hwloc will detect the resources of the entire node. In the context of two pods running on one node, despite any resource management done via cgroups, the library would erroneously discover the resources of the entire node twice. This means that Flux would detect double the resources that are actually available, and could schedule too much work on a single physical node that, to the resource manager, is seen as two separate nodes with identical resources. The 1:1 mapping of pods to nodes was originally achieved by way of a resource specification, a strategy that required the user to ask for just under the upper limit of CPU and memory offered by their cloud instance of choice. This strategy was later improved to not require knowledge of the instances by way of rules for pod affinity and anti-affinity. These are essentially constraints that tells the scheduler to ensure one pod per node, each with a hostname for Flux.

The cluster While it would be possible to deploy individual pods onto nodes, early Kubernetes offered further abstractions for sets of pods such as deployments and Stateful or Replica sets, and in 2015, an abstraction called a Job that was the first of its kind to approximate a traditional HPC job with the intention to run and complete. As of 2021, the batch working group introduced the indexed mode addition to Job where the same work could be done in parallel, expecting 1 to N completions. In that each node of a simple Flux cluster would be identical aside from subtle differences in startup, although other abstractions were considered, an indexed Job was ultimately chosen. The indexed Job is ideal in that it inherits needed features from the base Job such as having states, and adds an ability to create duplicates of the pods. To create pods it uses a batched approach, which is also advantageous to introduce an indexed ordering that ensures the bigger numbers are downscaled first. This allowed us to easily design the operator to use the index 0 pod as the lead broker, and any scaling up or down of the cluster (Section 4.1) would never risk deleting the lead broker. Within this cluster context, given the assignment of one pod to one node, for the remainder of this paper the terms “node" and “pod" are used interchangeably as they are mapped to the same resources, memory and CPU.

Networking One of the foundational components of Flux Framework is the scalable tree-based overlay network, or “TBON” that connects the core modules for scheduling and resource management. The Flux TBON is a rooted tree that features leader and follower processes (brokers), each of which is typically mapped to one node. The leader expects follower brokers to connect to it. Mapping this design to the indexed Job, the role of lead broker can be assigned to index 0, and the follower brokers to indices 1 through N-1. Along with being easy to remember, this design decision allows pods to be created in order of their index with the lowest first, which is ideal to have the lead broker up earlier for the follower brokers to find. ZeroMQ²² is used for the discovery of the node and to bootstrap the cluster. In this process, the follower brokers identify their place in the cluster by way of their rank in a shared system configuration file, and then connect to the lead broker on a specific port via transmission control protocol (tcp). If the lead broker is not up first the worker will wait to try connecting again, and by default the ZeroMQ library falls back to a tcp default retry timeout that increases exponentially. In practice this means delayed cluster startup times waiting for the follower brokers to retry. The scheduler and resource manager combined with this set of brokers that can communicate to run jobs is called a Flux instance. A Flux instance can be on the system level, meaning shared by many users, or owned by a single user. In both cases, the Flux instances handle user requests for submitting or otherwise interacting with jobs. The instance itself is hierarchical because it can spawn sub-instances whose resources are a subgraph of their parent.

The networking setup can give each pod a unique address that can be written into the Flux system configuration, and used to identify lead and follower brokers. For the first naive implementation, the Flux Operator created the pods, retrieved the IP addresses after creation from the Kubernetes API, and then added corresponding entries to the /etc/hosts file for domain name system (DNS) resolution. Automated management of the hosts file proved to be a bad design because it required restarting all the pods to recreate a configuration file with known hostnames. Instead, a later design created a Headless Service for the MiniCluster, meaning that each pod could be given a label, a key value pair, that was known to the Headless Service, and discovering the labeled pod would add it to the network provided by the service. The Job controller then assigns a predictible hostname, and the Headless Service makes the name discoverable to DNS. This series of steps is essential for Flux to identify each broker, and is supported by way of unique pod names that are published as A records. This simplified the creation of the cluster, and allowed for networking readiness as soon as the service object was ready. Once this is done and the brokers have connected over TCP, further communication for the overlay network is done via ZeroMQ sockets.¹⁶ However, for the cases of workflows that use a message passing interface (MPI), Flux has built-in modules for MPI and communication, meaning that Flux simply uses standard MPI libraries that can rely on sophisticated networking fabrics or other high speed interconnects. This reliance on cloud hardware has proven to be a challenge when deploying the Flux Operator to different cloud providers, and is a focus of collaborative work.

Volume mounts In terms of configuration, Flux requires a system configuration file, a ZeroMQ CURVE certificate used to encrypt communication, and a means to start the brokers. In a traditional HPC setup, this means could be using a systemd service, however in a Kubernetes environment with containers, it means a start command for the container with conditional logic for the lead vs. follower brokers. In Kubernetes, all of the above configuration can be achieved via volume mounts provided via ConfigMap objects. By way of mounting configuration files and other needed files to each pod container as read-only volumes, all nodes in the cluster have access to them. These are mounted at /etc/flux and /flux_operator for configurations and the starting script, respectively, and the choice of a root path affords discoverability.

The curve certificate presented a bootstrapping design problem, as the standard way to generate it is via Flux itself (ZeroMQ is compiled within and exposed via the flux keygen command). However, this content was also required to exist for the read-only volume before starting the pod container. For the earliest design of the Flux Operator, a one-off certificate generator container was brought up that ran this key generation command, and the key was printed to the log to be retrieved by the operator. It could then be straightforward to write into the ConfigMap to be shared by the MiniCluster pods. In a later design, by way of collaboration with authors of this paper following Kubecon Amsterdam’23²³ this bootstrapping problem was further improved by compiling ZeroMQ directly into the Flux Operator, and using cgo to interact with it directly to generate the certificate content for the ConfigMap inside the operator. This removed an entire step to generate the one-off pod and reduced startup latency, and is a beautiful example of how sharing ideas and collaboration can lead to improvements in design and functionality. The open source nature of these projects is key for enabling collaboration that can result in faster innovation.^24,25

A Flux container image The libraries and software needed for Flux must be built into a common Flux container that is replicated by the indexed Job. This container would need to support running configuration steps for Flux upon start, and additionally include any application of interest to be run by the user. This is a design flaw in that most containerized applications for HPC have not been built with Flux, and would need to be built again to install not only Flux, but required shared libraries. While the HPC community is attuned to building and optimizing components for different architectures, ideally a more modular, cloud-native solution would not require such a substantial time investment. Approaches need to be investigated that can achieve separation of application logic from Flux while not impacting performance, and further work is needed to investigate this. Once required software and configuration files are present, setup continues to create either a single-user or site-wide installation of Flux. The Flux Operator opts for a single-user design, and enables customization via variables exposed on the CRD. This customization includes (but is not limited to) archiving of data, creating multiple users, starting in an interactive mode, starting a RESTful application programming interface, or creating a custom batch job. The final component of the container is the “entrypoint” or the command that is run when the container is started. This varies between the lead and follower brokers, where the lead broker typically starts with a command to run a job, and the follower broker starts expecting to connect and receive work.

2.2 Operation

The first requirement for using the Flux Operator is access to a Kubernetes cluster with sufficient permission to deploy each of the abstractions discussed in Section 2, including Deployment, Service, Job, and ConfigMap. The Flux Operator pre-built container images are provided for nodes with amd64 or arm64 architectures. Operation comes down to deploying these objects with the kubectl command line tool and then using the same tool to deploy the MiniCluster CRD. Upon creation, the operator reconciles in a loop until the state of objects in Kubernetes matches the desired state specified in the CRD. This desired state encompasses the creation of the MiniCluster, which includes an indexed Job with a group of pods for the cluster, each containing a flux broker and volumes for configuration. A Headless Service networks the pods together to complete the MiniCluster. The specific attributes of the cluster (e.g., size, application image, command) can be customized in this YAML file, resulting in a Flux MiniCluster that exists to run a command and clean up (akin to a batch job), or an interactive cluster that can address several use cases and interactions discussed in Section 4. The final Flux MiniCluster is illustrated in Figure 1.

Figure 1. The Flux Operator design and creation process for a Flux MiniCluster.

The custom resource definition “CRD” that describes the specification for the cluster is provided by the user as a YAML file. Upon creation, the operator reconciles in a loop until the state of objects in Kubernetes matches the desired state specified in the CRD. This desired state encompasses the creation of the MiniCluster, which includes an indexed Job with a group of pods for the cluster, each containing a Flux broker and volumes for configuration. A Headless Service (purple) networks the pods together to complete the MiniCluster.

3.1 Experiment

Experiments were conducted on Amazon Web Services Elastic Kubernetes Engine (EKS), using the hpc6a.48xlarge instance type hpc6a with the elastic fiber adapter (EFA) intending to test the Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS)²⁷ in a strong-scaling configuration across cluster sizes and ranks of 64/6016, 32/3008, 16/1504, and 8/752, respectively (Figure 2). This design was chosen to mirror previous work.¹⁸ LAMMPS is used by the Collaboration of Oak Ridge, Argonne, and Livermore (CORAL) as a representative scalable science benchmark as part of CORAL-2. In testing, LAMMPS runs in parallel on MPI ranks (processes) across nodes, a molecular simulation²⁸ and the problem size 64×16×16 was chosen that would adequately test strong scalability across the chosen rank and node counts. LAMMPS scalability depends on network latency, and the experiment results report the total wall time recorded by the LAMMPS run as a metric for performance. A cluster setup that enables lower latency will run faster, and ideally the simulation should get faster as the number of nodes is increased with strong scaling. A second metric time of interest is the time for the launcher to submit and complete a job. For Flux this means timing the flux submit command that is given the command to run LAMMPS, and for the MPI Operator it means timing the mpirun command that does the same. The final metric time of interest was the total cluster creation and deletion times, which can be calculated based on the total runtime of the MiniCluster minus the LAMMPS total wall time. This time would include each pod preparing the broker, starting Flux, and networking with the lead broker. The runtime would ideally decrease across these chosen rank and node sizes.

Figure 2. Experiment design for testing the Flux Operator against the MPI Operator.

A single Kubernetes cluster of size 65 was created (blue outline) to test subsequently smaller cluster sizes, including 64 pods (6016 ranks), 32 pods (3008 ranks), 16 pods (1504 ranks), and 8 pods (752 ranks). An extra node (blue) is needed for the MPI Operator launcher to supplement the nodes doing work (green).

To ensure the nodes of the cluster are consistent and do not influence results, experiments were run on the same Kubernetes cluster, and simply used smaller portions of it. As the MPI Operator requires an extra launcher node, the maximum cluster size needed (64) was increased by 1, resulting in a size 65 node cluster for these experiments. Finally, a modified version of the MPI Operator was used¹⁴ to allow it to scale to over 100 MPI ranks.

The experiments proceeded as follows. The main Kubernetes cluster of size 65 is first created. Then, for each of the Flux Operator and MPI Operator:

For each experiment run, a single “throwaway” run is first performed to pull the container with Flux and LAMMPS to the node, where it is cached for further runs. This ensures that time recorded in creating the MiniCluster does not include pulling the container image, which would have variability depending on the image size. The experiments are then run in an automated fashion using Flux Cloud, a simple experiment orchestration tool for running experiments with Flux MiniClusters on Kubernetes. All experiment code, configuration files, and tagged containers are available.^29,30

3.2 Figures of merit

Cluster creation and deletion We created and deleted clusters of 8, 16, 32, and 64 nodes in 20 repetitions, observing a small increase in median times (Figure 3) indicating that the indexed job can efficiently create pods. All sizes were created and ready in under a minute with an interquartile range for any given size between approximately 1 and 5 seconds, and a maximum difference of approximately 10 seconds between the fastest time of the smallest cluster and the slowest time of the largest cluster. Likely this variability reflects the slowest or last node to come up, as the cluster is not considered to be completely up until all nodes are ready. Due to the design of the MPI Operator, there was not a corresponding measure of creation time appropriate for comparison, so the two are not comparable side by side.

Figure 3. Combined Flux MiniCluster creation and deletion times across MPI ranks, corresponding to cluster sizes of 8, 16, 32, and 64 nodes, respectively.

Times are calculated as the total time from creation to deletion, and subtraction of the time it took to run and job within. One outlier for size 6016 ranks stands out – one instance of the largest cluster completing creation, run, and deletion in a time closer to 50 seconds. These experiments were run on Amazon Web Services with the hpc6a.48xlarge instance type and the elastic fiber adapter for networking.

LAMMPS total wall time A primary time of interest for running LAMMPS is the total wall time, which is reported by the LAMMPS software itself. Consistently better performance of the Flux Operator over the MPI Operator was observed, with median times that are approximately 5% faster, meaning that the Flux Operator software completed the same workload more efficiently (Figure 4). While the differences in means for this experiment are small, this improvement would likely be more prominent for longer experiments, and could lead to reduction in total costs. Identifying the underlying reasons for the improved performance is another task suitable for investigation with performance tools in future work.

Figure 4. Total wall time, as reported by the LAMMPS software, between the Flux Operator and MPI Operator.

The Flux Operator is consistently faster, a result that could be more impactful for longer running experiments.

Launcher times Comparing launchers (flux submit for the Flux Operator, and mpirun for the MPI Operator) there is a slightly larger time difference (Figure 5), where both generally perform well under strong scaling (the time goes down). What is unknown is whether there is an inflection point at larger scales where the MPI Operator might plateau or otherwise show a different pattern. The resources were not available to run these larger experiments at the time, but these patterns and scaling can be explored in future work.

Figure 5. Launch time measuring flux submit (the Flux Operator) or mpirun (the MPI Operator).

The Flux Operator is consistently faster, a result that could be more impactful for longer running experiments.

Design considerations Anticipating interest in running experiments of this type, where there is generally some operator in Kubernetes that is going to pull one or more containers to Kubernetes and perform scoped work, we provide a visualization of the steps that have salient times. In Figure 6, there is a distinction between an operator setup that is using autoscaling (right) vs. not (left), and costs that are incurred once (blue) vs. repeated (green). Cluster creation generally means the start and setup of instances, along with any networking and devices that are needed. The primary difference between the two scenarios is that an autoscaling cluster is going to be adding new nodes, meaning that the cluster will need to provision those nodes, and the user will be required to wait. Thus, this update process for the cluster that requires the user to wait (and pay for the time) becomes a repeated cost. This also means that a typically one-time cost of pulling a container may occur several times, primarily when new nodes are added. Note that this diagram assumes experiments running on one cluster. An autoscaling setup that employs bursting to new clusters would need to consider the additional time of creating and deleting the new clusters.

Figure 6. Times to consider when performing experiments with a Kubernetes operator, from the point of creating the cluster to deleting it.

For a single cluster without autoscaling (left) many of the operations are one-time costs (blue), while for an autoscaling cluster that needs to provision new nodes and pull containers to them, the costs become repeated (green). An efficient approach might maximally use provisioned resources while waiting.

It is suggested to the reader to consider these times, along with the differences between a setup with autoscaling versus one without or potentially bursting, for future experiments when anticipating costs. Notably, the setup time for any particular operator could be generally consistent, and variance has been seen (but is not reported here) in the other steps between cloud providers. A more critical study and understanding of these times is warranted for future work.

4.1 Saving state

The goal of experiments to save state would involve starting a Flux MiniCluster, running some number of jobs, pausing them, saving the state of the job queue, and then bringing it down to bring up a different sized cluster to load the jobs into, where they would continue running. This concept of saving state is similar to forensic container checkpointing in Kubernetes and checkpoint/restart (e.g., Ref. 31) in HPC, and would be useful for pausing workflows for cost savings or waiting on resource availability. These experiments varied based on when the queue was paused. In the earliest tests, job completion was required before saving state, while for later tests, jobs were stopped mid-run.

In practice saving state meant waiting for the queue, pausing, and then saving to an archive in a volume shared between two MiniClusters. Initially, the Operator waited for the first MiniCluster pods to terminate and the new MiniCluster to come up before restoring the jobs. We observed that jobs would successfully save and load into the new cluster, maintaining job identifiers and size, however when stopping a running queue, 1-2 jobs could be lost between the transfer. While the reason for this loss would be interesting to understand, as it is an experimental prototype for a feature, the work is beyond the scope of this paper, and akin to other features discussed here, should be pursued with a compelling research use case. When this time comes, more analysis would be needed to understand exactly what is going on. The majority of jobs (e.g., roughly of 9 out of 10) transition successfully, meaning that a job on the previous queue can get scheduled to a new larger or smaller cluster. As would be expected, if a job is moved onto a cluster lacking enough resources, it would logically not be scheduleable. A write-up and tutorial to reproduce this work is available.

While this early use case of saving state was simple, it was a glimpse into the idea that scheduled workflows could in fact be moved. In changing the size of the resources available by way of creating a new MiniCluster, it was the earliest prototype for what might be called scaling or elasticity, discussed next.

4.2 Elasticity

Elasticity can be thought of as automated dynamic scaling.³² Instead of making a cluster larger or smaller by way of saving state and loading into a different size, true elasticity means changing the size of a single cluster, which in the context of the Flux Operator means that Flux must adapt dynamically. To enable MiniCluster elasticity we added limited support for resource dynamism within Flux. Our method is analogous to resource dynamism in Slurm in that it relies on configuring predefined resources. Support for resource dynamism by integrating arbitrary resources on demand is future work. The following steps enable an elastic Flux MiniCluster:

The above also carries the constraints that the cluster cannot be smaller than one node (only a lead broker) or larger than the initial maxSize. The larger indices are terminated first and the operator does not allow reduction to size zero, so the lead broker is never at risk of deletion – such a request would delete the entire MiniCluster that relies on it. The reason this works is that Flux sees the initial set of pods that do not exist as simply being down, which happens frequently in a high performance computing environment. When the nodes are created their corresponding follower brokers start, ping the lead broker on the port to connect (typically port 8050) and then they seamlessly join the cluster. From the standpoint of the user, they change the “size” value in their MiniCluster CRD, apply it, and then see their cluster grow or shrink. The Flux instance run by the lead broker simply sees a node come online. On the Kubernetes side, this ability for the indexed Job to have elasticity requires a minimum Kubernetes version of 1.27.

At this point, it needed to be decided what might trigger this change in size. Elasticity was implemented in two ways, first with an application-driven approach that required extra permissions to be given to the in-cluster service account, allowing the application inside the cluster to ask for more or fewer pods directly. It was then discovered that Kubernetes has autoscaling APIs intended for this use case. This autoscaling approach is discussed in the next section.

4.3 Autoscaling

In Kubernetes there are two types of scaling – horizontal and vertical. Horizontal typically refers to adding pods, while vertical refers to adding resources to existing pods. Both are based on the idea that resources should change in response to changing workload needs – if cluster or resources are too big, they are made smaller, and vice versa. In the case of the Flux Operator the primary interest was horizontal autoscaling, or changing the number of pods to dynamically increase or decrease the size of the MiniCluster to respond to the demands of a workload. This led to an initial implementation based on horizontal pod autoscaling (HPA) using the HorizontalPodAutoscaler API resource, a cluster API to watch a selected set of pods for resource consumption and increase or decrease the number depending on utilization. For the simplest cases, a default autoscaler was first deployed that considers a metric such as percent CPU usage and uses an algorithm to calculate a target scale value for the number of pods. This could be tested by running a CPU intensive simulation to observe the autoscaler adding and removing pods. However, the approach was not fine-tuned enough to the potential needs of an application being run by Flux. Instead of an arbitrary decision to add or remove pods based on CPU, a design more specific to Flux is warranted. As an example, one design might be that the Flux lead broker makes decisions about when and how to scale depending on the content of the queue. Another valid design would be to allow for changing the size of a single running job. Both of these ideas, and more generally designs for autoscaling, are valid and prime for future work.

With this in mind, we implemented a custom metrics API, meaning implementing an equivalent API endpoint controller that would be called by an autoscaler with instructions for how to scale the cluster. This resulted in the Flux metrics API, a standalone API that runs directly from the lead broker pod and provides decisions about scaling up or down based on the size or other metrics about the queue. With this API, it was possible to demonstrate an autoscaling operation running based on a trigger coming directly from Flux. More work will be needed to test this setup with real workflows. In the meantime, more details about this setup and basic elasticity are available in an external post.

One notable feature about the implementation of the autoscaling approaches described above is that regardless of whether the request comes directly from a user changing a value in a file or an application or a programmatic autoscaler, the same internal logic (functions) is used to validate and then perform the patch.

4.4 Multi-tenancy

Multi-tenancy refers to the ability to support multiple users on the same resources. In Kubernetes, ownership of resources is typically designated by namespaces, service accounts, custom permissions on connected resources like storage, and role based access controls (RBAC). Recognizing these challenges, as an early approach there are several modes of interaction:

In anticipation of the last two cases that implement multi-tenancy, a RESTful application programming interface (API) was designed that runs from the lead Flux broker pod, and thus exposes interactions with Flux to submit, get info, cancel, and otherwise interact with jobs via Flux Python bindings exposed by the API. This is made possible by exposing the internal port that the API is running on via an external NodePort and forwarding a port to an external client for interaction.

In all cases of requiring authentication, the Flux RESTful API uses an OAuth2 based approach, storing a database of user identifiers and encoded passwords, and first authenticates by using a base64 encoded username and password (typical of a basic authentication scheme), and then provides the user with an expiring token that can be used for further interaction. In the case of a single Flux user behind a multi-tenant API, the authentication and authorization occurs, allowing all users to submit jobs to a shared queue. In the case of true multi-tenancy with PAM, the custom resource definition asks for usernames (and optionally, passwords) in advance, and then creates the accounts on the system that are checked after authentication. The installation of flux-accounting can then be enabled for the lead broker’s queue, and use a traditional fair-share algorithm to determine job priority. This work can be extended with more cloud-native approaches that take advantage of namespaces and roles, such as is described later (Section 4.6).

4.5 Bursting

To complete the early work in autoscaling, we considered the concept of bursting,³³ which means not just extending the size of a local cluster, but actually extending work to external resources. The bursting work for Flux would extend this approach to not just deploy external resources, but allow the lead broker to connect to brokers that are deployed in the other clusters. As an example, a Kubernetes cluster running on Google Cloud might burst to a cluster running on Compute Engine (CE), or to a cluster on Amazon Elastic Kubernetes Service (EKS).

To implement a prototype for bursting, we chose a simple design first. A plugin service would be running from the lead broker of a primary cluster, and the running user would load one or more bursting plugins into it. Each bursting plugin is targeted to a particular provider (e.g., EC2 or CE). While there are many ways to trigger a burst, a simple approach of looking for an attribute “burstable” on a job set to true was chosen first. This request could be done on the command line. Upon discovery of this attribute, the bursting service attempts to schedule the job with the plugin. Each plugin is free to decide if the request is satisfiable by its own custom terms. If the burst is satisfiable, the job is assigned to the bursting plugin, and the plugin creates a new cluster or assign the job to an existing cluster. In the case of creation, the technique of telling the primary cluster that there are more nodes that are expected (and start down) than there actually are is used, and assign them namespaced hostnames that will correspond to the bursted cluster. The calls that are necessary to bring up the second cluster are run, which might mean deploying Terraform configuration files or creating a second Kubernetes cluster via API calls, and then the cluster starts just as a local MiniCluster would. The key difference, however, is that the lead broker of the primary cluster is exposed as a NodePort service that can be discovered by the external cluster. The secondary brokers, all followers, then come up, find their hostnames in the ranked system configuration, and connect to the lead broker IP address from another cluster. To the user, they simply see that the nodes are down, and then they come up when the cluster is ready. Jobs that are scheduled on the primary broker queue that possibly could not run due to insufficient resources can then run. At the time of this writing, the main bursting service is implemented along with four bursting plugins for each of GKE, EKS, CE, and a local burst.

Finally, the bursting service is designed to be modular and flexible. Aside from being able to load different plugins, it allows for customization of the function provided to select a burstable plugin, to interact with the queue, and to select jobs. A mock version of a Flux job is also available for development. The work in bursting is still early, and akin to elasticity, work on these prototypes should continue to eventually develop more hardened Flux modules and algorithms for bursting.

4.6 Workflow integration

Running a single MiniCluster to create an isolated Flux cluster in Kubernetes is an initial step, but it is insufficient for real-world use cases of complex workflows. While it would be possible to shell into or otherwise interact with the cluster and run a workflow tool that implements Flux as an executor,^34–36 this also does not enable features needed for complex, heterogeneous workflows that might require different sizes or configurations of MiniClusters. We are considering integration of the MiniCluster custom resource definition as a first-class citizen into workflow tools in future work.

After the Kubecon Amsterdam’23 presentation, collaborators (including author AC) were quickly motivated to add the Flux Operator as a job type to the workflow tool Kueue, a Kubernetes-native job submission operator that handles managing multi-tenancy of Kubernetes batch jobs that can be produced by a number of operators. We are developing a similar approach to control the creation and management of a MiniCluster through workflow tools, an idea being implemented into the Snakemake³⁴ workflow tool as a Kueue executor plugin. Defining even an example workflow for high performance computing is a non-trivial problem, as many codes are either private or challenging to port. This initial work with the Flux Operator sets the stage for studies of integrations of this type. Tackling this early problem is a two-fold challenge to design technologies and inspire more collaborative opportunities for the HPC community.