InterLink is an open-source service to enable transparent access to heterogeneous computing providers. It provides an abstraction for the execution of a Kubernetes pod on any remote resource capable of managing a Container execution lifecycle. The interLink component extends the Kubernetes Virtual Kubelet solution with a generic API layer for delegating pod execution on ANY remote backend. Kubernetes POD requests are digested through the API layer (e.g. deployed on an HPC edge) into batch job execution of a container.
The API layer foresees a plugin structure to accommodate any possible backend integration.
Executing payloads in response to an external trigger like a storage event or a web server call.
Frameworks for DAG workflow managements are usually well integrated with Kubernetes APIs
Users can exploit interLink via self-provisioned deployment (i.e. through already integrated high level services) or standalone Kubernetes deployment creating and deploying a simple container that can be scheduled on a remote system such as a Slurm batch on an HPC center. High Level services getting integrated are e.g.: Airflow/Kubeflow pipelines/Argo workflows/MLFlow, Jupyter notebooks
Release Notes
interLink is an open-source software product under development in the context of interTwin project. Its development started in order to provide an open-source solution capable of extending the container orchestration de-facto standard (kubernetes) to support offloading to any type of resource provider (Cloud/HTC/HPC) in a transparent manner, where little to no knowledge is required for the end user. The key objective of interLink is to enable a Kubernetes cluster to send containers/pod to a “virtual” node. This node seamlessly manages the entire lifecycle of the user’s applications, whether on a remote server or, preferably, within an HPC batch queue.
Transparency is granted by the fact that it keeps exposing the very same experience of running a pod on cloud resources, thanks to the API layer that exposes the regular set of Kubernetes APIs.
The interLink project is organised in 3 major subcomponents. Namely:
- A Virtual Kubelet
- The InterLink API
- Sidecars
Future Plans
In terms of features and capabilities, the next steps and priorities are further integrating the support for the Datalake and data management interoperability. The plan also foresees the offloading integration with INDIGO-PaaS intelligent orchestration.
Regarding the functionalities of the current software more in general we expect to focus more on logging capabilities in order to allow easier debugging in case of problems. Also, this information can be used for high-level reporting to users in case of problems
Additional improvements already planned based on early feedback include enhancements to volume management, Apptainer/Singularity integration. These updates aim to address site resources access policies and improve the management of File System permissions on remote resources.
