Built on more than a decade of experience, Rucio serves the data needs of modern scientific experiments. Large amounts of data, countless numbers of files, heterogeneous storage systems, globally distributed data centres, monitoring, and analytics. All coming together in a modular solution to fit your needs. Rucio provides a service that manages data locality. It provides a scalable solution for managing the dynamic locality of files in a heterogeneous, federated storage Datalake.
When deployed, the Rucio software provides a service that allows a group of researchers to manage non-trivial amounts of data. For any given file, dataset, collection of files, or container (a collection of files and datasets), it provides information on where that data is currently available. It also supports dynamic, time-limited data placement, with data being made available for some period (e.g., to support some computational workflow). It optimises the use of available storage by operating a cache, assuming that data that was previously used is more likely to be needed in the future. Desired data locality is expressed in terms of declarative rules. These rules may be applied both to existing datasets and anticipated, future data.
Release Notes
This release represents a technical preview of the interTwin federated data management solution.
There are two external software components: FTS and Rucio. They are fully established projects, independent of the interTwin project. The software is production-ready, at TRL 9, and hardened with many years of production-critical use. Both projects have multiple deployments of their software, operated by different communities.
The ALISE software is currently in a development phase, under the aegis of interTwin. At the time of release, ALISE is TRL 4. The user-facing functionality of ALISE is mostly feature-complete; however, anticipated changes to the API imply that the necessary integration work (whereby a service uses ALISE to identify a user) should be considered experimental. Feedback from early adopters is encouraged, but any plans to deploy ALISE should be tempered by the anticipated changes to the API.
The teapot software is also currently in a development phase, within the interTwin project. Teapot is TRL 3–4. The current release is sufficient to build a limited proof-of-concept demonstration, supporting the data transfer requirements of a single user.
Future Plans
A number of improvements are planned for teapot. Teapot will be updated to support multiple, concurrent users. The per-user WebDAV instance management will be automated, starting new services on demand, and terminating them if there is sufficient idle time. It will also be updated to integrate with ALISE, to support automated identity management.
For ALISE, we anticipate possible improvements and stabilisation of the service-integration API, based on experience gained from integrating ALISE into various services. In addition, we plan to add support for client authentication in future versions of ALISE. This will limit access to the identity mapping information, providing this information to authorised services only.
There are currently no immediate plans to improve FTS. Future experience from integrating the testbed with the interTwin service providers may identify necessary enhancements to FTS.
Plan to enhance Rucio by improving support for integrating Rucio with external catalogues; e.g., Lattice QCD’s catalogue. Other plans to enhance Rucio may emerge from our experience when integrating Rucio with DTE core service, DTE thematic modules, and with the science use cases.
