FTS3

Federated Data Infrastructure

Description

The software underlying a service responsible for globally distributing the majority of the LHC data across the WLCG infrastructure.

FTS3 is the software underlying a service responsible for globally distributing the majority of the LHC data across the WLCG infrastructure. It is a low-level data movement service, responsible for reliable bulk transfer of files from one site to another while allowing participating sites to control the network resource usage.

Transferring large volumes of data between facilities requires a component that manages those transfers: monitoring their progress, cancelling transfers that have stalled or that take too long, and retrying failed transfers (where appropriate).

Deploying FTS provides a common service for handling such transfers, allowing higher-level data management; e.g., bandwidth shaping links between facilities.

Target Audience

FTS is a core component that most users do not access directly. Instead, FTS is used by any DT user or manager who is (directly or otherwise) managing data locality.

FTS is used when a DT user or manager needs some data to be located at some facility and that data is not currently available.

User Documentation

- https://fts3-docs.web.cern.ch/fts3-docs/docs/cli.html

Technical Documentation

https://fts3-docs.web.cern.ch/fts3-docs/docs/developers.html

License

Apache 2.0

Source Code

https://gitlab.cern.ch/fts/fts3

Created by

CERN

Release Notes

This release represents the final release of the interTwin federated data management solution.

There are two external software components: FTS and Rucio. They are fully established projects, independent of the interTwin project. The software is production-ready, at TRL 9, and hardened with many years of production-critical use. Both projects have multiple deployments of their software, operated by different communities.

The ALISE software is currently in a development phase, under the aegis of interTwin. At the time of release, ALISE is TRL 4. The user-facing functionality of ALISE is mostly feature-complete; however, anticipated changes to the API imply that the necessary integration work (whereby a service uses ALISE to identify a user) should be considered experimental. Feedback from early adopters is encouraged, but any plans to deploy ALISE should be tempered by the anticipated changes to the API.

The teapot software has also been developed within the interTwin project. With this release, teapot is now TRL 6–7 and supports data transfer requirements of multiple, concurrent users. The per-user WebDAV instance management is automated, starting new services on demand, and terminating them if there is sufficient idle time.

Finally the first version of the Onedata S3 component is released, allowing integration of Onedata technology in the interTwin federated data management solution.

Future Plans

Some further improvements are planned for teapot. This includes integrating teapot with ALISE, to support automated identity management.

For ALISE, we anticipate possible improvements and stabilisation of the service-integration API, based on experience gained from integrating ALISE into various services. In addition, we plan to add support for client authentication in future versions of ALISE. This will limit access to the identity mapping information, providing this information to authorised services only.

As work continues with integrating the Datalake with various science use-cases, limitations may be found with the various components within the Datalake. Any such problems will be reported to the corresponding component’s development and support teams. The members of interTwin will offer effort to fix such issues, should such capacity become available during the project’s remaining lifetime.

Use Cases

interTwin Use Case: A Digital Twin for Lattice QCD simulation

Applying Machine Learning techniques to Lattice Simulation to speed up data generation and open up...

FTS3

Description

Release Notes

Future Plans

Use Cases

Related components