OSCAR has been adopted by the interTwin project to implement a generic framework for real-time data acquisition and processing that builds on event-triggered execution of workflows, to i) improve response times and minimise data transfers, and to ii) support new event sources.
For this release, Apache NiFi, an open-source system that supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic, has been firmly integrated with OSCAR. The following aspects outline the recent updates and current implementations:
Data Lake integration:
- Using Apache NiFi, Amazon S3 + Amazon SQS is employed; its configuration is managed via DCNiOS (see DCNiOS).
- Using Apache NiFi, dCache is supported as a source of events; its configuration is managed via DCNiOS.
Real-time data acquisition:
- Using Apache NiFi, Kafka is supported as a source of events; its configuration is managed via DCNiOS.
Integration with other DTE components:
- interLink: The integration of the OSCAR component with the interLink component to offload computation to HPC has already been completed.
- itwinai, an AI pipeline management tool, has been integrated into OSCAR.
- JupyterHub has been integrated employing MinIO as a storage backend, seamlessly connecting the storage of both systems.
Use case integration:
- The integration of CERN’s 3DGAN inference for particle simulation in the LH-LHC was successfully completed, demonstrating the use of itwinai, dCache, and interLink.
- The integration of DT-Flood (a digital twin for flood detection and flood impact estimation) by Deltares and TU Wien is about to be completed, enabling the offloading of computationally intensive steps in their CWL workflow to OSCAR.
- The integration of EURAC’s Drought Forecasting use case has been achieved by enabling the offloading of the computationally intensive steps in their workflow within OSCAR.
Software quality:
- The integration with the SQAaaS platform enables OSCAR to automatically perform quality assessments for each release, facilitated by a GitHub action on every pull request.
