An open-source command-line tool to easily manage the creation of event-driven data processing flows. When files are uploaded to data storage, events are ingested by Apache NiFi, which can queue them up depending on the (modifiable at runtime) ingestion rate. Then, they are delegated for processing into a scalable OSCAR cluster, where a user-defined application, based on a Docker image, can process the data file.
Updated 13/02/2024
Core DTE Modules
DCNiOS
Description
A new open-source command-line tool to easily manage the creation of event-driven data processing flows.
Target Audience
+
- Technology Integrators
Documentation
+
License
+
Apache 2.0
Source Code
+
Created by
+
Release Notes
DCNiOS, initially developed within the interTwin project to integrate dCache with OSCAR, has significantly evolved to support a myriad of data sources. This component now includes implementations for supporting Amazon S3 along with Amazon SQS and Kafka as source of events, as well as any generic element that can be created as an Apache NiFi ProcessGroup. Additionally, DCNiOS supports alterations that modify the received input data during the execution of the data flow.
DCNiOS uses the SQAaaS platform to perform quality assessments for each release, ensuring thorough evaluation and reliability.
Future Plans
DCNiOS will continue to adapt to emerging requirements, ensuring it remains a versatile and reliable component within the interTwin project, with a focus on use case integration and the improvement of the documentation.
Related Components and Integrations
Core DTE Module: OSCAR
An open-source platform to support the event-driven serverless computing model for data-processing applications. It can...
February 13, 2024
Core DTE Module: SQAaaS platform
Platform for quality assessment and awarding of multiple digital objects (source code, services, data)
...February 14, 2024