Work package 5

Task 5.1 – High-throughput Enactment Engine

Develop a portable, resilient Enactment Engine to execute complex data pipelines across the edge–cloud continuum as scheduled by the Mapper (T5.3). Focus on transparent handling of data parallelism and control-flow for portable pipeline execution.
The EE will monitor execution metrics (e.g., via Prometheus) such as task runtimes, data transfer, and deployment/invocation times, forwarding them to the Unified Knowledge Layer (WP3). On predicted or actual SLO violations, the EE will notify the Mapper (input to T5.3).

Task 5.2 – Pipeline Predictor

Develop ML-based predictors for dynamic pipeline ensembles and uncertainties in workloads and compute resources. Inputs include monitoring data (T5.1), performance experiments, and feedback from T5.3.
The Predictor will (a) forecast near-future workloads and (b) predict resource availability and utilization using RNNs, LSTMs, TCNs, and other time-series models. It supports runtime updates and provides localized/specialized AI components for WP5.

Task 5.3 – Cognitive Mapper

Develop the Mapper, a hybrid scheduler that maps pipeline ensembles to edge–cloud resources. Pipelines will first be transformed for seamless integration with Smart Data Management (T6.1).
The Mapper combines global optimization with runtime heuristics based on T5.2 predictions to:

Maximize throughput and minimize costs.

Adapt rapidly to runtime changes, prediction errors, and load fluctuations (e.g., using Shortest Transmission Time First or Shortest Queue Length First).

After optimization, it negotiates resource SLOs with the Resource Ensemble Manager (WP7). The hybrid scheduler and recommender service balance near-optimal scheduling with adaptive performance.