Incremental Stream Query Deployment under Continuous Infrastructure Changes in the Cloud-Edge Continuum

Abstract

Distributed data stream processing engines (DSPEs) operating over the cloud-edge continuum must deploy data processing operators across a distributed infrastructure. However, the volatile nature of these infrastructure nodes—where devices frequently join, leave, or move—can invalidate existing query operator-to-topology node mappings, leading to interruptions in query execution and potential data loss. To ensure continuous processing while maintaining correctness, DPSEs must dynamically adapt these mappings and redeploy (part of) affected queries. In this paper, we introduce incremental stream query deployment ISQD, a framework that efficiently redeploys queries affected by topology changes. ISQD employs a greedy strategy to identify and redeploy only affected operators. It uses ad-hoc queries to migrate operator state seamlessly, and leverages reconfiguration markers to synchronize the redeployment process. Our evaluation shows that ISQD achieves up to 7.5x lower deployment latency and up to 39x lower event time latency compared to state-of-the-art approaches, even under high-frequency topology changes.