<abstract xmlns="http://www.w3.org/1999/xhtml">

<p>Data stream processing has been gaining attention in the past decade. Apache Flink is an open-source distributed stream processing engine that is able to process a large amount of data in real time with low latency. Computations are distributed among a cluster of nodes. Currently, provisioning the appropriate amount of cloud resources must be done manually ahead of time. A dynamically varying workload may exceed the capacity of the cluster, or leave resources underutilized. In our paper, we describe an architecture that enables the automatic scaling of Flink jobs on Kubernetes based on custom metrics, and describe a simple scaling policy. We also measure the e ects of state size and target parallelism on the duration of the scaling operation, which must be considered when designing an autoscaling policy, so that the Flink job respects a Service Level Agreement.</p>
</abstract>

Data stream processing has been gaining attention in the past decade. Apache Flink is an open-source distributed stream processing engine that is able to process a large amount of data in real time with low latency. Computations are distributed among a cluster of nodes. Currently, provisioning the appropriate amount of cloud resources must be done manually ahead of time. A dynamically varying workload may exceed the capacity of the cluster, or leave resources underutilized. In our paper, we describe an architecture that enables the automatic scaling of Flink jobs on Kubernetes based on custom metrics, and describe a simple scaling policy. We also measure the e ects of state size and target parallelism on the duration of the scaling operation, which must be considered when designing an autoscaling policy, so that the Flink job respects a Service Level Agreement.

Towards autoscaling of Apache Flink jobs

ELTE Eötvös Loránd University Budapest

Acta Universitatis Sapientiae, Informatica

This work is licensed under the Creative Commons Attribution 4.0 International License.

{"article-title":"Towards autoscaling of Apache Flink jobs"}

Data stream processing has been gaining attention in the past decade. Apache Flink is an open-source distributed stream processing engine that is able to...

Towards autoscaling of Apache Flink jobs

Published Online: Jul 08, 2021

Page range: 39 - 59

Received: Mar 22, 2021

Accepted: Apr 11, 2021

DOI: https://doi.org/10.2478/ausi-2021-0003

Keywords
Apache Flink, autoscaling, data stream processing, big data, kubernetes, distributed computing

© 2021 Balázs Varga et al., published by Sciendo

This work is licensed under the Creative Commons Attribution 4.0 International License.

Towards autoscaling of Apache Flink jobs

Published Online: Jul 08, 2021

Page range: 39 - 59

Received: Mar 22, 2021

Accepted: Apr 11, 2021

DOI: https://doi.org/10.2478/ausi-2021-0003

KeywordsApache Flink, autoscaling, data stream processing, big data, kubernetes, distributed computing

© 2021 Balázs Varga et al., published by Sciendo

This work is licensed under the Creative Commons Attribution 4.0 International License.

Keywords
Apache Flink, autoscaling, data stream processing, big data, kubernetes, distributed computing