February 10, 2021 -
Yang Wang
Flink has supported resource management systems like YARN and Mesos since the early days; however, these were not designed for the fast-moving cloud-native architectures that are increasingly gaining popularity these days, or the growing need to support complex, mixed workloads (e.g. batch, streaming, deep learning, web services). For these reasons, more and more users are using Kubernetes to automate the deployment, scaling and management of their Flink applications.
From release to release, the Flink community has made significant progress in integrating natively with Kubernetes, from active resource management to “Zookeeperless” High Availability (HA).
...
Continue reading »
January 29, 2021 -
Xintong Song
The Apache Flink community released the third bugfix version of the Apache Flink 1.10 series.
This release includes 36 fixes and minor improvements for Flink 1.10.2. The list below includes a detailed list of all fixes and improvements.
We highly recommend all users to upgrade to Flink 1.10.3.
Updated Maven dependencies:
<dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-java</artifactId> <version>1.10.3</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java_2.11</artifactId> <version>1.10.3</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients_2.11</artifactId> <version>1.10.3</version> </dependency> You can find the binaries on the updated Downloads page.
...
Continue reading »
January 19, 2021 -
Xintong Song
The Apache Flink community released the first bugfix version of the Apache Flink 1.12 series.
This release includes 79 fixes and minor improvements for Flink 1.12.0. The list below includes a detailed list of all fixes and improvements.
We highly recommend all users to upgrade to Flink 1.12.1.
Attention: Using unaligned checkpoints in Flink 1.12.0 combined with two/multiple inputs tasks or with union inputs for single input tasks can result in corrupted state.
...
Continue reading »
January 18, 2021 -
Jun Qin
Stream processing applications are often stateful, “remembering” information from processed events and using it to influence further event processing. In Flink, the remembered information, i.e., state, is stored locally in the configured state backend. To prevent data loss in case of failures, the state backend periodically persists a snapshot of its contents to a pre-configured durable storage. The RocksDB state backend (i.e., RocksDBStateBackend) is one of the three built-in state backends in Flink.
...
Continue reading »
January 11, 2021 -
Robert Metzger
(@rmetzger_)
Apache Flink is a very versatile tool for all kinds of data processing workloads. It can process incoming data within a few milliseconds or crunch through petabytes of bounded datasets (also known as batch processing).
Processing efficiency is not the only parameter users of data processing systems care about. In the real world, system outages due to hardware or software failure are expected to happen all the time. For unbounded (or streaming) workloads, Flink is using periodic checkpoints to allow for reliable and correct recovery.
...
Continue reading »
January 7, 2021 -
Jianyun Zhao
(@yihy8023)
Jennifer Huang
(@Jennife06125739)
About the Pulsar Flink Connector # In order for companies to access real-time data insights, they need unified batch and streaming capabilities. Apache Flink unifies batch and stream processing into one single computing engine with “streams” as the unified data representation. Although developers have done extensive work at the computing and API layers, very little work has been done at the data messaging and storage layers. In reality, data is segregated into data silos, created by various storage and messaging technologies.
...
Continue reading »
January 2, 2021 -
Tzu-Li (Gordon) Tai
(@tzulitai)
The Apache Flink community released the second bugfix release of the Stateful Functions (StateFun) 2.2 series, version 2.2.2.
The most important change of this bugfix release is upgrading Apache Flink to version 1.11.3. In addition to many stability fixes to the Flink runtime itself, this also allows StateFun applications to safely use savepoints to upgrade from older versions earlier than StateFun 2.2.1. Previously, restoring from savepoints could have failed under certain conditions.
...
Continue reading »
December 18, 2020 -
Xintong Song
The Apache Flink community released the third bugfix version of the Apache Flink 1.11 series.
This release includes 151 fixes and minor improvements for Flink 1.11.2. The list below includes a detailed list of all fixes and improvements.
We highly recommend all users to upgrade to Flink 1.11.3.
Updated Maven dependencies:
<dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-java</artifactId> <version>1.11.3</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java_2.11</artifactId> <version>1.11.3</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients_2.11</artifactId> <version>1.11.3</version> </dependency> You can find the binaries on the updated Downloads page.
...
Continue reading »
December 10, 2020 -
Marta Paes
(@morsapaes)
Aljoscha Krettek
(@aljoscha)
The Apache Flink community is excited to announce the release of Flink 1.12.0! Close to 300 contributors worked on over 1k threads to bring significant improvements to usability as well as new features that simplify (and unify) Flink handling across the API stack.
Release Highlights
The community has added support for efficient batch execution in the DataStream API. This is the next major milestone towards achieving a truly unified runtime for both batch and stream processing.
...
Continue reading »
December 2, 2020 -
Andrey Zagrebin
The Flink community has been working for some time on making Flink a truly unified batch and stream processing system. Achieving this involves touching a lot of different components of the Flink stack, from the user-facing APIs all the way to low-level operator processes such as task scheduling. In this blogpost, we’ll take a closer look at how far the community has come in improving scheduling for batch workloads, why this matters and what you can expect in the Flink 1.
...
Continue reading »