September 13, 2019 -
Seth Wiesman
(@sjwiesman)
Fabian Hueske
(@fhueske)
Whether you are running Apache FlinkⓇ in production or evaluated Flink as a computation framework in the past, you’ve probably found yourself asking the question: How can I access, write or update state in a Flink savepoint? Ask no more! Apache Flink 1.9.0 introduces the State Processor API, a powerful extension of the DataSet API that allows reading, writing and modifying state in Flink’s savepoints and checkpoints.
In this post, we explain why this feature is a big step for Flink, what you can use it for, and how to use it.
...
Continue reading »
September 11, 2019 -
Jark Wu
(@JarkWu)
The Apache Flink community released the second bugfix version of the Apache Flink 1.8 series.
This release includes 23 fixes and minor improvements for Flink 1.8.1. The list below includes a detailed list of all fixes and improvements.
We highly recommend all users to upgrade to Flink 1.8.2.
Updated Maven dependencies:
<dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-java</artifactId> <version>1.8.2</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java_2.11</artifactId> <version>1.8.2</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients_2.11</artifactId> <version>1.8.2</version> </dependency> You can find the binaries on the updated Downloads page.
...
Continue reading »
September 5, 2019 -
Marta Paes
(@morsapaes)
This has been an exciting, fast-paced year for the Apache Flink community. But with over 10k messages across the mailing lists, 3k Jira tickets and 2k pull requests, it is not easy to keep up with the latest state of the project. Plus everything happening around it. With that in mind, we want to bring back regular community updates to the Flink blog.
The first post in the series takes you on an little detour across the year, to freshen up and make sure you’re all up to date.
...
Continue reading »
August 22, 2019 -
The Apache Flink community is proud to announce the release of Apache Flink 1.9.0.
The Apache Flink project’s goal is to develop a stream processing system to unify and power many forms of real-time and offline data processing applications as well as event-driven applications. In this release, we have made a huge step forward in that effort, by integrating Flink’s stream and batch processing capabilities under a single, unified runtime.
...
Continue reading »
July 23, 2019 -
Nico Kruber
Piotr Nowojski
In a previous blog post, we presented how Flink’s network stack works from the high-level abstractions to the low-level details. This second blog post in the series of network stack posts extends on this knowledge and discusses monitoring network-related metrics to identify effects such as backpressure or bottlenecks in throughput and latency. Although this post briefly covers what to do with backpressure, the topic of tuning the network stack will be further examined in a future post.
...
Continue reading »
July 2, 2019 -
Jincheng Sun
(@sunjincheng121)
The Apache Flink community released the first bugfix version of the Apache Flink 1.8 series.
This release includes more than 40 fixes and minor improvements for Flink 1.8.1. The list below includes a detailed list of all improvements, sub-tasks and bug fixes.
We highly recommend all users to upgrade to Flink 1.8.1.
Updated Maven dependencies:
<dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-java</artifactId> <version>1.8.1</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java_2.11</artifactId> <version>1.8.1</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients_2.11</artifactId> <version>1.8.1</version> </dependency> You can find the binaries on the updated Downloads page.
...
Continue reading »
June 26, 2019 -
Fabian Hueske
(@fhueske)
Since version 1.5.0, Apache Flink features a new type of state which is called Broadcast State. In this post, we explain what Broadcast State is, and show an example of how it can be applied to an application that evaluates dynamic patterns on an event stream. We walk you through the processing steps and the source code to implement this application in practice.
What is Broadcast State? # The Broadcast State can be used to combine and jointly process two streams of events in a specific way.
...
Continue reading »
June 5, 2019 -
Nico Kruber
Flink’s network stack is one of the core components that make up the flink-runtime module and sit at the heart of every Flink job. It connects individual work units (subtasks) from all TaskManagers. This is where your streamed-in data flows through and it is therefore crucial to the performance of your Flink job for both the throughput as well as latency you observe. In contrast to the coordination channels between TaskManagers and JobManagers which are using RPCs via Akka, the network stack between TaskManagers relies on a much lower-level API using Netty.
...
Continue reading »
May 17, 2019 -
Fabian Hueske
(@fhueske)
Andrey Zagrebin
A common requirement for many stateful streaming applications is to automatically cleanup application state for effective management of your state size, or to control how long the application state can be accessed (e.g. due to legal regulations like the GDPR). The state time-to-live (TTL) feature was initiated in Flink 1.6.0 and enabled application state cleanup and efficient state size management in Apache Flink.
In this post, we motivate the State TTL feature and discuss its use cases.
...
Continue reading »
May 14, 2019 -
Marta Paes
(@morsapaes)
Figuring out how to manage and model temporal data for effective point-in-time analysis was a longstanding battle, dating as far back as the early 80’s, that culminated with the introduction of temporal tables in the SQL standard in 2011. Up to that point, users were doomed to implement this as part of the application logic, often hurting the length of the development lifecycle as well as the maintainability of the code.
...
Continue reading »