August 4, 2023 -
Elphas Toringepi
(@elphastori)
New connectors # We’re excited to announce that Apache Flink now supports three new connectors: Amazon DynamoDB, MongoDB and OpenSearch! The connectors are available for both the DataStream and Table/SQL APIs.
Amazon DynamoDB - This connector includes a sink that provides at-least-once delivery guarantees. MongoDB connector - This connector includes a source and sink that provide at-least-once guarantees. OpenSearch sink - This connector includes a sink that provides at-least-once guarantees.
...
Continue reading »
July 3, 2023 -
Hang Ruan
Apache Flink received the 2023 SIGMOD Systems Award, which is awarded to an individual or set of individuals to recognize the development of a software or hardware system whose technical contributions have had significant impact on the theory or practice of large-scale data management systems:
The 2023 SIGMOD Systems Award goes to Apache Flink:
“Apache Flink greatly expanded the use of stream data-processing.”
Winning of SIGMOD Systems Award indicates the high recognition of Flink’s technological advancement and industry influence from academia.
...
Continue reading »
May 25, 2023 -
Weijie Guo
(@WeijieGuo12)
The Apache Flink Community is pleased to announce the second bug fix release of the Flink 1.16 series.
This release includes 104 bug fixes, vulnerability fixes, and minor improvements for Flink 1.16. Below you will find a list of all bugfixes and improvements (excluding improvements to the build infrastructure and build stability). For a complete list of all changes see: JIRA.
We highly recommend all users upgrade to Flink 1.16.2.
...
Continue reading »
May 25, 2023 -
Weijie Guo
(@WeijieGuo12)
The Apache Flink Community is pleased to announce the first bug fix release of the Flink 1.17 series.
This release includes 75 bug fixes, vulnerability fixes, and minor improvements for Flink 1.17. Below you will find a list of all bugfixes and improvements (excluding improvements to the build infrastructure and build stability). For a complete list of all changes see: JIRA.
We highly recommend all users upgrade to Flink 1.17.1.
...
Continue reading »
May 17, 2023 -
Gyula Fora
(@GyulaFora)
The Apache Flink community is excited to announce the release of Flink Kubernetes Operator 1.5.0! The release focuses on improvements to the job autoscaler that was introduced in the previous release and general operational hardening of the operator.
We encourage you to download the release and share your feedback with the community through the Flink mailing lists or JIRA! We hope you like the new release and we’d be eager to learn about your experience with it.
...
Continue reading »
May 12, 2023 -
Etienne Chauchot
(@echauchot)
Introduction # The Flink community has designed a new Source framework based on FLIP-27 lately. This article is the continuation of the howto create a batch source with the new Source framework article . Now it is time to test the created source ! As the previous article, this one was built while implementing the Flink batch source for Cassandra.
Unit testing the source # Testing the serializers # Example Cassandra SplitSerializer and SplitEnumeratorStateSerializer
...
Continue reading »
May 9, 2023 -
Etienne Chauchot
(@echauchot)
Introduction # The Flink community has been deprecating the DataSet API since version 1.12 as part of the work on FLIP-131: Consolidate the user-facing Dataflow SDKs/APIs (and deprecate the DataSet API) . This blog article illustrates the migration of a real-life batch DataSet pipeline to a batch DataStream pipeline. All the code presented in this article is available in the tpcds-benchmark-flink repo. The use case shown here is extracted from a broader work comparing Flink performances of different APIs by implementing TPCDS queries using these APIs.
...
Continue reading »
May 3, 2023 -
Etienne Chauchot
(@echauchot)
Introduction # The Flink community has designed a new Source framework based on FLIP-27 lately. Some connectors have migrated to this new framework. This article is a how-to for creating a batch source using this new framework. It was built while implementing the Flink batch source for Cassandra. If you are interested in contributing or migrating connectors, this blog post is for you!
Implementing the source components # The source architecture is depicted in the diagrams below:
...
Continue reading »
April 19, 2023 -
Dong Lin
The Apache Flink community is excited to announce the release of Flink ML 2.2.0! This release focuses on enriching Flink ML’s feature engineering algorithms. The library now includes 33 feature engineering algorithms, making it a more comprehensive library for feature engineering tasks.
With the addition of these algorithms, we believe Flink ML library is ready for use in production jobs that require feature engineering capabilities, whose input can then be consumed by both offline and online machine learning tasks.
...
Continue reading »
March 23, 2023 -
Leonard Xu
(@Leonardxbj)
The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache Flink is the leading stream processing standard, and the concept of unified stream and batch data processing is being successfully adopted in more and more companies. Thanks to our excellent community and contributors, Apache Flink continues to grow as a technology and remains one of the most active projects in the Apache Software Foundation. Flink 1.17 had 172 contributors enthusiastically participating and saw the completion of 7 FLIPs and 600+ issues, bringing many exciting new features and improvements to the community.
...
Continue reading »