Alibaba Acquires Apache Flink Backer data Artisans

Chinese Internet giant Alibaba Group has acquired data Artisans, the German company behind the Apache Flink data processing framework. Alibaba is one of the biggest users of Flink, and plans to continue developing and Flink and contributing enhancements back to the open source project.

Apache Flink originated from the Stratosphere research project at the Technical University of Berlin in 2009, and in 2015 became a top level project at the Apache Software Foundation. The software features a dataflow engine developed in Java and Scala that’s designed to run batch and streaming analytic workloads on distributed systems, including Hadoop clusters and cloud-based systems.

Flink is a contemporary of Apache Spark, and in fact boasted certain advantages over Spark, particularly when it comes to real-time streaming analytics, although Spark closed the gap last year when it replaced “micro-batch” streaming with a more continuous approach. Nevertheless, Flink’s fault-tolerance, exactly once semantics, support for ACID, high throughput, and low latency has earned it the business of big customers who need to process large amounts of data in a fast manner, including Disney, Netflix, Uber, and Alibaba.

The creators of Flink founded data Artisans to build commercial software based on Flink, called dA Platform, which debuted in 2016. The core differentiating element between the open source project and dA Platform is dA Application Manager, a component designed to help organizations more quickly build Flink applications, with specific capabilities around container orchestration, continues integration and continuous development, and logging, metrics.

Last year, data Artisans added a Streaming Ledger, which brought support for ACID transactions on streaming data. The company said the proprietary Streaming Ledger feature would be useful for banks and other institutions building applications that had strict requirements for accuracy and recoverability of transactional data.

Like the American Web giants, Alibaba Group is heavily investing in big data and AI technologies to develop a competitive edge across its business lines. The company, which has CN¥250 billion (about $40 billion) in 2018 revenues, is a leading provider of e-commerce, search engine, and cloud computing services, and is expanding into other areas, including media, venture capital, payment networks, and personal assistants.

Jingren Zhou, a vice president for Alibaba Group, says the new deal with data Artisans will be a win-win for Alibaba and the community at large.

“By leveraging the technology expertise of both teams and shared passion to develop the open-source community, we are confident that this strategic tie-in will further strengthen the growth of the Flink community, accelerate the data-processing technologies and help bolster an open, collaborative and constructive environment for global developers who are passionate about stream processing and enabling real-time applications for modern enterprises,” Zhou says in a blog post published today.

Executives with data Artisans say they welcome the opportunity to work more closely with Alibaba.

“Alibaba, as one of the largest production users and biggest contributors to Apache Flink, in close collaboration with the open source community and data Artisans team, has made numerous contributions to the Flink codebase over the last 2 years,” co-founders Stephan Ewen and Kostas Tzoumas writes in a blog post on the data Artisans website.

As a first step Alibaba have announced their commitment to contribute their in-house developments around Flink, called Blink, to the open source community. There will be even more contributions to open source in the future, say Tzoumas, the CEO, and Ewen, the CTO.

“We can expand to new areas that we have not explored in the past and make sure that Flink becomes a more valuable data processing framework for the modern data-driven, real-time enterprise,” the data Artisans co-founders say.

While neither company explicitly stated that the deal was an acquisition in their blog posts (the Alibaba post described it as a “deepening of the partnership” while the data Artisans post said they were “joining forces”), Datanami has confirmed with somebody with knowledge of the deal that it was, in fact, an acquisition.

Terms of the deal were not disclosed.

Related Items:

Flink Delivers ACID Transactions on Streaming Data

Apache Flink Gets An SQL Client

Apache Flink Takes Its Own Route to Distributed Data Processing