Managing Streaming Flink Apps Is About To Get Easier

Apache Flink has emerged as a powerful platform for building real-time stream processing applications. However, not every organization has the resources to go all in on Flink the way Netflix, Uber, and Alibaba have. That’s why data Artisans, the company behind Flink, is simplifying the management of Flink applications with a new software offering unveiled today.

Momentum continues to build around Flink, the distributed stream processing framework that came out of the Stratosphere research project at the Technical University of Berlin around the year 2009. Buoyed by big deployments at places like Netflix, which uses Flink to process 1.3 trillion events per day, and Uber, which uses Flink to process more than a petabyte of data per day, the open source project looks well positioned to capture its share of the rapidly emerging market for real-time stream processing, which one analyst said would be worth $13.7 billion by 2021.

Like many powerful big data processing frameworks, however, Flink brings simplicity to some things that used to be complex, while introducing new complexity in other areas. (It would seem to be an unavoidable consequence of making technical progress.) In Flink’s case, the framework simplifies the development of real-time streaming applications, which is great for developers, who no longer need to be Java or Scala experts to build powerful streaming applications.

The tradeoff, however, has been in managing these real-time applications. That’s not an easy task, and required a certain level of engineering and development expertise. That wasn’t necessarily a problem for a big company like Netflix, which has a huge engineering department and build its own home-grown tools for managing its massive Flink infrastructure. However, the management aspect presented a serious hurdle for those organizations who can’t justify building new teams of engineers specifically for this task.

This is the problem that data Artisans has tackled with its commercial Flink offering, dubbed the dA Platform. Today, as the fourth Flink Forward conference kicks off in Berlin, data Artisans unveiled dA Platform 2, which introduces a major new component called the Application Manager.

The dA Platform’s architecture (Source: data Artisans)

The Application Manager, in short, automates much of the nitty-gritty technical tasks that Flink users would have to manually perform during the course of supporting their Flink applications – and, in particular, moving Flink code changes from test into production. Stephan Ewen, data Artisans co-founder and CTO, explained the significance of the new release.

“Developing applications for Flink is easy, it’s simple. It’s actually fun,” Ewen tells Datanami. “Solving this question of how do you guarantee that everything goes well 24/7, how do you integrate it with your continuous integration and deployment pipeline, making sure if you run this on your test cluster, and if everything works, that you can go forward on production cluster…. that’s exactly the friction point we’re trying to eliminate with the Application Manager.”

The Flink framework already supported the notion of snapshots, checkpoints, and save points. Because data is constantly flowing through Flink applications, being able to save the state of the application at any given point in time is important for ensuring consistency. Big Flink customers have taken advantage of these features ensure their Flink-based applications can retain state, even while upgrading their Flink applications or adding new features.

The new Application Manager makes it easy for customers to manage this state without having to write their own tools or become engineering experts, Ewen says.

“Flink gives you building block in some sense to say, if I have a version of the application.  I can actually kill the old application and bring the new application up and move the old snapshot into the new application,” he says. “It gives you the raw ingredients, but it’s a low-level abstraction. With the Application Manager, we’re basically raising the abstraction.”

All of the companies that are running Flink at scale have built their own version of the Application Manager, Ewen says. When dA Platform 2 ships – most likely in early 2018 – that will no longer be necessary, because it will be built into the commercial Flink distribution.

In addition to orchestrates the lifecycle of Flink applications through development cycles, the Application Manager will integrate with Logstash and Grafana to provide developers with logging and metrics, which will help during debugging and development processes. It will also integrate with Kubernetes for dynamic resource allocation, and maintain a record of all user actions and automatic processes for auditing purposes.

“The idea behind the dA platform is to make stream processing available to a wide audience. To make it easily accessible to all these different teams without them needing to rebuild tools again and again, and making it available to companies that don’t have the big teams to say, OK let’s take a year to develop a ton of streaming tools around a stream processor,” the CTO says. “With the dA platform, we basically allow them to focus on the business problem rather than the infrastructure, because we solved the infrastructure [challenges] with our stream processing product.”

Related Items:

Flink Aims to Simplify Stream Processing

Flink: Worth a Second Look

Apache Flink Takes Its Own Route to Distributed Data Processing