Ion Stoica’s First Startup Ready to Spread Streaming Data Wings

(INGARA/Shutterstock)

Before co-creating Apache Spark and co-founding Databricks, computer scientist Ion Stoica co-founded Conviva, a video streaming startup used by publishers like NBC and Disney+ to track user metrics and quality of service. Having tackled some of the toughest real-time data processing challenges in Internet video, and seeing a surge of interest in real-time data, Conviva is now looking to apply its streaming data platform to other data use cases in other industries.

Conviva was founded in 2006, at the dawn of the Internet video age. While we take video on the Web for granted today, it was a much different experience back then. Netflix (which just killed its DVD service), wasn’t streaming yet, and YouTube had just started. Ensuring a good customer experience was critical for those early video Internet pioneers, says Aditya Ganjam, chief product officer and co-founder.

“The company delivering the video doesn’t control the network as it was in the cable business where [the cable company] controls the pipes, the set-top box. They encode everything,” he says. “On the Internet, a Disney+ doesn’t control the network, they don’t control the ISP, they don’t control the device in the home. So they need data to inform them on how to optimize their video player.”

Conviva collected data from various sources, including the video player and the application, then correlated it to help customers pinpoint any issues with quality of service, such as buffering or bitrate problems. “It’s a lot of small things that they have to keep on doing on a continuous basis to make sure you can get great quality,” Ganjam says.

Computer scientist Ion Stoica’s first startup is Conviva

One pressing data challenge Conviva had to solve involved resolving complex queries over long-running sessions. In some cases, a single session would take several hours, and Conviva needed to be able to execute queries that took the whole session into account. That was easier said than done.

Ganjam uses the example of a food delivery company to explain the importance of complex, stateful queries on a stream of incoming data. Relatively simple metrics, such as how long customers wait on average between when they place their order and when they get their food, can be processed easily enough with real-time streaming systems.

“But if you want to answer a question like how long did the food order wait on the counter for before it was picked up? Or how long did the delivery food delivery person wait at the restaurant for the order to be completed in cases where the person changes their order? This is a fairly complex metric,” Ganjam says. “Now you need to correlate many events…This is not something many data platforms can do very well. They struggle with that kind of complex correlation.”

Like other companies, Conviva’s original solution was to fuse two frameworks together, in the Lambda architecture style, to address this challenge in video analytics. It used a real-time “speed” layer to answer simple queries while using a separate batch layer to answer more complex queries. The company brought various frameworks to bear on this challenge, including Apache Hadoop as the slower but more thorough batch layer and Apache Spark and Apache Flink as the faster but less complete speed layer. Nothing really worked to the satisfaction of Conviva, particularly those using SQL to process the data, Ganjam says.

“If you try to write this in SQL, it’s actually quite a complex SQL query,” he says. “It can be done. Not to say SQL can’t do it, but it gets complex. And it ends up being hard to write. It gets more error prone…It also ends up suffering terms of performance. So we built a platform.”

An illustrative example of the challenges of TimeState Analytics in video content distribution (Source: Conviva paper “in “Raising the Level of Abstraction for Time-State Analytics With the Timeline Framework”)

The platform that Conviva built also uses two stages. Conviva’s innovation lies in the streaming component, which it calls a time-state processor. The other component Aditya calls a time-series database, such as Apache Druid or ClickHouse (also known as real-time analytics databases). The time-state processor handles the complex stateful calculations, while the time-series database handles OLAP-type queries.

“Most time-series databases are great at multidimensional analytics, so we use that and don’t need to reinvent [the wheel],” Ganjam says. “But they’re not great at this time-state analytics, and so that’s the new piece we built. And we glued these together so we can do time-state and multidimensional.”

Conviva’s key innovation resides in its Time-State processor, which is able to deliver complex queries on large amounts of stateful data. It also created a timeline query language and a visual interface that, together, make it easier for a user to create complex metrics, Ganjam says

“Our timeline is a higher-level abstraction and we understand the intent much better at that abstraction, so the code that implements it can be much more optimized than a lower level abstraction like SQL,” he says.

The current iteration of the timeline framework was developed using Scala and runs atop Akka, which gives better performance than anything Conviva has tried to date. “We actually ran our framework on Spark, on Flink, and the performance was nowhere close to what we could get going down to the levels of Akka,” he says.

Several of the company’s founders recently published a paper explaining how its timeline framework works. Stoica, who graduated from Carnegie Mellon University before taking his current teaching job at UC Berkeley, is listed as one of the co-authors in “Raising the Level of Abstraction for Time-State Analytics With the Timeline Framework,” which you can read here.

Conviva today is a successful business with $100 million in annual recurring revenue. The company boasts that it’s able to process 5 trillion events across 7 billion sensors from 500 million unique viewers people daily. Its technology is brought to bear to ensure an enjoyable Internet video experience for popular events, like the Super Bowl and the World Cup.

With a high-performing Akka-based timeline processor and timeline query language that pairs with a UI to facilitate development of complex stateful queries, the folks at Conviva think they have something that can work outside of the video analytics space.

“We see [complex time-state queries] in many industries,” Ganjam says. “User behavior analytics, behavioral analytics for security for example, IoT–there are many cases where that type of time state analytics shows up and that’s what we’ve been solving and kind of we built a platform that can do that very efficiently and at very high scale.”

The company is currently in the process of making its product applicable to a wider range of user cases and industries. “We are working on making it more general,” Ganjam says. “Then we’ll be launching that as a more general platform.”

Related Items:

Climbing the Ladder of Abstraction

Is Real-Time Streaming Finally Taking Off?

To Improve Data Availability, Think ‘Right-Time’ Not ‘Real-Time’