IBM Throws In with Hortonworks’ Hadoop

IBM and Hortonworks will distribute each other’s products as part of a major extension of their strategic partnership unveiled today at Hortonworks DataWorks Summit/Hadoop Summit in San Jose, California.

As part of the deal, IBM will distribute Hortonworks‘ Hadoop and stream processing distributions, dubbed Hortonworks Data Platform (HDP) and Hortonworks Data Flow (HDF) to its vast customer base, marking the end of its own Hadoop distribution, dubbed BigInsights.

Hortonworks, meanwhile, will include IBM’s data science notebook, called the Data Science Experience, as part of its distributions for HDP and HDF. It will also include IBM’s analytical SQL engine for Hadoop, BigSQL as part of its packaged data warehousing solution.

It’s as if Hortonworks had “eaten IBM,” just as it ate Microsoft earlier, says Forrester analyst Brian Hopkins.

“Working on distributed open source is new and complex,” Hopkins writes in a blog post. “The young developers with the new skills and energy to work on this stuff don’t go generally to work for IBM or Microsoft. They go to work for valley startups who are challenging the big boys.”

While IBM may have the inclination to gobble up Hortonworks — which was spun off from Yahoo in 2011 to develop and productize Hadoop — the advantage would not last long. “As soon as Microsoft or IBM buys Hortonworks, they’ll start bleeding talent,” Hopkins writes.

So instead, we have a situation where Hortonworks has “turned the tables on these giants,” and created a situation where the biggest IT vendors “are now dependent on a relative newcomer for the biggest growth opportunities in the market — machine learning in the cloud using the open source big data ecosystem of tools as infrastructure.”

Hortonworks is also the only Hadoop distributor supporting IBM’s Power8 server, which runs little endian Linux as well as AIX and IBM i. The fact that Hortonworks is also the only Hadoop distributor that supports Windows points to a certain shrewdness on the part of the Santa Clara, California company.

The deal leaves Hortonworks as the main Hadoop and stream processing platform provider for IBM, leaving Big Blue to focus on data science and machine learning software apps, such as the Data Science Experience and its PowerAI offering.

Meeting of the Robs: Bearden (left) greets Thomas on stage during the DataWorks Summit keynote this morning

Hortonworks CEO Rob Bearden says the Data Science Experience, which IBM launched one year ago, will be a key differentiator for his clients. “This will be our strategic data science platform that we take forward to our entire customer base today, both for HDP and HDF,” he said during this morning’s keynote address.

The Hadoop distributor plans to treat the Data Science Experience — which is largely based on the Jupyter data science notebook and Apache Spark —  as its reference architecture upon which it builds more industry-specific solutions to help its customers use data science techniques to leverage their big data. “We could not be more thrilled about the collaboration with IBM’s engineering team, the power of this platform and what this brings to our customer base going forward,” Bearden said.

Data science, Bearden says, is a game changer for companies that can harness it. “Data science is going be the transformational next leg of the journey for all of us,” Bearden continued. “It’s what companies are going to need in order to have a very strategic approach to turn data into value.”

Joining Bearden on stage was Rob Thomas, general manager of IBM’s analytics group, who says focusing on data science will be a boon for IBM.

“It’s a logical extension to where we go now,” Thomas says. “We’ve got to the point now where today we’re shipping the same pieces of a Hadoop distribution.  I’d much rather leverage HDP as part of what we’re doing. I can direct my effort on the focus on data science and machine learning.”

Wall Street seemed to like the news from Hortonworks. Its stock, which is traded on the NASDAQ under the symbol HDP, gained about 10% today. However, its stock is still trading at a level that’s about half of its 2014 IPO price.

Related Items:

Hortonworks: Over 1,000 Customers Served

Hortonworks Shares Vision of Connected Data Planes

IBM Seeks Data Science Unity with New Spark-Based ‘Experience’