Mega-Retail’s Big Data Hunch

When it comes to on-the-fly big data analytics for web-based experiences, there are a host of competing platforms that promise an instant, seamless meld of diverse data to create a unique, customized visit. We heard about the craft retail giant Etsy’s Hadoop and Splunk-driven real-time big data strategy last week, and now eBay is coming clean on how it conjures customer experiences.

It makes solid sense for companies like eBay, Etsy, Amazon, iTunes and others to be able to deliver real-time recommendations and “you might like this too” functionality. Until recently, eBay was pioneering its approach to these features using open source frameworks, but now it has a new tool in its arsenal.

eBay has walked the cutting edge of ultra-fast analytics at web scale since its inception and was among the first to leverage Hadoop in production. For a few insights about how they managed this, there are some rich details available from the company’s Anil Madan, Director of Engineering and Analytics Platform Development.

At this time last year, Madan said eBay was working to address a number of big data challenges, including finding solutions to meet scalability, availability, data discovery, and data movement issues. In 2010 the retailer was also concerned with finding appropriate policies to handle retention, archival and backup (an area that he admits was weak in their overall use of Hadoop) as well as being able to create and maintain active metrics tools that would allow them to “generate metrics for data sourcing, consumption, budgeting, and utilization.” In his description of eBay’s Hadoop experience, he says that “the existing metrics for some of the Hadoop enterprise servers [were] either not enough, or transient, which ma[de] patterns of cluster usage hard to see.

Now eBay has announced another component to their ability to deliver real-time value based on customer and web data with their acquisition of recommendation engine Hunch. As a relatively new company in an increasingly competitive space, this is probably the best thing that could have happened for the almost three-year-old company. The service, as described today by Forbes, “draws on machine learning, data mining and predictive modeling to make suggestions.” According to the same article, eBay will extend this not only to its site-wide functionality, but to carry over the same benefits to advertising and marketing.

In a statement from eBay about the news today there was a good high-level overview of what the company can offer the web giant. They claim the ability to mine across the web (including social networks) to scour for things members like, who they follow, and answers to Hunch’s own questions . They say this will allow eBay “to build unique user profiles that make more personal recommendations and model individuals’ tastes. Once integrated, unlike traditional online retail approaches, Hunch will enable eBay to move beyond standard item-to-item recommendations and use a broader variety of members’ online tastes and interests to suggest new and interesting items for them to browse and buy on eBay.”

According to eBay, this new addition will allow for the generation of “meaningful, yet often non-obvious recommendations for items available on eBay based on their specific tastes.” According to Elizabeth Woyke, “For the past year, eBay has been testing a tool called Discover that serves up unexpected results through a sophisticated algorithm. Hunch’s technology will augment that initiative by mining shoppers’ actions outside eBay, including their activity on social networks. Adding this data will enable eBay to move beyond standard item-to-item recommendations by suggesting products based on a broader variety of members’ online tastes and interests.”

Although they’ve been acquired Hunch will continue to operate its platform independently as usual via its website.