Elastic Gives Search Engine a Graph Option

Elastic today announced that it’s added a graph query engine to Elasticsearch engine users now have the option of using their search indexes as the basis for conducting graph analyses. The new option will make it relatively easy for customers to conduct big data analysis for use cases such as fraud detection and product recommendations.

For some time, Elastic has been adding elements to the search engine that collects various statistics about the content that customers are storing and indexing within the Elasticsearch engine. The statistics are used to compute, among other things, the relevancy of individual pieces of content, and the ways in which they might be connected.

According to Steve Kearns, senior director of product management for Elastic (the commercial venture behind the open source software), the company realized that it had the start of a graph analytics engine, and that with a little bit of effort, it could offer it as a formal option atop Elasticsearch, which is essentially a key value store, or a NoSQL database.

“We recognize that there’s a lot of interesting relationships that are represented,” Kearns tells Datanami. “We started to recognize that, if we thought about these relationships different, we can actually have a new type of questioning.”

Instead of delivering summarized results of the properties of documents, or enabling users to slice and dice those properties to find interesting things, the company realized that it could enable customers to ask questions directly against those relationship entities that are calculated by the search engine, in much the same way that graph analytic engines do.

“What we’re trying to do here is shift the way you think about asking questions,” he says. “We’re allowing you to query the relationship and then use relevance ranking to prioritize the ones that are more open. This opens up a host of new use cases atop Elasticsearch.”

One use case that comes to the forefront is fraud detection. Given 50 instances of credit card fraud, the graph product can instantly analyze all the accounts and pick out the commonalities. Because people tend to shop at the same places—Amazon, Starbucks, CVS, to name a few—the first set of results wouldn’t be that useful, Kearns said.

But thanks to Elasticsearch’s capability to detect relevance within the data, the graph product will automatically bubble up the most interesting and useful results. It’s not replacing the job of the skilled analyst who’s driving the investigation, Kearns says. “You still need the analyst,” he says. “But we’re going to give them a much better prioritized list.”

The company is selling access to the Graph product alongside its other add-on solutions for Elasticsearch, which includes things like enhanced security and monitoring. With 50 million downloads of Elasticsearch, there is a considerable built-in audience who might be receptive to this product.

In many ways, Graph offers users the same types of capabilities that users can get by building a machine learning model in Hadoop or implementing a dedicated graph database product. But by adding the graph analysis capability to one of the world’s most popular search engine, Kearns envision giving a much broader group of customers access to sophisticated big data analytics.

“I don’t expect this to put Hadoop out of business or to put any graph database out of business,” he says. “I do expect people to be solving a new class of problem with a much smaller set of technology. Whereas before you might have had to have a Hadoop cluster to answer a question like a recommendation engine, now you don’t need that. You may decide you need to go there [Hadoop] for certain reasons, but it won’t be because you can’t do it on Elasticsearch, which in many cases is where this data already lives.”

The fact that the relationships are continually calculated upon data that’s already being collected gives the Elastic Graph an edge over competing technologies, Kearns says. All things being equal, the simpler solution typically wins.

“A lot of it comes down to where does this data live naturally,” he says. “Do I have to take it out and change the structure? How much human and machine time has to go into this, and what’s the lag?  If I’m building a machine-learned recommendation model, there’s a whole bunch of challenges here…whereas with us, you’re just running a query.  It’s very powerful in letting people experiment and understand and work with their data.”

Besides fraud detection, Elastic foresees Graph being used for behavioral analysis, cybersecurity, drug discovery, personalized medicine, and making personalized recommendations.

Related Items:

Solr or Elasticsearch–That Is the Question

Here’s Another Option for Hadoop Enterprise Search