Elastic Adds ‘One Click’ Anomaly Detection to Stack

(enzozo/Shutterstock)

The company behind Elasticsearch today unveiled the addition of machine learning algorithms to its enterprise stack for the purpose of finding anomalies in time-series log data. It’s just the start of the IoT use cases, Elastic CEO Shay Banon tells Datanami.

Elastic says its new “one-click” experience will make it much easier to deploy machine learning-powered analytic capabilities atop its stack, while integration with its Kibana user interface layer will allow companies to visualize the data to better detect operational issues, incidence of cybersecurity, and detect fraud.

The unsupervised machine learning algorithms were originally developed by Prelert, which Elastic acquired last year. The clustering algorithms previously lived separate from the Elastic cluster, and were also integrated with Splunk.

Now that Prelert is part of Elastic, the company decided to make the algorithm “fully native” to the Elastic Stack product and eliminate the Splunk integration. The result of that is better performance for Elastic Stack users, Banon says.

“Once we removed all the constraints and just decided that it becomes a native feature of the stack, we can go 10x or 100x in terms of ease of use,” says Banon, who was one of Datanami‘s “People to Watch” for 2017. “And what that means is, we have a one-click anomaly detection experience on our stack.”

Bringing the algorithm under the Elastic hood also provides scalability and availability benefits. “It’s just another component in that cluster,” Banon says. It’s fully distributed. If a single machine dies, then the stream of data is picked up and analyzed, without any loss of data, so it’s highly available – everything that you expect from a distributed system.”

Elastic will automatically generate visualizations based on the results of the algorithmic analyses using its Kibana software, one component of what it used to call the “ELK” stack (the others are Elasticsearch, Logstash, Beats, and Prelert).

Kibana automatically generates visualizations based on machine learning results in Elastic

“There’s a really nice presentation layer on top of it to make sure that we bubble up and show you the right type of anomalies that we need you to see,” Banon says. “As those streams of data are being analyzed, anomalies are being detected and users are notified.”

The machine learning happens in conjunction with the initial data indexing, says Steve Dodson, Elastic’s machine learning technology lead and the former CTO of Prelert. “We’re actually querying the data as it’s being indexed, so it’s slightly behind real time from that perspective,” he says.

Because the algorithm is unsupervised, it’s able to detect anomalies on time-series data without any training, Banon says. “We see quite a bit of time series data stored and visualized in ElasticSearch,” he tells Datanami.

It’s common to find Elastic customers streaming operational data, such as CPU or memory usage for hundreds or thousands of servers, or other metrics like key performance indicators (KPIs) into ElasticSearch. The company is also eyeing other IoT use cases as well.

“Those data sets are the sweet spots of our anomaly detection engine,” Banon says. Customers will be able to stream data “from thousands or hundreds of thousands of devices into Elasticsearch then be able to detect the device that’s causing a problem. That’s something we’ll be able to provide out of the box to our users starting from 5.4.”

This first release targets anomaly detection in time-series data using unsupervised machine learning, but Elastic has plans to broaden the use cases as well as the technology. That includes supervised machine learning, Dodson says.

“We’re starting with log files and time series metrics,” he says. “But we’re also now developing additional machine learning capabilities beyond that to add value to people who don’t necessarily have time series data. We’re broadening our set of technologies to give customer insight beyond what they can get through search, aggregation, and visualization.”

The new time-series anomaly detection is available with X-Pack, an enterprise add-on to the open source Elasticsearch stack that also includes security, monitoring, alerting, graph analytics, and reporting capabilities.

Banon created the Elasticsearch search engine in 2004 as a modified version of Lucene that featured a JSON data type and Java and REST APIs. He founded the company Elasticsearch BV in 2012 to productize the search engine and the associated ELK stack. Banon recently took over the CEO of the company (which changed its name to Elastic in 2015) and also moved from Amsterdam, where the company was founded, to Mountain View, California to guide the next stage of the company’s growth.

Related Items:

Elastic Stack Searches for Bigger Data Problems

Elastic Gives Search Engine a Graph Option

Search Engines Get a Machine Learning Boost