Elastic Launches Search AI Lake to Scale Low Latency Search

The rapidly growing scale of data has led to the emergence of data lakes, which offer a centralized repository for the storage of structured and unstructured data at any scale. Data lakes architecture often separates computing and storage to enable scalability and flexibility in handling large volumes of data.

However, these architectures often prioritize scalability over performance, making them less suitable for real-time applications that need both low-latency querying and access to all the data. To help address this issue, Elastic, an enterprise search technology provider, has introduced a new lake architecture. 

With the Search AI Lake, Elastic offers a cloud-native architecture optimized for low latency applications including search, retrieval augmented generation (RAG), observability, and security. The new service has the ability to scale search across exponentially large data sets for rapid querying of data in the form of vectors. 

The approach taken by Elastic for using data lakes is significantly different from other competitors, including Snowflake and Databricks. Unlike those platforms, Elastic brings search functionality into the data lake to enable real-time data exploration and queries within. This eliminates the need for any predefined schemes. 

Most of the major data lake and data lakehouse vendors use one or more data lake table formats such as Apache Iceberg or Databricks Delta Lake. However, ElasticSearch AI Lake doesn’t use any of these table formats. Search AI Lake uses Elastic Common Schema format and the Elasticsearch Query Language to explore data in a federated way across the Elastic clusters. 

“To meet the requirements of more AI and real-time workloads, it’s clear a new architecture is needed that can handle compute and storage at enterprise speed and scale – not one or the other,” said Ken Exner, chief product officer at Elastic. 

Exner further added, “Search AI Lake pours cold water on traditional data lakes that have tried to fill this need but are simply incapable of handling real-time applications. This new architecture and the serverless projects it powers are precisely what’s needed for the search, observability, and security workloads of tomorrow.”

The new Search AI Lake also powers the Elastic Cloud Serverless service, helping remove operational overhead by automatically scaling and managing workloads. With its rapid onboarding and hassle-free administration, Elastic Cloud Services is tailored to harness the speed and scale of Search AI Lake. 

Elastic Cloud Serverless and Search AI Lake are currently available in tech preview. Users looking for more control can use Elastic Self-Managed service, while users who prefer greater simplicity can benefit from Elastic Cloud Serverless. 

The introduction of the new capabilities signals a significant transformation in data architecture, heralding a new era of low-latency apps powered by Elastic. With Search AI Lake and Elastic Cloud Serverless, Elastic has positioned it as a comprehensive data platform for GenAI models. Elastic deployments can help enhance the performance and efficiency of LLMs by enabling access to the most relevant data as it becomes available in real-time.  

Related Items 

Elastic Enhances Security Operations with AI-Assisted Attack Discovery and Analysis

How Real-Time Vector Search Can Be a Game-Changer Across Industries

Elastic Security Labs Releases Guidance to Avoid LLM Risks and Abuses