SingleStore Turbocharges Data Lakehouses with Iceberg Support and Faster Vector Search

Graphic courtesy of SingleStore.

SingleStore, a real-time data platform for transacting, analyzing, and contextualizing data, has announced a bi-directional integration with Apache Iceberg to enable easier data access for users, and to allow them to build intelligent applications on data lakehouses. 

With its outstanding efficiency and flexibility in data management, Apache Iceberg is quickly emerging as the de facto standard for storing diverse datasets. It can handle large-scale datasets with high efficiency, supports scheme evaluation and versioning, and can integrate with various big data tools. 

However, enterprises often face challenges in the vast amounts of data stored in their data lakehouses. According to SingleStore, 90% of data remains “frozen” in lake lakehouses and is unusable for analytics, real-time applications, and AI. 

One of the major challenges is the costly and complex process of moving data into Iceberg-based data lakes, a process known as extract/transform/load. With the bidirectional integration with Apache Iceberg, SingleStore aims to solve this problem by providing low-latency ingestion and real-time operation.

“Our vision has always been to provide one single data store for all companies to be able to take advantage of speed, scale and simplicity,” said Raj Verma, CEO, SingleStore. “Our data platform is designed to unlock all types of enterprise data — including data that is frozen in data lakes — to enable our customers to build modern intelligent apps. With this release, we believe we are enabling a significant portion of the market that today cannot build real-time modern applications on data stored in data lakes.”

The bidirectional Apache Iceberg integration is available in public preview. A general availability date hasn’t been announced yet. 

SingleStore also announced improved vector search capabilities, available in public preview. The company claims it has improved the performance of its vector search by about 40% compared to the previous iterations of the platform. It has achieved this by leveraging hierarchical navigable small world (HNSW) algorithms that can efficiently find the most relevant data points for a query in large datasets. 

In addition, SingleStore has added new filtering capabilities for vector searches to help enterprises easily build and scale GenAI applications.

The new release also includes the general availability of improved full-text search capabilities including improved relevance scoring, keyword-proximity-based ranking, fuzzy matching, and phonetic similarity. This can help simplify data architectures by minimizing the need for specialized databases to build GenAI and real-time applications. 

To allow for optimized application performance without billing surprises, SingleStore has released a public preview of its Autoscaling feature which can scale compute resources as needed. 

SingleStore has also introduced a fully managed cloud offering called Helios BYOC that offers a wide range of management features and scalability of a database-as-a-service platform. Available in private preview on AWS, this new offering enables customers to keep their data in a virtual private cloud (VPC) for security and governance purposes and deploy SingleStore within that VPC.

Earlier this year, SingleStore entered into a strategic partnership with AWS to accelerate its go-to-market strategy.  The newly announced offerings take SingleStore closer to its goal of offering a real-time data platform for the next wave of generative AI and data applications. Having surpassed $100 million in annual recurring revenue (ARR) in 2023, SingleStore is on its way to establishing a strong foothold in the market.

Related Items 

Databricks Nabs Iceberg-Maker Tabular to Spawn Table Uniformity

Dremio Integrates Apache Iceberg REST to Promote Vendor-Agnostic Ecosystem

Fivetran Expands Snowflake Partnership with Iceberg Table Support and Enterprise-Ready Native Applications