Data Warehouse Market Ripe for Disruption, Gartner Says

While mega-vendors with names like IBM (NYSE: IBM) and Oracle (NYSE: ORCL) continue to lead the data warehousing space, shifts in the market are creating opportunities for smaller vendors to innovate in areas like cloud deployments and streaming data, Gartner says in its latest Magic Quadrant report.

Disruption is accelerating in the market for data warehousing solutions, Gartner says in its February report. New requirements–such as the need to store and analyze an increasingly diverse array of data types–are leading to a “significant augmentation” of existing data warehouse strategies.

The phrase “data warehouse” no longer conjures an image of a massive relational database used to store normalized data gleaned from the transactional systems of corporations. Since 2014, Gartner has used the phrase to also refer to Hadoop clusters storing sensor data from the IoT, NoSQL databases used to store clickstream data, or cloud-based repositories that store pretty much everything under the sun.

Gartner sees the market splitting into two parts, including enterprise data warehouses (EDWs) on the one hand and logical data warehouses (LDWs) on the other. EDWs refer to what you might consider a traditional data warehouse: an integrated collection of subject-oriented data running on centralized hardware that’s optimized for performance.

Magic Quadrant for Data Warehouse and Data Management Solutions for Analytics

LDWs can hold much of the same data, but are less centralized and rely more on distributed processes and virtualization to create a logical whole. Gartner says LDWs will account for most of the growth in the overall data warehousing space over the next five years.

As the LDW concept takes hold, more companies will move their analytics to the cloud. This will necessitate more hybrid warehousing setups, where some part of the warehouse resides on premise and other parts live on the cloud. Gartner sees this shift to LDWs and the cloud impacting the market for data warehousing appliances, which appear to be losing steam, to the chagrin of data warehousing vendors.

The splitting of the data warehouse camp means a bigger overall tent. Two years ago, Gartner had solutions from 16 vendors in its Magic Quadrant for data warehousing, and a year ago, it had 17. This year, the Magic Quadrant sports solutions from 21 vendors, including new additions like Hadoop distributor Hortonworks, NoSQL database vendor MongoDB, in-memory NewSQL database vendor MemSQL, and Transwarp, a Chinese provider of Hadoop-based analytic software.

The rise of big data lakes (often deployed atop a Hadoop cluster) is clearly impacting the data warehouse environment. In 2015, Gartner says it saw more companies adopting data lakes for three types of uses, including: serving as an archive to the main data warehouse; serving as a sandbox for data discovery and data science exploration; and migration from the data warehouse for extract, load transform (ELT) workloads.

Another trend spells the end of BOB, or best of breed. Instead of assembling a solution by picking the best of each product category, Gartner sees the rise of “best-fit engineering,” where companies select products based on the technological merits and capabilities of each product.

Companies’ engaging in best-fit engineering take a much more granular approach to product selection. In 2016, this approach directly led to the new vendors—specifically MongoDB and MemSQL–being added to the data warehousing magic quadrant. These vendors came from the operational database management space, but their “unique capabilities to perform analytics over operational data collected from diverse applications” netted them entry, Gartner says.

All told, Gartner sees the data warehouse segment being in significant flux for the next few years. Over that time, the market share of traditional data warehouse vendors will erode as EDWs and LDWs take their own separate courses.

Diving deeper into the report shows a mix of winners and losers. Oracle continues to lead the overall pack, which it has led since displacing Teradata (NYSE: TDC) in 2015. Microsoft (NASDAQ: MSFT), SAP (NYSE: SAP) and IBM form a cluster in the leaders quadrant.

Hewlett-Packard Enterprise (NYSE: HPE), once a top-tier data warehousing leader with its Vertica product, has fallen from the leaders quadrant into the challenger’s quadrant, which it shares with NoSQL database vendor MarkLogic; cloud-based data warehouse providers 1010data and Amazon Web Servcies; and Infobright, and up-and-coming provider of a MySQL-based columnar analytic solution that lived in the niche players quadrant last year.

The visionaries quadrant is home to Hadoop distributors (Hortonworks, MapR Technologies, Cloudera, and Pivotal), as well as MemSQL and Transwerp, which has attracted 200 clients in less than 18 months.

The niche players quadrant features MongoDB; Kognitio, which develops an in-memory, massively parallel processing (MPP) NoSQL analytical database; in-memory MPP database vendor Exasol; and Hitachi, which debuted its analytics offering in 2014.

You can find a copy of the latest Magic Quadrant here.

Related Items:

Hadoop and NoSQL Now Data Warehouse-Worthy: Gartner

Why Gartner Dropped Big Data Off the Hype Curve