Hands-Off: Manual Data Integration Tasks Plummeting, Gartner Says

While the need to integrate data has never been greater, the addition of machine learning and other forms of automation is driving a large reduction in the amount of manual data management tasks that human workers are required to do, Gartner says in a new Magic Quadrant report.

According to Gartner analysts, manual data management tasks will decrease by 45% through 2022 as a result of ML and service-level management activities. What’s more, the rise of AI in data management and integration will reduce the number of dedicated IT specialists needed by 20%, the analysts write.

The rise of new design patterns, including data fabrics, multi- and hybrid cloud, and DataOps–not to mention the competing challenges and opportunities unleashed by COVID-19–is helping to drive new life into the data integration space, according to Gartner analysts.

“The data integration tools market grew at 6.8% in 2020 as compared with 7.0% in 2019,” the analysts wrote. “The market was largely not impacted by the pandemic, and this is because data integration is key to running mission-critical systems–not a luxury/discretionary tool purchase.”

In terms of vendors, Gartner noted that big integration tool vendors have lost ground to newcomers. From 2017 to 2021, the share of market owned by large vendors has dropped from 71% to 56%, it says. The analyst group attributes this to several factors, including a focus on delivering integration as a service offerings by the larger vendors.

The growing balkanization of the data integration market into specific segments has also helped to level the overall playing field, Gartner says. For example, some vendors focus on batch data integration, through ETL or ELT processes (such as Fivetran and Matillion) while others focus on data replication through change data capture (CDC) processes (such as Qlik/Attunity and HVR Software) while others still focus on data virtualization processes (Denodo).

Overall, the market for data integration tools continues to push toward “distributed and dynamic data-management-based designs,” the analysts write. “The push is inherent in the modern data fabric architecture…Data fabric is critical and driven by the end-user push toward augmented data integration.”

Gartner’s Magic Quadrant for Data Integration 2021

Gartner says the bigger vendors will need to find a balance between pushing for all-encompassing platform solutions, such as data fabric offerings, versus selling easily accessible point solutions, such as the cloud-based offerings that Matillion and Fivetran have found success selling. The smaller, independent solutions also have an advantage when it comes to helping clients avoid vendor lock-in, which is a concern with the vertically integrated BI and CRM environments, the analysts say.

Another trend spotted by Gartner is a shift toward more data engineering use cases. Companies are increasingly looking to leverage automation to build pipelines that move data to various analytics platform (many of which run in the cloud). Programming languages like Python and Scala are increasingly being used to “automate data pipeline builds, regression tests, deployments and operations monitoring,” the analysts write.

Hybrid and multi-cloud environments are becoming increasingly common, and this impacts data integration tool choice selection. Gartner says that about half of the companies it surveyed have data on-prem and in the cloud, which means support for hybrid deployment scenarios is “critical,” it says.

Keeping all of the data flowing, well-managed, and in-synch across multiple cloud and on-prem locations is not easy, but it’s important to get it right, Gartner says. “Data location impacts performance, data sovereignty, application latency SLAs, high-availability and disaster recovery strategies, and financial models,” the analysts write, adding that they are giving a higher weight in the “vision” segment of its rating system to tools that can run in hybrid environment effectively .

Interestingly, Gartner analysts are seeing more blending of data and application integration disciplines. For example, more CDC tooling that can capture application changes into message queues, has the potential to benefit customers, they say.

Machine learning  is being adopted in data integration (a-image/Shutterstock)

“By combining data integration and application integration, organizations are exploiting the intersection of the two disciplines,” Gartner analysts write. “They align both infrastructure for composing applications and data pipelines for supporting a broad range of operational flows, while simultaneously optimizing costs and competencies.”

While a blending of some capabilities is seen as a good thing, customers in the end demand modularity, Gartner analyst write. “Cost optimization through modularity and tightly integrated (but loosely coupled) data integration tool options is critical,” they write.

“The market is now expected to deliver modular solutions through loosely coupled architectures, where specific capabilities that are urgently needed by organizations (such as data catalogs or data virtualization) can be delivered and consumed as needed (without having to buy the entire platform upfront,” the analysts continue.

Informatica continues to lead the pack in the Magic Quadrant, followed by industry bigs IBM, SAP, and Oracle. Talend revisits its role in the Leaders quadrant, along with Denodo and Microsoft.

In the Challengers Quadrant, Qlik, SAS, TIBCO Software, and Precisely provide a high ability to execute (as measured by the Y axis), while SnapLogic is the lone entry in the Visionaries Quadrant. In the Niche Players Quadrant, Matilliion, Fivetran, and HVR are joined by Adeptia, Safe Software, and CloverDX.

The vendors reacted to the Gartner report. “We are proud to be included in the Gartner Magic Quadrant and we plan to continue moving up and to the right in the Magic Quadrant,” stated Matillion CEO Matthew Scullion in a press release.

“As we continue to help companies on their journey toward healthier data, we believe this recognition is another important reflection of Talend’s commitment to innovation and our customers’ success,” stated Talend CTO Krishna Tammana.

“Our unique, agnostic platform is helping enterprises access, transform and deliver the real-time data essential to data-driven decision making, and is a key element in helping customers drive their modern analytics, cloud migration, data lake management, and data integration strategies,” stated James Fisher, Chief Product Officer for Qlik.

“I am thrilled to see Denodo recognized yet again in the Leaders’ Quadrant based on our Ability to Execute and Completeness of Vision,” said Ravi Shankar, the senior vice president and chief marketing officer at Denodo.

Fivetran CEO George Fraser pointed out this was the second straight year his company made the Magic Quadrant. “We’re proud to be recognized by the industry experts at Gartner,” he stated in a press release. “We believe this recognition proves our position as the leading cloud-native data integration provider helping automate enterprise data pipelines to decrease time to insight and improve data team impact.”

You can find a copy of the Gartner’s Magic Quadrant on Matillion’s website, on Talend’s website, on Qlik’s website, on CloverDX’s website, and on Fivetran’s website.

Related Items:

Data Fabrics Emerge to Soothe Cloud Data Management Nightmares

A Peek At the Future of Data Management, Courtesy of Gartner

Back to Basics: Big Data Management in the Hybrid, Multi-Cloud World