Data Automation Poised to Explode in Popularity, Ascend.io Says

(Ye-Liew/Shutterstock)

The amount of data automation deployed in the wild is quite small at the moment, but it’s set to grow significantly in the months to come as overworked data teams seek respites from grueling manual data tasks. That’s the key conclusion coming out of a new survey from Ascend.io today.

Only 3.5% of data professionals are using data automation today, but that figure is expected to grow to 85% over the next 12 months, according to Ascend.io’s third annual DataWare Pulse Survey, which is based on a survey of hundreds of data analysts, data engineers, data scientists, and chief data officers.

That delta stood out in a big, bold way to Ascend.io CEO and founder Sean Knapp, who delivered a keynote at the start of Ascend.io’s Data Automation Summit this morning.

“The sheer, striking discrepancy between those who have currently employed data automation and those who intend on using it I have never seen in any other domain,” Knapp. “And it highlights how much demand there is for automation.”

The DataWare Pulse survey also found that 95% data teams are at or over capacity, which restricts their ability to take on new data projects. When you consider that the number of data pipelines is expected to grow by 57% over the coming year, and that the demand for new data products is growing significant faster than the data teams who are tasked with building them, then you realize what a data-driven predicament that many companies have gotten themselves into.

One potential solution, of course, is data automation, which Knapp roughly defined as spanning various disciplines like ETL/ELT, DataOps, and data fabrics and meshes. Data automation will be a critical asset that overworked data professionals can lean on as the volume and complexity of data and data products grows in the future, he said.

(Source: Dataware Pulse Survey 2022)

“Today our data teams are underwater,” Knapp, who is a 2021 Datanami Person to Watch, said during his keynote. “And they are some of our highest impact, most valuable, most sought-after and hardest-to-retain talent.  Yet oftentimes we have them deep in the weeds, deep in the muck, reinventing the wheel over and over again, repeating things over and over again, stalled and encumbered with monotonous manual tasks.”

Data automation brings the potential to free those data professionals from working at the lower levels of the data stack, and instead leveraging their considerable talents to tackle bigger data challenges that bring a greater return on personnel and data investments, Knapp said.

“The productivity and overall benefit of automation is opening up some of the most exciting opportunities and use cases around the world,” he said. “We see a tremendous number of companies…automate first through orchestrion, then [go] deeper, taking what used to require dozens and dozens of weeks, and accomplishing the same feats in one or two weeks.”

(Source: DataWare Pulse Survey 2022)

Data meshes and data fabrics are emerging as potentially impactful data automation solutions. Because of the way data meshes and fabrics centralize some aspects of data management while allowing for storage and control to be decentralized, they are seen as key architectural components of data automation.

“One of the top priorities for teams is to start to invest in their data mesh strategy, to embrace the fact that their data exists in a very heterogenous ecosystem,” Knapp said. “They have teams with a very heterogenous set of needs, yet at the same time, they want that network effect of value unlocked for them.”

Data meshes and other data automation strategies will be important because they allow data teams to solve data management and governance issues at the metadata layer, rather than at the infrastructure layer, Knapp said.

“What we see a number of companies today investing in is multiple data architectures, multiple locations where you can store your data, where you can process your data, where you can push and publish your data to other systems that may be in different aft data infrastructure, maybe in other clouds…maybe entirely different contents,” Knapp said. “That’s one of the reasons why we’re really excited to be stepping into the next era of automating with Ascend.”

Data Automation Summit 2022 continues through April 14.

Related Items:

Data Mesh Vs. Data Fabric: Understanding the Differences

Meet Sean Knapp, a 2021 Datanami Person to Watch

Data Pipelines of a Higher Order