Insights From Ascend’s Annual DataAware Pulse Survey

Since its launch four years ago, the Annual DataAware Pulse Survey has established itself as the leading industry benchmark survey. It provides key insights into current industry best practices, current sentiment around trends, and priorities of data teams. 

The 2023 Annual DataAware Pulse Survey provides an opportunity for anyone who relies on data within their organization to understand the current market conditions. The results of the fourth annual survey shed some light on the impact of rapidly growing data volume and complexity on data teams. 

Overburdened Data Teams 

While the number of data tools is increasing, there is a decline in workload capacity and productivity, and this is a result of excessive manual work. According to the latest DataAware Pulse Survey, data engineers are working at capacity and are now looking at automation technologies to get some much-needed relief. 

The biggest inhibitors to productivity include hiring constraints (60 percent), lack of automation (55 percent), and limitation of tools (46 percent). Data engineers bear the burden of increased workload as they are 27 percent more likely to report a severe gap between team growth and company demands. 

A key reason why data engineers are overburdened is that they spend too much time on maintenance. According to the survey results, the majority of data engineers spend 50 percent or more of their time on maintaining existing programs. 

Sean Knapp, Founder and CEO of Ascend, and one of Datanami’s People to Watch 2021, shared his take on the state of affairs, “It’s been a tough year for data teams. Many were planning to hire their way out of the crunch back in mid-2022 and have been thrown for a loop. As they grappled with how to do more with less this year, we can see that initial productivity gains from last year are starting to erode as workloads shift to maintenance. But don’t give up hope yet. There are still ways for teams to take control of their workloads through increasing levels of automation.”

(Monkey Business Images/Shutterstock)

Executives And Engineers Are Split in Their Approach

The DataAware Survey highlights the misalignment of KPIs across the board. The executives are 3 times more likely to rank presentations created as their top impact measures compared to individual contributors, who rank errors fixed and tickets closed as being more important.  There is disagreement over major initiatives, with individual contributors five times more likely to deny having any plans to implement data mesh or data fabric compared with executives, who report plans to implement one within the next year. 

Individual contributors are up to twice as likely to favor cutting tools out of their data stack. On the other hand, executives are up to 36 percent more likely to favor adding more tools to the stack. In addition, executives are 1.9 times more likely to feel their teams should consolidate data sources used in their analytics to save money. The rest of the technical team favors pipeline optimizations to drive down costs.

Sean Knapp believes that managers and directors hold the key to bridging the gap between data engineers and executives. They can help the teams understand the overall business objectives while helping executives understand where to invest and how they can facilitate the teams. 

Enterprise Organizations Are Turning To New Technologies in a Post-Modern Data Stack

Around 69 percent of respondents are already using or plan to implement a data mesh or fabric in the next 12 months. The change in momentum during 2022 is highlighted by a 12 percent increase in the number of reported plans to implement mesh/fabric in the next 12 months.

While individual contributors are 2.1 times more likely to favor cutting tools out of their data stack, executives are 16 percent more likely to add more tools to the stack. Team leads are more in favor of consolidating platforms. 

(3rdtimeluckystudio/Shutterstock)

Use of Generative AI 

The survey highlights the top use cases for generative AI are 50 percent in automation, 45 percent in code generation, and 42 percent in documentation. Large companies, with more than 500 employees, are 80 percent more likely to have plans for implementing generative AI tools based on internal data. 

Sean Knapp expects the AI trend to evolve in the months as organizations become more familiar with the technology. He believes that it is important for executives to do more to inspire their teams with the potential of this new technology. AI has enormous potential to provide contributors with more autonomy and job satisfaction in their roles. 

Related Items 

Data Automation Poised to Explode in Popularity, Ascend.io Says

KNIME Releases a State of Data Science and Machine Learning Survey

Ascend.io Survey: Automation Seen as Key Solution for Overburdened Data Teams