Will GenAI Modernize Data Engineering?

(hmorena/Shutterstock)

Generative AI is being asked to do many things, from replacing call center workers to enhancing developer productivity. There’s also an intriguing possibility that it will supercharge data engineering, if a new survey from Prophecy is correct.

The new Prophecy survey, dubbed “The Impact of GenAI on Data Teams,” paints a promising picture of GenAI use among data engineers and others tasked with the day-to-day management of data in the enterprise.

Perhaps the most surprising findings from the survey–which was conducted by Wakefield Research and included 500 senior-level data and analytics execs at companies with minimum revenue of $1 billion–are that data teams using GenAI are seeing productivity gains far ahead of other GenAI uses, and that they’re going into full production at a faster rate.

First, the productivity gains. Prophecy found:

Among the data teams that have already implemented GenAI, 41% are seeing productivity growth of 15% to 30% for overall data delivery. Another 46% are reporting productivity gains of 31%-50%. And 12% say that they’re seeing a productivity boost of more than 50%. No survey participants said they are realizing any productivity gains, while 1% said the gains are less than 15%.

Remarkably, the productivity gains of data teams that have already implemented GenAI are even higher than the expected gains of the data teams who have not yet implemented it. That statement is true in both the 31%-50% and 50%< productivity gain cohorts.

Source: Prophecy report “The Impact of GenAI on Data Teams”

Compared to the average GenAI productivity gain of 25%–a figure that Prophecy pulled from a 2024 Goldman Sachs Research study–the survey findings suggest that data engineering may be a sweet spot for GenAI.

Next, GenAI implementation. Prophecy found:

Every organization surveyed said they are either planning to use GenAI or already using it, while 64% say they are already using GenAI. The survey found that 23% say their GenAI efforts are already “fully scaled,” with 21% in limited deployment and 33% in the pilot or proof of concept (POC) phase. The most common uses are automatic data curation (used by 58% of the GenAI-using cohort), followed by conversational analytics (51%), data tests and quality (51%), on speeding data access with text-to-SQL (46%), and writing documentation (36%).

Executive enthusiasm over GenAI is high, with 25% of execs saying they’ll back any project  with AI as a core component. The vast middle are also optimistic but more cautious about GenAI in data engineering, while only 3% don’t see AI playing a role in data infrastructure.

Looking ahead, data leaders are investing in GenAI with the several goals in mind, including to make existing data teams more efficient (reported by 58% of survey-takers) and to upskill existing teams (50%). More than half (53%) of survey respondents reported difficulty in finding and hiring highly skilled data engineers, while half also said they struggle at onboarding new data engineers.

GenAI figures to play a central role in addressing both of these trends. Companies will be looking to hire data engineers who are proficient at GenAI. They will also be looking to use GenAI to make existing data engineers more efficient. As for whether GenAI will be autonomous or assistive, there was an even split, with 51% saying GenAI is assistive in nature and 49% saying its’ autonomous.

The Prophecy survey mirrors other findings, including one report that came out of Alteryx yesterday. Alteryx found that AI tools are not only making data analyst jobs easier in some ways, but AI is also elevating the importance of the analyst’s role in the company. Thanks to AI, data analysts are able to adapt faster to changing job demands, a change which many said was “profound.”

Another report last fall from ThoughtSpot hinted at possible big returns from GenAI in business intelligence. Half of early GenAI adopters predicted a three-year return-on-investment of more than 100%. Cindi Howson, the company’s chief data strategy officer, told BigDATAwire that GenAI was “going to take away the doldrums and the silly work” for dashboard developers.

Using GenAI for data engineering may be destiny. The quality and availability of data has been one of the biggest inhibitors of AI projects of all types. A 2024 report from Deloitte found that 55% of companies attempting to build GenAI are running into data-related issues. The fact that GenAI could turbocharge the work of data engineers and data analysts may result in better data for downstream GenAI uses.

Prophecy CEO Raj Bains says there’s a secular shift underway in the industry when it comes to GenAI.

“GenAI has clearly landed in data teams and organizations are already getting value,” he said in a press release. “Over the next few years, GenAI will transform data operations top to bottom–from data management to data transformation to data integration, and everything in between.”

Prophecy develops a GenAI-powered data copilot that helps to automate many data engineer tasks, such as building data pipelines. The company raised $47 million last month in a Series B round, bringing its total funding to $159.2 million. Previous funding rounds included investments from Databricks, which is one of the platforms its SaaS tools run on.

Related Items:

AI Making Data Analyst Job More Strategic, Alteryx Says

Early GenAI Adopters Seeing Big Returns for Analytics, Study Says

Reality Check for GenAI: Deloitte Finds Enthusiasm Tempered by Adoption Hurdles