Prophecy 4.0 Offers Fully Governed Self-Service Data Prep for Databricks SQL

Source: Shutterstock

In 2024, data integration company Prophecy unveiled its AI Copilot for Databricks, marking a significant step forward in data transformation. Leveraging the power of GenAI, the tool enabled business users and analysts to construct data pipelines using natural language. It aimed to simplify workflows and offer organizations a more efficient path to managing and utilizing their data resources effectively. 

Building on that foundation, Prophecy has now introduced Prophecy 4.0, the latest iteration of its AI-driven data integration platform optimized for Databricks SQL. While last year’s copilot was all about enabling AI-powered pipeline creation, the new release is more focused on adding full self-service capabilities without breaking IT governance rules. 

The relationship between Prophecy and Databricks is rooted in a shared goal of enhancing data usability across organizations. Prophecy builds on Databricks’ data infrastructure by offering tools designed to simplify data preparation and transformation for a broader range of users.

“Organizations have put their most valuable data assets into Databricks, and Prophecy 4.0 makes it easier than ever to make that data available to analysts,” said Roger Murff, VP of Technology Partners at Databricks. “And because Prophecy is natively integrated with the Databricks Data Intelligence Platform, platform teams get centralized visibility and control over user access, compute costs, and more.”

The gap between analysts and data teams often creates challenges, especially for larger organizations. While analysts need reliable data for AI and strategic decision-making, central teams focus on governance and security. This disconnect or misalignment in priorities often results in inefficiencies, delays, and missed opportunities to fully leverage data’s potential.

A recent Wakefield Research survey revealed that 50% of data executives struggle to hire data engineers, while 53% face difficulties scaling their teams to meet demand. Despite ongoing efforts to recruit and expand, these challenges remain a significant bottleneck.

Prophecy referenced this survey’s findings to show the need for tools that can ease the workload of data teams, and that’s exactly what it aims to do with the latest release. Giving analysts the ability to prepare data on their own helps reduce the dependency and burden on data engineers and improves workflow efficiency for organizations.

One of the key updates to the platform is that it supports secure connections to widely used sources like SFTP, SharePoint, Salesforce, and desktop files such as Excel and CSV. By eliminating the need for custom scripts or manual transfers, this functionality not only saves time but also minimizes potential errors. 

The updated platform is also designed to optimize the “last mile” of data workflows. This refers to tasks such as integration of results into visualization tools like Tableau, and sending automated notifications to stakeholders via email. 

Additionally, Prophecy has introduced built-in automation with a drag-and-drop interface. This is designed to enable analysts to design, run, and validate workflows easily without relying on technical expertise or additional tools.

According to Stewart Bond, Vice President of Data Intelligence and Integration Software at IDC, the complexity of modern data environments, characterized by their distributed and dynamic nature, poses significant challenges for enterprises. He emphasized that as organizations increasingly aim to allow non-technical users access to corporate data, solutions like Prophecy 4.0 could become more vital. 

While rolling out the upgrades to the platform, Prophecy ensured its prioritized governance. The platform features simplified version control, letting analysts and engineers work together smoothly to refine and deploy pipelines. It also includes data profiles that give users insights into things like data distribution and completeness. 

Prophecy has added reusable packages to make it easier for users to create workflows that meet predefined compliance standards. This feature could also be useful in helping observability teams identify and resolve issues early. 

Raj Bains, CEO of Prophecy, emphasized that inefficiencies caused by disconnected workflows between data teams are a major pain point. These disjointed processes often have a negative impact on productivity and waste valuable resources.

“We’ve visited companies where analysts would outline data workflows in their data prep tools and then engineers downstream would recode the entire pipeline from scratch with their ETL software,” said Bains. “It was a huge waste of time and energy. With Prophecy 4.0, everything is done once. Analysts can design and publish pipelines whenever they want, with security, performance, and data access standards predefined by IT.” 

By combining self-service data preparation and governance-first capabilities, Prophecy 4.0 has the potential to stand out as a valuable tool for enterprises looking to refine their data workflows. 

However, its success will depend on overcoming challenges such as ensuring compatibility with legacy systems and meeting diverse compliance requirements across industries. The recent $47M Series B funding should help provide the company with the resources needed to enhance the platform further.

Related Items 

AI Making Data Analyst Job More Strategic, Alteryx Says

Reality Check for GenAI: Deloitte Finds Enthusiasm Tempered by Adoption Hurdles

Grafana’s Annual Report Uncovers Key Insights into the Future of Observability