Monte Carlo Announces New GenAI Capabilities to Streamline Data Quality Management at Enterprise Scale

SAN FRANCISCO, Nov. 15, 2024 — Monte Carlo, the data and AI observability company, has announced a series of product enhancements and new capabilities at its annual IMPACT Data Observability Summit, aimed at helping enterprises deliver reliable, trustworthy data to power their data and AI products.

Among the enhancements to its data observability platform is the introduction of GenAI Monitor Recommendations, a first-of-its-kind capability that allows data teams to more easily and quickly deploy data quality rules. This is the first time a data observability platform uses generative AI to understand and monitor data relationships within an asset.

Additionally, to help these and other data and AI reliability initiatives scale and operationalize more effectively, Monte Carlo introduced the Data Operations Dashboard, which gives insights into key operational metrics like number of incidents by data asset owner, time to detection and time to resolution.

Finally, to provide true end-to-end observability into all issues that could take place in an organization’s data, systems, and code, the company unveiled new integrations for Microsoft Fabric, Databricks Workflows, and Informatica, to give users much-needed visibility into these three widely-used data pipelines.

Introducing: GenAI Monitor Recommendations

Headlining this week’s announcements are GenAI Monitor Recommendations, a new native capability within Monte Carlo’s industry-leading data and AI observability platform that provides users with recommendations for data quality monitors using an LLM. This new capability allows technical and non-technical data practitioners alike to quickly and easily deploy meaningful, relevant, and powerful data quality monitors for their most critical data and AI products.

Powered by Monte Carlo’s data profiling engine Data Profiler, GenAI Monitor Recommendations uses a generative AI model to determine relationships between columns and suggest data quality rules. Monte Carlo feeds the LLM with sample data to analyze statistical relationships, query log data to analyze usage patterns, and other table metadata to build a deeper contextual understanding of the asset.

GenAI Monitor Recommendations detect patterns in the data that are otherwise difficult if not impossible to discover using traditional AI and machine learning methods, and then suggest monitors and rules to alert when there are anomalies in those patterns that may indicate poor data quality.

For example, a data analyst at a professional baseball team may use this new capability to quickly spin up data quality rules for a critical table labeled ‘pitch_history,’ in which Monte Carlo will identify relationships between the column ‘pitch_type,’ (fastball, curveball, etc.), and pitch speed. Monte Carlo would then automatically recommend data quality rules that make sense based on the history of the relationship between those two columns, such as ‘fastball’ should have pitch speeds of greater than 80mph.

This week’s announcement comes in the midst of a data industry that is yearning for answers to challenges related to ensuring that the data powering mission-critical data and AI products is trustworthy, reliable, and of high quality. A recent survey from Monte Carlo and Wakefield Research revealed that while 100% of data leaders surveyed are building generative AI products, 2 out of 3 of them (67 percent) are not confident that their company’s data is of high enough quality to ensure their reliability.

“Practitioners in the data and AI space have more on their plate than ever before. They need solutions that will allow them to do more to ensure the quality and reliability of their data, without actually having to do more work,” said Barr Moses, co-founder and CEO of Monte Carlo. “At Monte Carlo, we build capabilities that our customers need and tell us they will use to make their lives easier and their work more impactful. We saw an opportunity to combine a real customer need with new and exciting generative AI technology, to provide a way for them to quickly build, deploy, and operationalize data quality rules that will ultimately bolster the reliability of their most important data and AI products.”

This announcement is the first in a series of enhancements to the data observability platform’s suite of monitoring recommendation tools to be powered by generative AI.

Operationalizing Data Quality with a Single Source of Truth

As enterprises look for ways to better operationalize data quality and make it a P0 priority across organizations, the need for tools to measure and fine-tune that operational performance continues to grow. To help data teams sharpen the way they measure this effort, Monte Carlo introduced the Data Operations Dashboard to allow users to easily track and report the progress of their data quality initiatives.

Data Operations Dashboard provides insights into critical operational metrics from a given time frame, and provides key insights that contribute to decisions on improvements. Among the metrics the dashboard tracks include response time for monitor alerts, incident resolution time, and the total number of incidents (classified by severity) and the incident owners.

With this new functionality, data teams have better ability to track and report, and ultimately improve these two critical functions, toward the continued north start of improving their overall data health.

Learn more about the Data Operations Dashboard via Monte Carlo’s docs.

End-to-End Observability Across Your Data Systems with Microsoft Fabric, Informatica, and Databricks Workflows Integrations

Data teams are constantly dealing with new and increasingly complex data pipelines. In the same 2024 survey from Monte Carlo and Wakefield research, 85% of data leaders indicated they had added a significant amount of new data sources, resulting in new pipelines for their teams to manage, ensure their reliability, and allow the data to seamlessly flow downstream.

To that end, Monte Carlo also unveiled at IMPACT 2024 new integrations with three widely-used ETL pipelines across enterprises: Microsoft Fabric (starting with Azure Data Factory), Informatica, and Databricks Workflows. With these new integrations, data teams can now get full visibility into how each Microsoft Fabric, Informatica or Databricks Workflows job interacts with downstream assets such as tables, dashboards, and reports.

“For many organizations, pipeline and system failures are a daily fact of life,” said Lior Gavish, co-founder and CTO of Monte Carlo. “Traditional approaches to data quality focus exclusively on detecting that data is broken, not understanding the why, which often goes beyond the data itself. Data observability is the only approach that monitors data, systems, and code to both detect and resolve data quality issues at scale and across your entire data ecosystem.”

Learn more about these and Monte Carlo’s end-to-end suite of integrations via Monte Carlo’s docs.

IMPACT 2024: Driving Data + AI Trust at Scale

These announcements were made at Monte Carlo’s fourth-annual IMPACT Data Observability Summit. In addition to being the world’s only data observability conference, this year’s IMPACT Summit brought together some of the foremost experts on the topic of reliable data and AI, including:

• Allan Lichtman, creator of the 13 Keys to the White House
• Allyson Felix, 7-time Olympic gold medalist
• Amir Netz, CTO, Microsoft Fabric
• Pierre A. Fischer, Product Line Lead, Data Platforms, Roche
• Iwao Fusillo, Global Head of Data and Analytics, Digital Commerce, PepsiCo
• Surekha Durvasula, Former VP of Data & Analytics, Walgreens Boots Alliance
• Stefanie Tignor, PhD, Head of Product Data Science and Engineering, Grammarly
• Sri Subramanian, Director of Data Engineering, SurveyMonkey
• Tomasz Tunguz, General Partner at Theory Ventures
• Philip Zelitchenko, VP, Data & Analytics, ZoomInfo
• And more!

To learn more about Monte Carlo’s vision for enterprise-ready AI and data observability, visit www.montecarlodata.com or request a demo.

About Monte Carlo

As businesses increasingly rely on data to drive better decision making and power digital products, it’s mission-critical that this data is trustworthy and reliable. Monte Carlo, the data observability company, solves the costly problem of broken data through their fully automated, SOC-2 certified data observability platform. Billed by Forbes as the New Relic for data teams and backed by Accel, Redpoint Ventures, GGV Capital, ICONIQ Growth, and IVP, Monte Carlo empowers companies to trust their data.

Source: Monte Carlo