‘Dark Data’ Continues to Stymie Analytics Efforts

Not only is there way too much data flying around organizations, but those charged with keeping track of it often don’t know where critical data is located. What’s more, new European data privacy rules are making that task harder.

Those the key findings of an industry survey released earlier this month by in-memory analytics database vendor Exasol, which concludes that so-called “dark data” remains a challenge as businesses transition from business intelligence to data analytics.

The European study looked at how companies in Germany and the U.K. are faring in the transition to data analytics. Among the wild cards in deploying data analytics are new consumer privacy regulations such as the General Data Protection Regulation (GDPR), the vendor survey found.

Three-quarters of respondents said they are moving from business intelligence to analytics, but face data migration issues. The majority said they are at least a quarter of the way toward completing the shift from business intelligence platforms.

The primary incentive for doing so was the promise of batter data quality and availability, but companies reported difficulty in aggregating big data and reconciling data sources required to build the advanced data sets needed for analytics platforms. More than half (55 percent) said data fragmentation across multiple databases is slowing progress.

“Organizations are getting their hands on better data and they see the possibilities, but the data analytics they seek are often out of reach because that data is residing in departmental databases or mounting up in data lakes,” said Exasol CTO Mathias Golombek.

“They are racing to get to the next stage—becoming more practical and applying that data to business decision making—but most data science teams don’t have the data infrastructure they need to surface that dark data and make data analytics available on demand,” Golombek added.

Hence, only 1 percent of respondents said they consider themselves to be “data-driven” organizations.

The primary reason cited for early project failures are data security and privacy issues, particularly uncertainty about how to implement EU data privacy rules. Among the issues faced by European companies is how to meet GDPR “Privacy by Design” provisions that require privacy to be “baked in” by default at every step in the analytical life cycle—from data discovery and visualization to data preparation and deployment.

“To achieve this, organizations who are looking to use personal data as a source of insight and competitive differentiator will need a complete and mature analytics platform,” Olivier Penel, a data management specialist with analytics software vendor SAS told Datanami in July.

Meanwhile, Exasol reported that 40 percent of survey respondents said they are looking at AI and machine learning tools to help address data analytics challenges. Nevertheless, most said they still lack the data volumes and quality needed to leverage those automation tools.

Recent items:

Will Privacy Kill Innovation?

The Enlightening Side of GDPR Compliance