AI Bias a Real Concern in Business, Survey Says

(Aleutie/Shutterstock)

The potential benefits of AI for businesses are huge. But those benefits could be eroded by hidden biases that damage brand reputation and customer trust, according to a survey of US and UK businesses commissioned by DataRobot.

According to DataRobot‘s “The State of AI Bias in 2019,” which was released last week, 42% of organizations surveyed reported being “very to extremely” concerned about AI bias occurring in their organizations.

This number jibes with another finding from the DataRobot survey: that 38% of the organizations surveyed reported they use “black box” machine learning systems that offer no insight into how it makes decisions.

The juxtaposition of AI bias concerns and black box systems is enough to warrant serious questions about the direction compaines should take with their machine learning, according to John Giannandrea, Apple’s senior vice president of machine learning and AI strategy.

“If someone is trying to sell you a black box system… and you don’t know how it works or what data was used to train it, then I wouldn’t trust it,” DataRobot quotes Giannandrea as saying in its report.

The survey indicates that organizations are aware of the potential pitfalls and are actively working to mitigate it. DataRobot found that 64% of survey respondents say they’re “very to extremely” confident in their ability to identify AI bias.

The survey found some interesting differences among US and UK companies. While UK companies were more concerned about “technical bias” in the ML algorithms than their US counterparts (47% compared to 34%, respectively), the survey suggests that US companies were more concerned about “emergent bias” than UK firms (35% versus 22%, respectively).

The survey defines technical bias as bias “arising from technical limitations, (e.g., algorithms that perpetuate bias.” Emergent bias, meanwhile, is bias “arising in the context of using the system when there’s misalignment between the user and system design, (e.g., only seeing content that confirms an already-held belief and not seeing content that conflicts).” The survey also looks at human bias.

How organizations are mitigating AI bias today (Source: DataRobot)

DataRobot had a theory for the discrepancy between the bias concerns of UK and US firms. “It’s likely that because UK companies operate under tighter regulatory climates, they are more cautious about the technical aspects of AI,” the company says.

Compromised brand reputation and loss of customer trust was the number one concern as it relates to AI bias, the survey found. But the survey found other concerns, including mismatch with personal ethics, loss of employee trust, and legal penalties.

The report also found that the majority of AI systems in use (69%) are developed by third-parties. About 70% of organizations are using AI to automate fewer than 20 business functions, while 19% are using it to automate 49 or fewer, and 10% are automating more than 50 business functions.

Overall, the survey shows there are real concerns about the algorithms that companies are deploying, with trust being a major factor in whether these systems will be accepted.

“More organizations are deploying AI as they recognize the technology as a critical success factor for competing in today’s business climate,” Ted Kwartler, vice president of Trusted AI at DataRobot, said in a press release. “Despite this fact, we’ve observed that AI maturity varies widely — with many organizations still using untrustworthy AI systems.”

Related Items:

Three Ways Biased Data Can Ruin Your ML Models

Why Relying on Data Hygienists Is Not a Realistic Way to Ensure Bias-Free AI

Don’t Be a Big Data Snooper