AI in Turbulent Times

In the face of the unprecedented events brought on by COVID-19, one of the key concerns among many data scientists is that their current models could be generating inaccurate or misleading predictions. In DataRobot’s webinar, AI in Turbulent Times: Navigating Changing Conditions, data scientists outlined the steps that data scientists can take to incorporate robustness into their model building processes.

With critical supply chain management functions in danger, organizations need their machine learning models to operate accurately and efficiently. Yet, now more than ever, many data scientists don’t feel they can trust their models. In fact, a poll taken at the beginning of the webinar about the confidence level of AI models in production was telling.  For those with models in production:

• 5% said they were certain some AI models are/will be producing bad predictions
• 5% said they suspect their AI models were producing bad predictions, but were unsure
• 17% said they have no idea how their AI models are performing

Because machine learning models are based on historic data to predict future events, the COVID-19 pandemic presents a unique challenge for data scientists because there are no similar events for comparison that can inform competent predictions.

Take a Step Back

A starting point for data scientists is to take a big step back and ask how their models could be impacted. Data science teams should also consider which actions the models are impacting and how operations could change as a result. Travel, retail, and oil prices, for example, have all changed significantly in the past few weeks alone, so it falls to data science teams and their organizations to assess which features are most crucial to their models.

Understand the Changes

After data science teams have evaluated and assessed which features have the greatest impact on their models, they can then consider how evaluating these feature changes will impact their operations. Once these changes are in place, organizations can get a more realistic understanding of where their new business priorities should be as they contend with constantly changing circumstances.

Predicting missed appointments for medical outpatient visits isn’t as important right now. On the other hand, building models to appropriately staff hospitals at the department level is currently vital.

Have a Plan

Amid constantly shifting developments, data science teams need to do more than just monitor models. They also have to monitor changes to the models’ input distributions to consider how data drift could be impacting findings and understand how these changes affect their overall accuracy.

Organizations should develop a monitoring strategy that is individualized for different models and allows for setting data drift or accuracy thresholds. Without this approach, model monitoring becomes like watching water boil and ineffective.

The COVID-19 pandemic has resulted in significant disruptions to communities worldwide. Understanding how to adjust models to new developments and prepare for even further disruptions will be essential to navigating the uncertain landscape ahead.

Listen to our on-demand webinar AI in Turbulent Times: Navigating Changes Conditions, to learn more.