Abstract: 

As Artificial Intelligence is becoming part of user-facing applications and directly impacting society, deploying AI reliably and responsibly has become a priority for Microsoft and several other industry leaders. Rigorous model evaluation and debugging are at the heart of responsible machine learning development. Yet, many of the standard practices continue to focus on high-level aggregated evaluation that uses only single accuracy numbers to report model accuracy on large benchmarks. Such practices can be misleading because they hide important failure modes that happen either for unexpected input conditions, corner cases, or specific demographic groups. At the same time, understanding where hidden pockets of errors lie in large data manifolds can be tedious and time consuming for practitioners. This presentation will introduce Error Analysis to the audience, as a tool and methodology for effectively identifying and diagnosing errors in machine learning models, beyond aggregated accuracy scores. The tool provides different views for quick error identification and enables error diagnosis either via active data exploration or model explanations generated using the InterpretML library. We will deep dive into the tool functionalities by presenting case studies and a live step-by-step demo. Finally, we will conclude with a discussion on future opportunities we are considering on further integrations with other RAI tools, as a quest towards a better integrated RAI ecosystem.

Website: erroranalysis.ai
Github repository: github.com/microsoft/responsible-ai-widgets
Practitioner-oriented blog on error analysis: https://techcommunity.microsoft.com/t5/azure-ai/responsible-machine-learning-with-error-analysis/ba-p/2141774

Bio: 

Mehrnoosh Sameki is a senior technical program manager at Microsoft, responsible for leading the product efforts on machine learning interpretability and fairness within the Azure Machine Learning platform. She earned her PhD degree in computer science at Boston University, where she currently serves as an adjunct assistant professor and lecturer, offering courses in responsible AI. Previously, she was a data scientist in the retail space, incorporating data science and machine learning to enhance customers’ personalized shopping experiences.