Error Analysis for Accelerating Responsible Machine Learning
Error Analysis for Accelerating Responsible Machine Learning


As Artificial Intelligence is becoming part of user-facing applications and directly impacting society, deploying AI reliably and responsibly has become a priority for Microsoft and several other industry leaders. Rigorous model evaluation and debugging are at the heart of responsible machine learning development. Yet, many of the standard practices continue to focus on high-level aggregated evaluation that uses only single accuracy numbers to report model accuracy on large benchmarks. Such practices can be misleading because they hide important failure modes that happen either for unexpected input conditions, corner cases, or specific demographic groups. At the same time, understanding where hidden pockets of errors lie in large data manifolds can be tedious and time consuming for practitioners. This presentation will introduce Error Analysis to the audience, as a tool and methodology for effectively identifying and diagnosing errors in machine learning models, beyond aggregated accuracy scores. The tool provides different views for quick error identification and enables error diagnosis either via active data exploration or model explanations generated using the InterpretML library. We will deep dive into the tool functionalities by presenting case studies and a live step-by-step demo. Finally, we will conclude with a discussion on future opportunities we are considering on further integrations with other RAI tools, as a quest towards a better integrated RAI ecosystem.

Github repository:
Practitioner-oriented blog on error analysis:


Besmira Nushi is a researcher in the Adaptive Systems and Interaction group at Microsoft Research. Her interests lie at the intersection of human and machine intelligence focusing on Reliable Machine Learning and Human-AI Collaboration. In the last five years, she has made practical and scientific contributions on implementing and deploying Responsible AI tools for debugging and troubleshooting ML systems. Prior to Microsoft, Besmira completed her doctoral studies at ETH Zurich in 2016 on optimizing data collection processes for Machine Learning.

Open Data Science




Open Data Science
One Broadway
Cambridge, MA 02142

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Consent to display content from - Youtube
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google