Enhance Trust with Machine Learning Model Error Analysis


A.I. has a trust issue. Part of it has to do because models aren't appropriately de-risked but also another part has to do with communication. It is essential to communicate the model's strengths and weaknesses to stakeholders to manage expectations. More importantly, weaknesses include potential threats, trends, constraints, and uncertainties ranging from fairness to consistency concerns. Unfortunately, the standard predictive performance metrics don't capture all the nuance in these concepts, and they are usually tough to communicate to non-technical stakeholders.

In this session, we will leverage Microsoft's Error Analysis tool to explore different ways of assessing model errors using a traffic prediction problem.

*I use (https://erroranalysis.ai/) and Interpret ML (https://interpret.ml/) tools in my job with similar problems in agriculture (which I cannot disclose). I also use IBM's Uncertainty Quantification 360, which is very useful for the understanding of model uncertainty.

Session Outline:

Lesson 1: XAI and Model Error Analysis and Uncertainty
Short presentation about the importance of Error analysis and uncertainty estimation and what they have to do with XAI
Lesson 2: Hands-On Error analysis Workflow
Leveraging Microsoft's Error Analysis tool to figure out what are the weaknesses of the model.
Lesson 3: Hands-On Uncertainty estimation Workflow
Using IBM's Uncertainty Quantification 360 to understand what inputs create the most uncertainty in the model

Background Knowledge:

Intermediate skills with Python and at least basic understanding of machine learning


Serg Masís has been at the confluence of the internet, application development, and analytics for the last two decades. Currently, he's a Climate and Agronomic Data Scientist at Syngenta, a leading agribusiness company with a mission to improve global food security. Before that role, he co-founded a search engine startup, incubated by Harvard Innovation Labs, that combined the power of cloud computing and machine learning with principles in decision-making science to expose users to new places and events efficiently. Whether it pertains to leisure activities, plant diseases, or customer lifetime value, Serg is passionate about providing the often-missing link between data and decision-making. He wrote the bestselling book "Interpretable Machine Learning with Python" and is currently working on a new book titled "DIY AI" for Addison-Wesley for a broader audience of curious developers, makers, and hackers.

Open Data Science




Open Data Science
One Broadway
Cambridge, MA 02142

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Consent to display content from - Youtube
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google