A Systematic Approach for Building Full-Spectrum Model Monitoring

Abstract: 

At many companies ML models make high stakes decisions each day which makes it critical to monitor models to prevent poor decisions. However, there are lots of technical and organizational challenges in doing so effectively at scale. In this talk we’ll discuss a systematic framework to build and roll-out full-spectrum Model Monitoring for identifying and preventing problems with models. We’ll do a deep dive into Lyft’s model monitoring architecture (which includes real-time feature validation, performance drift detection, anomaly detection, and model score monitoring), how we leveraged open source, and the cultural change needed to get data scientists to effectively monitor their models. We’ll also discuss why we decided to build vs buy, our wins and learnings, and why monitoring in itself may not be sufficient for preventing model degradation.

Bio: 

Mihir Mathur is the lead Product Manager for Machine Learning at Lyft, where he works on building ML and AI tools that power automated intelligent decisions across realtime pricing, ETAs, fraud detection, safety classification etc. In the past Mihir has worked on building delightful products for millions of users at Quora, Houzz, and Thomson Reuters. Mihir graduated magna cum laude from UCLA with a Bachelor’s and Master’s in Computer Science.

Open Data Science

 

 

 

Open Data Science
One Broadway
Cambridge, MA 02142
info@odsc.com

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Youtube
Consent to display content from Youtube
Vimeo
Consent to display content from Vimeo
Google Maps
Consent to display content from Google