Abstract: At many companies ML models make high stakes decisions each day which makes it critical to monitor models to prevent poor decisions. However, there are lots of technical and organizational challenges in doing so effectively at scale. In this talk we’ll discuss a systematic framework to build and roll-out full-spectrum Model Monitoring for identifying and preventing problems with models. We’ll do a deep dive into Lyft’s model monitoring architecture (which includes real-time feature validation, performance drift detection, anomaly detection, and model score monitoring), how we leveraged open source, and the cultural change needed to get data scientists to effectively monitor their models. We’ll also discuss why we decided to build vs buy, our wins and learnings, and why monitoring in itself may not be sufficient for preventing model degradation.
Bio: Mihir Mathur is the lead Product Manager for Machine Learning at Lyft, where he works on building ML and AI tools that power automated intelligent decisions across realtime pricing, ETAs, fraud detection, safety classification etc. In the past Mihir has worked on building delightful products for millions of users at Quora, Houzz, and Thomson Reuters. Mihir graduated magna cum laude from UCLA with a Bachelor’s and Master’s in Computer Science.
