Monitoring AI applications with AI
Monitoring AI applications with AI


The best performing offline algorithm can lose in production. The most accurate model does not always improve business metrics. Environment misconfiguration or upstream data pipeline inconsistency can silently kill the model performance. Neither prodops, data science or engineering teams are skilled to detect, monitor and debug such types of incidents.
Was it possible for Microsoft to test Tay chatbot in advance and then monitor and adjust it continuously in production to prevent its unexpected behaviour? Real mission critical AI systems require advanced monitoring and testing ecosystem which enables continuous and reliable delivery of machine learning models and data pipelines into production. Common production incidents include:
- Data drifts, new data, wrong features
- Vulnerability issues, malicious users
- Concept drifts
- Model Degradation
- Biased Training set, training issue
- Performance issue
- Deployment, configuration issues

In this demo based talk we discuss a solution, tooling and architecture that allows machine learning engineer to be involved in delivery phase and take ownership over deployment and monitoring of machine learning pipelines.
It allows data scientists to safely deploy early results as end-to-end AI applications in a self serve mode without assistance from engineering and operations teams. It shifts experimentation and even training phases from offline datasets to live production and closes a feedback loop between research and production.

Technical part of the talk will cover the following topics:
- Automatic Data Profiling
- Anomaly Detection
- Clustering of inputs and outputs of the model
- A/B Testing
- Service Mesh, Envoy Proxy, trafic shadowing
- Kafka
- Deployment clusters: AWS SageMaker, ML Lambda
- Stateless and stateful models
- Monitoring of regression, classification and prediction models


Stepan Pushkarev is a CTO of His background is in engineering of data platforms. He spent last couple of years building continuous delivery and monitoring tools for machine learning applications as well as designing streaming data platforms. He works closely with data scientists to make them productive and successful in their daily operations.

Open Data Science




Open Data Science
One Broadway
Cambridge, MA 02142

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Consent to display content from - Youtube
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google