Abstract: 

The best performing offline algorithm can lose in production. The most accurate model does not always improve business metrics. Environment misconfiguration or upstream data pipeline inconsistency can silently kill the model performance. Neither prodops, data science or engineering teams are skilled to detect, monitor and debug such types of incidents.
Was it possible for Microsoft to test Tay chatbot in advance and then monitor and adjust it continuously in production to prevent its unexpected behaviour? Real mission critical AI systems require advanced monitoring and testing ecosystem which enables continuous and reliable delivery of machine learning models and data pipelines into production. Common production incidents include:
- Data drifts, new data, wrong features
- Vulnerability issues, malicious users
- Concept drifts
- Model Degradation
- Biased Training set, training issue
- Performance issue
- Deployment, configuration issues

In this demo based talk we discuss a solution, tooling and architecture that allows machine learning engineer to be involved in delivery phase and take ownership over deployment and monitoring of machine learning pipelines.
It allows data scientists to safely deploy early results as end-to-end AI applications in a self serve mode without assistance from engineering and operations teams. It shifts experimentation and even training phases from offline datasets to live production and closes a feedback loop between research and production.

Technical part of the talk will cover the following topics:
- Automatic Data Profiling
- Anomaly Detection
- Clustering of inputs and outputs of the model
- A/B Testing
- Service Mesh, Envoy Proxy, trafic shadowing
- Kafka
- Deployment clusters: AWS SageMaker, Hydrosphere.io ML Lambda
- Stateless and stateful models
- Monitoring of regression, classification and prediction models

Bio: 

Stepan Pushkarev is a CTO of Hydrosphere.io. His background is in engineering of data platforms. He spent last couple of years building continuous delivery and monitoring tools for machine learning applications as well as designing streaming data platforms. He works closely with data scientists to make them productive and successful in their daily operations.