Uber’s Experimentation Monitoring Tool
Uber’s Experimentation Monitoring Tool

Abstract: 

At Uber we run 100s of experiments at any given time. The goal of these experiments is to continuously improve our products and the user experience. However, sometimes an experimenter might have some key metrics in mind, and may not be aware of the impact on few other metrics important to the team and the company overall, and as the experiment is run these unmonitored metrics may regress.

In order to detect and mitigate such scenarios, we have built out our experimentation monitoring platform. The goal is to identify and monitor few guardrail metrics that we do not want to degrade during the experiment runs. We apply a variation of the sequential A/B methodology to continuously monitor these guardrail metrics and detect any regression between the treatment and the control group of an experiment. If regression detected, we send alerts to the owner of the experiment.

Bio: 

Suman Bhattacharya is a senior data scientist at Uber’s Platform organization and lead the experimentation platform data science at Uber. Previously, he has held different data science positions at Gap, Inc, Samsung Research and LeTote. He hold a PhD in Astrophysics. His interests lies in the application of different statistics and machine learning techniques to solve business problems.

Open Data Science

 

 

 

Open Data Science
One Broadway
Cambridge, MA 02142
info@odsc.com

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Youtube
Consent to display content from - Youtube
Vimeo
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google