Abstract: 

Shop Cash Offers, an ads product enabling merchants to bid for customer acquisition, incorporates personalized purchase incentives called Offers. Our objective is to construct and present offers that optimize expected revenue. In this talk, we will demonstrate how to bootstrap a ranking model with a small team and limited training data, gradually transforming it into a state-of-the-art system.

We will begin by outlining the overarching architecture of the ranking model that powers the Shop Cash Offers page. Specifically, we will delve into two key components: 1) the process of identifying the most relevant offers from the available set, and 2) learning the performance of offers while effectively balancing exploration of new offers and exploitation of the best performing ones.

Furthermore, we will explore practical considerations for implementing, optimizing, and maintaining such a system in real-world scenarios. Our discussion will encompass the implementation of a simplified version of the system, and a path to extend it. Throughout the session, we will cover essential topics like evaluating system performance offline, conducting online testing with a focus on proper instrumentation, determining, and considering factors that play a pivotal role in maximizing expected revenue. These factors include proper calibration of probability scores, and measuring tripwire metrics that can affect the long term health of the product like value provided to advertisers, content quality and diversity.

Join us to gain valuable insights learned from building the ML engine powering Shop Cash Offers, and learn how to apply them in your own data science projects within an industry setting.

Bio: 

Mike serves as Distinguished ML Scientist at Shopify, where he leads the ML engineering discipline and as a Continuing Data Science Faculty member at UC Berkeley. He is known for his work developing LLM natural language understanding applications to classify misinformation, founding the fakerfact.org project, as well as for his work as Chief Scientist at SIG and Head of Data Science for the Uber Advanced Technologies Group. He has a wide breadth of ML applications experience including extensive work with deep learning for NLP, time series forecasting, scalable machine learning, adaptive test based reinforcement learning for self-driving, and recommender applications.