From Zero to Airflow: Bootstrapping into a Best-in-Class Risk Analytics Platform

Abstract: To effectively compete in the Fintech space decisions have to be lightning fast and accurate. There is an inherent tradeoff between the two, and to push the needle on both requires a union of excellent models and top notch infrastructure. At BlueVine, we chose to leverage Apache Airflow as an engine for the wide array of models, heuristics and supplemental processes that form our analytics ecosystem. Our key automation tasks (approvals, rejections, fraud assessment) are always underlied by a complex web of dependencies, which is why a workflow design and execution tool such as Airflow is perfect for our needs. But how can a company successfully transition from a legacy system into Airflow, or optimize such a feature packed product so that it actually yields a sufficient return on investment? These are critical and daunting considerations for any company looking to make the switch without incurring any downtime. Indeed, a 2017 McKinsey report found that about 88% of AI initiatives never get deployed despite huge efforts in the experimental stage. In this presentation we will describe the entire process from end-to-end as a case study of how BlueVine got the implementation of Airflow past this hurdle. We will cover: what was planned, which unexpected problems were encountered, and what effect it had on the relationship between the various data teams. We will also detail the mechanism that is in place today, note some real world insights about the strengths and limitations of Airflow, and track several key metrics that were affected by the switch.

Bio: Ido Shlomo is the head of BlueVine’s data science team in the US, where he works on applying machine learning and other automation solutions for risk management, fraud detection and marketing purposes. Recent work is focused on implementing complex NLP tasks in production systems, and specifically on dealing with the the challenge of consuming unstructured data. Previously Ido worked in the Economics department at Tel Aviv University as a researcher in structural macroeconomic modeling. Ido holds a dual BA in mathematics and philosophy and an MA in economics, both from Tel Aviv University.