Deploying Large Spark Models and scoring in Real time at Scale
Deploying Large Spark Models and scoring in Real time at Scale


I will answer these five simple questions;

1. How does one build a pyspark model and deploy it in a scala pipeline with no code rewrite - Solving the greatest fights between datascientist who want to code in python and data engineers who like the tried and tested type safety of the JVM.

2. How does one beat the spark context latency to serve spark models in milliseconds to handle near realtime business needs

3. How does one build a ML model, zip it up and deploy it across platforms in a completely vendor neutral way i.e. build your model on AWS and deploy it on GCP or vice-versa.

4. How does one leverage the years of efforts spent in software engineering and use it directly in building datascience pipelines without reinventing the wheel and pain.

5. How does on build a completely GDPR compliant machine learning model with 0.88 on the ROC curve.


A dataengineer/datascientist and chief builder of AbundanceAI. Having worked in core distributed systems, high frequency finance and ecommerce, he loves solving problems with data and scale. He measures his success by the number of successful ML models that he pushes into production and the number of times he did not order pizza after trying out his latest culinary experiment. An avid long distance runner and an aspiring chef.

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Consent to display content from Youtube
Consent to display content from Vimeo
Google Maps
Consent to display content from Google