Automating Data Engineering Workloads Using Generative AI


A typical data organization spend a lot of time building the following artifacts among other things:
* Ingesting customer files * Generating Business Metrics * Building Analytics Reports * Publishing Data Extracts * Creating Feature Stores for ML. Lot of these tasks are repetitive, time consuming and creates lot of complexity for the data organization team which creates technical debt and a long delivery cycles. Welcome to the world of Generative AI where we will use commercially available Large Language Models (LLMs) such as Open AI to demonstrate how solve these problems using innovative design patterns. In this session, we will be talking about Generative Data Engineering, starting from basic concepts and contexts and progressing to how to generate your data pipeline and analytics code rather than writing the code manually. These design patterns go a long way in shortening the lead time, automating undifferentiated work, and allowing data teams to move up the data value chain.
These can be learned relatively quickly and can be implemented either as a simple one off solution or can be deployed as an enterprise framework.


Ashish Mrig is a Boston based data technologist and AI/ML practitioner and has been practicing the data engineering craft for 20+ years before it was considered cool ! Ashish has deep understanding of core design patterns, data abstractions, automations and one of his core competencies is to reduce data chaos and build scalable data platforms. He loves to tackle new challenges and push the engineering boundary. In his free time loves to play tennis and visit our beautiful national parks.

Open Data Science




Open Data Science
One Broadway
Cambridge, MA 02142

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Consent to display content from - Youtube
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google