Data Engineering & MLOps

Training | MLOps & Data Engineering | ML for Programmers | Beginner-Intermediate

By attending this workshop, you will learn how you can become a contributor to Apache Airflow project. You will learn how to setup development environment, how to pick your first issue, how to communicate effectively within the community and how to make your first PR – experienced committers of Apache Airflow project will give you step-by-step instructions and will guide you in the process. When you finish the workshop you will be equipped with everything that is needed to make further contributions to the Apache Airflow project.

Lesson 1: Development environment

Learn how to setup and configure Apache Airflow development environment in less than 10 minutes. You will learn how modern tooling is used to provide consistent, reproducible environment for making contributions for distributed contributors of Apache Airflow. You will learn how to run tests and debug the code, and make sure that the tests are passing in the Continuous Integration environment as well as locally…more details

Training | Machine Learning | MLOps & Data Engineering | Beginner-Intermediate

In this half-day hands-on training, we will use free-tier resources in the Google Cloud Platform (GCP) to introduce learners to the practical use of cloud computing resources in data science and machine learning. Learners should have some experience with data analytics, data science or machine learning. Learners should also have a Gmail account with no former GCP use associated with it, or be willing to create such an account. While fluency in R or Python will be very helpful, it is not rigorously required, as well-annotated scripts will be provided. No previous exposure to or use of cloud computing is required; this is introductory-level in terms of its cloud computing assumptions. This training will be useful for those considering cloud adoption, interested in data engineering, or interested in working with public data as citizen scientists…more details

Track Keynote | MLOps & Data Engineering | Intermediate

Although machine learning algorithms and open source libraries have greatly advanced in the last decade, many challenges remain to building production data science and machine learning applications. Data science teams still spend the majority of their time acquiring and cleaning input data, and once a team launches an application, it has to spend a substantial amount of effort just to keep it running. At Databricks, we have experienced these challenges across thousands of organizations and domains, so we launched two new open source projects recently to simplify data operations and machine learning. Delta Lake is a transactional layer on top of data lake storage such as S3 or HDFS that enables reliable data pipelines, rollback, time travel, and multi-stage bronze/silver/gold patterns for managing production datasets. This allows teams to set up high quality ingest pipelines and rapidly roll back errors. MLflow, on the other hand, is an open source platform for managing the machine learning lifecycle, including experiments, models, workflows and deployments…more details

Workshop | MLOps & Data Engineering | Intermediate

Kedro is a development workflow tool open sourced by QuantumBlack, a McKinsey company. Many data science teams have started using the library for their pipelines but are unsure how to integrate with other model tracking tools, such as MLflow. In this tutorial, we will give an overview of Kedro and MLflow and demo how to leverage the best of both. The goal of this session is to demonstrate how Kedro and MLflow fit together in a scalable AI architecture. To start, we will give an overview of Kedro and an overview of MLflow: – What are they used for? – What functionality do they provide? – How do they compare as tools? Next, we will walk through a demo of a Kedro project that has MLflow integrated into it. Finally, we will go over deployment options…more details

Workshop | ML for Programmers | MLOps & Data Engineering | Intermediate-Advanced

Organizations have long used relational databases for a wide variety of data-intensive applications. Data scientists and analysts need to understand how to work with relational databases, particularly for common data science tasks, such as finding, exploring, analyzing and extracting data within a relational database. SQL is an expressive declarative language designed for working with tabular data. Since its inception, new features have been added that allow for complex expressions and computation, such as regular expressions, sliding window functions, and analytical operations that create non-relational results. Understanding the advanced features of SQL will help data scientists reduce the need for programming custom data manipulation functionality…more details

Talk | Deep Learning | MLOps & Data Engineering | Intermediate-Advanced

Large scale distributed training has become an essential element to scaling the productivity for ML engineers. Today, ML models are getting larger and more complex in terms of compute and memory requirements. The amount of data we train on at Facebook is huge. In this talk, we will learn about the Distributed Training Platform to support large scale data and model parallelism. We will touch base on Distributed Training support for PyTorch and how we are offering a flexible training platform for ML engineers to increase their productivity at facebook scale…more details

Training | MLOps & Data Engineering | Intermediate

Ray is an open-source distributed framework from U.C. Berkeley’s RISELab that easily scales Python applications from a laptop to a cluster, with an emphasis on ML/AI systems, such as reinforcement learning. It is now used in many production deployments. In this tutorial, we’ll use several hands-on examples to explore the problems that Ray solves and the useful features it provides, such as rapid distribution, scheduling, and execution of “tasks” and management of distributed stateful “serverless” computing. We’ll see how it’s used in several ML libraries (and play with examples using those libraries). You’ll learn when to use Ray and how to use it in your projects…more details

Workshop | MLOps & Data Engineering | Machine Learning | Beginner-Intermediate

The speed in which intelligent decision making can take place is changing the very fabric of the modern industry. From autonomous routing decisions for real-time traffic avoidance to intelligent integrated systems for customer service and assisted analysis or forecasting, the time to correct decision can be a huge differentiator. During the course of this workshop, you will learn how to harness the power of Apache Spark to build up a data ingestion pipeline that can process and learn from streaming data. These learnings can then be applied to make simple decisions using SparkSQL in a parallel streaming system…more details

Talk | ML for Programmers | MLOps & Data Engineering | Beginner-Intermediate

Kubernetes & container platforms provide desired agility, flexibility, scalability, & portability for data scientists to train, test, & deploy ML models quickly, without IT dependency. The session will provide an overview of containers and Kubernetes, and how these technologies can help solve the challenges faced by data scientists, ML engineers, and application developers. Next, we will review the key capabilities required in a containers and kubernetes platform to help data scientists easily use technologies like Jupyter Notebooks, ML frameworks, programming languages to innovate faster. Finally we will share the available platform options (e.g. Red Hat OpenShift, KubeFlow, etc.), and some examples of how data scientists are accelerating their ML initiatives with containers and kubernetes platform…more details

Workshop | Deep Learning | MLOps & Data Engineering | Advanced

In this tutorial, we will guide you through hands-on examples of integrating tf.keras model with different data input sources through tf.data in production environments, from simple csv/json files to SQL databases and to cloud data warehouse services such as Google Cloud BigQuery. We will also cover Apache Kafka as a data input to illustrate the streaming data pipeline architecture for continuous data processing with machine learning. As a bonus, attendees will learn the basics of distributed machine learning and its production usage…more details