Introduction to AutoML: Hyperparameter Optimization and Neural Architecture Search


This tutorial is targeted towards Data Scientists and machine learning engineers who work on machine learning and deep learning models. Given a task, one is interested in finding a well-performing model to solve that task. Very often, this would involve tweaking the model either by changing the hyper parameters or modifying its architecture in order to find a better performing model. In the past, this was always done manually. But, with the advent of Automated Machine Learning, we can now leave that to the machines. In this tutorial, we will provide an overview of Hyperparameter Optimization (HPO) and Neural Architecture Search (NAS).

For HPO, we will introduce some common algorithms (Bayesian optimization, Hyperband) and tools (Ray, Hyperopt) that are used. For NAS, we will briefly touch upon some search algorithms and go into detail about evolutionary architecture search and building a surrogate model.

This tutorial also includes snippets of code written in Python using Ray and Pytorch frameworks. Codes will be uploaded on Github.

Session Outline:

Part 1: Neural Architecture Search:
We begin with an overview of Neural Architecture Search (NAS). This includes a quick primer on search space and various flavours of NAS algorithms. We will elaborate the specifics of evolutionary NAS and also describe how to accelerate NAS using a surrogate model and early stopping. We will then address how to add constraints such as latency, model size etc. to the search.

Part 2: Hyperparameter Optimization:
In this section, we will provide an introduction to Hyperparameter Optimization (HPO), Bayesian Optimization and Hyperband. We will demonstrate how to design a search space and apply these algorithms for HPO using Ray to run on a distributed cluster.

Background Knowledge:

The audience are required to know basics of machine learning and deep learning.


Tejaswini Pedapati works at IBM Research. Her research is focused on interpretability and automating deep learning. To that end, she was involved in developing tools and algorithms to provide these capabilities for IBM products. She has a masters’ degree from Columbia University.

Open Data Science




Open Data Science
One Broadway
Cambridge, MA 02142

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Consent to display content from - Youtube
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google