Learning Functions: Understanding Gradient Descent, Backpropogation, and Vanishing Gradients
Learning Functions: Understanding Gradient Descent, Backpropogation, and Vanishing Gradients


A neural network model, no matter how deep or complex, implements a function, a mapping from inputs to outputs. The function implemented by a network is determined by the weights the network uses. So, training a network (learning the function the network should implement) on data involves searching for the set of weights that best enable the network to model the patterns in the data. The most commonly used algorithm for learning patterns from data is the gradient descent algorithm. By itself the gradient descent algorithm can be used to train a single neuron, however it cannot be used to train a deep network with multiple hidden layers. Training a deep neural network, involves using both the gradient descent algorithm and the backpropagation algorithm in tandem. These algorithms are at the core of deep learning and understanding how they work is, possibly, the most direct way of understanding the potential and limitations of the deep learning. This talk provides a gentle but still comprehensive introduction to these two important algorithms. I will also explain how the problem of vanishing gradients arises, and how this problem has been addressed in deep learning.


Prof. John D. Kelleher is the Academic Leader of the Information, Communication and Entertainment Research Institute at the Dublin Institute of Technology. His areas of expertise include machine learning, artificial intelligence, natural language processing, and spatial cognition. John has worked in a number of different academic and research focused institutes, including Dublin City University, Media Lab Europe, and DFKI (the German Centre for Artificial Intelligence Research). Currently, his research is supported by the Science Foundation Ireland ADAPT Research Centre (Grant Number 13/RC/2016). He is the co-author of Fundamentals of Machine Learning for Predictive Data Analytics, MIT Press, 2015.

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Consent to display content from Youtube
Consent to display content from Vimeo
Google Maps
Consent to display content from Google