Diving into Deep Learning

Neural networks have rapidly become the de facto standard in computer vision and speech recognition. Growing interest in understanding these algorithms have motivated a boon of learning resources.

So where does someone look to obtain a basic understanding of neural networks? While the high level intuitions behind neural networks are easy to pick up, the details can be subtle and challenging to grasp. Luckily, several brilliant professors have stepped up to the challenge.

Nando De Freitas (Oxford/Google Deepmind) - Currently my number one recommendation for newcomers to deep learning.

De Freita’s primer requires some comfort with basic machine learning, linear algebra, and calculus, but it provides the most well rounded education on neural nets. The course goal is to teach students the tools to analyze the strengths and weaknesses of any neural net papers by the end of it –and he certainly succeeded. The best part of this series is the ease with which one can transition from lecture learning into developing custom nets. The lectures are taught in the context of the popular Torch deep learning framework. Seeing how nets are implemented in the real world is the best way to understand how they work, and De Fretias’ unique perspective on deep learning streamlines the learning process.

The only weakness to this series is the teaching of convolutional neural networks. He uses great visuals to explain the intuition behind how they work, but the amount of hand waving and new terminology makes it feels a bit rushed. Conv nets are complicated enough to be a standalone course, so I understand the struggle in trying to fit it all in 53 minutes. This topic will be tough for anyone seeing it for the first time.


Fei-Fei Li/Andrej Karpathy (Stanford) - CS231n at Stanford is probably the best set of self-contained convolutional neural net resources. While there are no videos for the course, the notes are supplemented by informative visuals and helpful javascript animations. The first half of the lectures cover necessary machine learning terminology, but the course assumes some familiarity with the subject. What is great about the notes is that although they are dense, the notes are self contained enough that a diligent student can go through these notes in a couple of days. The notes do a great job of cementing in the fundamentals of conv nets using the numpy python framework as a teaching tool.


Michael Nielson (Open Knowledge) - For people with a moderate amount of coding experience who want a quick introduction to the power of deep learning, Neilson’s e-book is supplemented with plenty of python examples. The course starts with basic neural networks and expands on those ideas to demonstrate different concepts. This first half of this e-book and Andrew Ng’s machine learning course are great for beginners. Both assume that the student has no prior knowledge of the subject and provide a balance between theory and application. I found Nielson’s e-book to be helpful when I started learning neural networks because it was one of the only resources that explained nets in plain english. It can get a little bit overwhelming when he starts talking about how to solve the optimization problem, so I recommend Andrew Ng’s explanation of backpropagation for a supplementary perspective on the same concept.

Do note that this e-book is not meant to be a comprehensive overview of deep learning - It introduces ideas for the student to learn in further detail elsewhere. It is best to go from this book into another video series to get a complete picture.


Andrew Ng (Stanford/Baidu) - Ng’s machine learning videos are among the most viewed videos on coursera because of how accessible they are to someone with little technical experience. For any newcomer to the subject, I would recommend skimming through Ng’s class first to develop good fundamentals. His section on neural networks focuses on the intuition behind how they work without getting too bogged down with mathematical jargon and notation.


Geoffrey Hinton (University of Toronto/Google) - Hinton is one the most respected researchers in neural networks because of his 30+ years of research in this area. His ImageNet winning work in 2012 propelled his research lab to international recognition and shaped computer vision and speech recognition into how we know it today. Given his accomplishments, most of Hinton’s video lectures are surprisingly easy to follow. However, the second half of the lectures are a challenge to follow without prior machine learning experience. That being said, these lectures are great references and I often go back to them to get his perspective on certain topics.


Hugo Larochelle (Université de Sherbrooke, Twitter) - Larochelle’s lectures are more suited to someone who wants to do research in the area of neural networks than any of the other resources I have described. While he does not handhold the learner through the technically intense portions of the course, his explanations are supplemented by visuals and very insightful explanations. I particularly like the video format which has the lectures split into 10 minute sessions for each topic. Compared to other courses Larochelle uses far less code to reinforce the theory but does a wonderful job of teaching neural networks from a mathematically rooted perspective.

One interesting note about these lectures is the interactivity between the audience and the teacher. Larochelle is surprisingly active on the youtube comments for the videos, often leaving nuggets of wisdoms for his commenters.


Yoshua Bengio and Ian J. Goodfellow and Aaron Courville (University of Montreal)- I have the least experience with this book, but I did have some time to skim over over it. This book is an appropriate reference for learners lacking machine learning experience and wanting to get into neural network research. The authors provide a nice primer on the necessary vocabulary to begin a neural network education relevant for probability, linear algebra, and classical machine learning techniques. Right now I’m putting this book closer to the theory side of things than the application, but by the time I get through it this perspective may change.


The beauty of this cacophony of resources is that you don’t have to pick one, in fact you shouldn’t. Build your own course that suits your personal learning style. Something doesn’t make sense in Bengio’s book? Jump and see if De Freitas has some torch code on it. Want a justification to the mathematical underpinning that De Fretias skips? Check what Larochelle has to say about it. This is the golden age of learning.