EECS 395, 495: Deep Learning Foundations from Scratch

Quarter Offered

Fall : MWF 2-2:50 ; Borhani, Watt
Winter : W 5-8 ; Borhani, Watt


Prior machine learning experience (e.g. an introductory machine learning course EECS 395/495 or EECS 349 or a similar course) , a thorough understanding of Linear Algebra and Vector Calculus, and strong familiarity with the Python programming language (e.g., basic data manipulation libraries, how to construct functions and classes, etc.). Python will be used for all coding assignments. No other language can be used to complete programming assignments.


The​ ​last​ ​decade​ ​has​ ​seen​ ​a​ ​resurgence​ ​of​ ​deep​ ​neural​ ​networks​ ​as​ ​the​ ​machine​ ​learning  tool​ ​of​ ​choice​ ​for​ ​many​ ​application​ ​areas​ ​including​ ​computer​ ​vision,​ ​reinforcement​ ​learning,​ ​and​ ​natural  language​ ​processing.​ ​This​ ​course​ ​offers​ ​a​ ​holistic​ ​and​ ​hands-on​ ​introduction​ ​to​ ​deep​ ​networks,​ ​their​ ​many  varieties​ ​and​ ​applications,​ ​as​ ​well​ ​as​ ​the​ ​algorithms​ ​used​ ​to​ ​train​ ​them.​ ​In​ ​this​ ​course​ ​we​ ​discuss​ ​the  foundations​ ​of​ ​Feedforward​ ​Networks,​ ​Convolutional​ ​Networks,​ ​and​ ​Recurrent​ ​Networks,​ ​as​ ​well​ ​as​ ​their  usage​ ​within​ ​popular​ ​Reinforcement​ ​Learning​ ​frameworks.​ ​​Using​ ​real​ ​datasets​ ​and​ ​basic​ ​Python​ ​libraries​ ​for  data​ ​manipulation,​ ​vector/matrix​ ​algebra,​ ​and​ ​automatic​ ​differentiation​ ​students​ ​will​ ​code​ ​up​ ​-​ ​from​ ​scratch​ ​-  popular​ ​examples​ ​of​ ​each​ ​such​ ​network​ ​architecture​ ​and​ ​apply​ ​them​ ​to​ ​applications​ ​in​ ​natural​ ​language  processing,​ ​computer​ ​vision,​ ​speech​ ​recognition,​ ​and​ ​automatic​ ​control.

  • This course fulfills the Technical Elective requirement.

INSTRUCTORS: Dr. Reza Borhani and Dr. Jeremy Watt


1. Deep Feedforward Networks

  • a. Supervised learning recap
  • b. Feedforward networks in the context of function approximation
  • c. Simple recipes for building deep networks
  • d. Deep nets vs Kernel methods
  • e. How deep networks came back

2. Technical Issues with Deep Networks

  • a. Mathematical optimization recap
  • b. Automatic differentiation, implementations in Python
  • c. First order stochastic gradient methods
  • d. Regularization techniques
  • e. Technical tricks

3. Convolutional Networks

  • a. From fixed feature extractors to convolutional networks
  • b. Composing filters and functions
  • c. Applications in computer vision and speech processing
  • d. Transfer Learning

4. Recurrent Networks

  • a. Recurrence relations
  • b. Parameterized recurrence relations and feedforward networks
  • c. Deriving vanilla recurrent networks
  • d. Popular architectures, Long-Short-Term-Memory
  • e. Applications in NLP and speech recognition

5. Deep Representation Learning

  • a. Unsupervised learning recap
  • b. Nonlinear extensions of PCA, autoencoders

COURSE HAND-OUTS: This course has no required textbook. Handouts authored by the instructors will be made freely available to students for notes. In addition a small number of relevant sections from the author’s textbook Machine Learning Refined (Cambridge University Press) will be made freely available to students.

PROBLEM SETS: 4-5 problem sets will be assigned and graded.  

COURSE EXAM: There will be one final exam for the course worth 25% of the final grade.

COURSE GRADE: Final grades for the course are based on homework (75%) and final exam (25%)