CS584: Deep Learning

Prof. Shamim Nemati (OH: Mon 1:00pm-2:00pm in BMI (36 Eagle Row, 5th Floor South) 579)
TF: Supreeth Prajwal (OH: Wed 10:15am-11:15am BMI 581)
TF: Ali Ahmadvand (OH: Tue 9am-10am BMI 581)
Time: Monday and Wednesday, 11:30am-12:45pm
Location: MSc E408
Contact: Instructor firstname dot Instructor lastname at emory.edu
Course Website: http://nematilab.info/CS584.html
GitHub: https://github.com/NematiLab/CS584 (permission only)

Announcements

January 11, 2017: Please bring your laptops for Lecture 2 (Wed 18 Jan 2017). We will have a hands-on demonstration of various computational resources available at Emory for running large scale deep learning computations.
January 23, 2017: Please complete Assignment 1 by Sunday, 01/29/2017.
January 30, 2017: Please complete Assignment 2 by Sunday, 02/05/2017.
February 8, 2017: Please complete Assignment 3 by Sunday, 02/12/2017.
February 16, 2017: Please complete Assignment 4 by Sunday, 02/26/2017.
February 20, 2017: Midterm project presentations are on Wed 22 March. Midterm project reports are due Sunday 26th of March.
March 7, 2017: Optional Assignment 5 has been posted.
March 15, 2017: Please complete Assignment 6 by Wednesday, 03/29/2017.
April 3, 2017: Please complete Assignment 7 by Monday, 04/10/2017.
April 16, 2017: Please complete Assignment 8 by Monday, 04/23/2017.

Schedule

Subject to change.

Wed 11 Jan 2017

Lecture 1: Introduction to Deep Learning

[required] Book: Goodfellow -- Chapters 1-3 -- Introduction, Linear Algebra, Probability and Info. Theory

[optional] Book: Murphy -- Chapter 28 -- Deep Learning

[required] Video: Andrej Karpathy -- Introduction to Deep Learning and Historical Context
[optional] Video: de Freitas -- Introduction to Deep Learning
[optional] Paper: LeCun, Bengio & Hinton Deep Learning
[optional] Paper: Jürgen Schmidhuber Deep Learning Conspiracy

Wed 18 Jan 2017

Lecture 2: Intro to Deep Learning Software and Hardware

[required] Book: Goodfellow -- Chapter 4-5 -- Numerical Compu., ML Basics
[required] Video: Andrej Karpathy -- Deep Learning libraries
[required] Video: Andrej Karpathy -- Data-driven approach, kNN, Linear Classification 1
[required] Video: Andrej Karpathy -- Linear Classification 2, Optimization
[optional] Video: de Freitas -- Linear Models
[optional] Video: de Freitas -- Maximum likelihood and information

Feedforward Networks

Mon 23 Jan 2017

Lecture 3: Deep Feedforward Networks

[required] Book: Goodfellow -- Chapter 6-7 -- Deep Feedforward Networks, Regularization for Deep Learning
[required] Video: Andrej Karpathy -- Backpropagation, Neural Networks 1
[optional] Video: de Freitas -- Regularization, model complexity and data complexity (part 1)
[optional] Video: de Freitas -- Regularization, model complexity and data complexity (part 2)
[optional] Paper: Jaderberg et al, Decoupled Neural Interfaces Using Synthetic Gradients
[optional] Paper: Lillicrap et al., Random synaptic feedback weights support error backpropagation for deep learning

Wed 25 Jan 2017

Lecture 4: Optimization

[required] Book: Goodfellow -- Chapter 8 -- Optimization for Training Deep Models
[required] Video: Andrej Karpathy -- Neural Networks Part 2
[optional] Video: de Freitas -- Optimization
[optional] Paper: LeCun, Efficient BackProp
[optional] Paper: Bengio, Understanding the difficulty of training deep feedforward neural networks

Mon 30 Jan 2017

Lecture 5: Convolutional Neural Networks

[required] Book: Goodfellow -- Chapter 9 -- Convolutional Networks
[required] Video: Andrej Karpathy -- Neural Networks Part 3 / Intro to ConvNets
[required] Video: Andrej Karpathy -- Convolutional Neural Networks
[optional] Video: de Freitas -- Convolutional Neural Networks
[optional] Paper: DeshpandeThe 9 Deep Learning Papers You Need To Know About

Wed 1 Feb 2017

Lecture 6: Convolutional Neural Networks

[required] Book: Goodfellow -- Chapter 9 -- Convolutional Networks
[required] Video: Andrej Karpathy -- Localization and Detection
[required] Video: Andrej Karpathy -- Visualization, Deep Dream, Neural Style, Adversarial Examples
[required] Video: Andrej Karpathy -- ConvNets in practice
[optional] Paper: Gulshan, Peng et al., Deep Learning Algorithm for Detection of Diabetic Retinopathy

Time Series Models

Mon 6 Feb 2017

Lecture 7: Recurrent Neural Networks

[required] Book: Goodfellow -- Chapter 10 -- Sequence Modeling
[required] Video: Andrej Karpathy -- Recurrent Neural Networks, Image Captioning, LSTM
[optional] Video: de Freitas -- Recurrent Neural Nets and LSTMs
[optional] Paper: Greff et al. -- LSTM: A Search Space Odyssey
[optional] Paper: Lipton et al. -- Learning To Diagnose With LSTM Recurrent Neural Networks

Wed 8 Feb 2017

Lecture 8: Dynamic Bayesian Networks

[required] Book: Murphy -- Chapter 17, Section 17.1-17.5 -- Markov and Hidden Markov Models
[optional] Book: Murphy -- Chapter 10, Sections 10.1-10.5 -- Directed Graphical Models (Bayes Nets)
[optional] Video: Zoubin Ghahramani -- Graphical Models
[optional] Paper: Choi et al. -- Using recurrent neural network models for early detection of heart failure onset
[optional] Paper: Liao and Ahn, Combining Deep Learning and Survival Analysis for Asset Health Management

Reinforcement Learning

Mon 13 Feb 2017

Lecture 9: Deep RL, Discrete Action Spaces

[required] Paper: Mnih et al., Human-level control through deep reinforcement learning
[required] Video: de Freitas -- Reinforcement learning and neuro-dynamic
[optional] Video: Pineau -- Introduction to Reinforcement Learning
[optional] Paper: Lipton, Combating Reinforcement Learning's Sisyphean Curse with Intrinsic

Wed 15 Feb 2017

Lecture 10: Deep RL, Continuous Action Spaces

[required] Paper: Mnih et al., Recurrent Models of Visual Attention
[required] Video: de Freitas -- Deep Reinforcement Learning - Policy search
[required] Repo: A List of Deep Reinforcement Learning Papers

Applications & Practicals

Mon 20 Feb 2017

Lecture 11: Deep Learning in Practice

[required] Book: Goodfellow -- Chapter 11 -- Practical Methodology
[optional] Paper: Bengio, Practical recommendations for gradient-based training of deep architectures
[optional] Paper: Lipton, The Mythos of Model Interpretability
[optional] Video: de Freitas, Bayesian Optimization

Wed 22 Feb 2017

Lecture 12: Biomedical Applications

[required] Book: Goodfellow -- Chapter 12 -- Applications
[required] Paper: Min, Lee et al., Deep Learning in Bioinformatics
[optional] Paper: Ranganath et al., Deep Survival Analysis
[optional] Paper: Katzman et al., Deep Survival: A Deep Cox Proportional Hazards Network

Unsupervised Deep Learning

Monday 27 Feb 2017

Lecture 13: Linear Factor Models

[required] Book: Goodfellow -- Chapter 13 -- Linear Factor Models
[required] Book: Murphy -- Chapter 12 -- Latent Linear Models
[optional] Video: Zoubin Ghahramani -- Graphical Models
[required] Paper: Sam Roweis and Zoubin Ghahramani. A Unifying Review of Linear Gaussian Models. Neural Computation 11(2), 1999.

Mon 13 March 2017

Lecture 14: Autoencoders

[required] Book: Goodfellow -- Chapter 14 -- Autoencoders
[required] Video: Andrej Karpathy -- Unsupervised Learning
[required] Paper: Miotto et al., Deep Patient
[required] Paper: Cheng et al., Computer-Aided Diagnosis with Deep Learning
[required] Paper: Lasko et al, Computational Phenotype Discovery Using Unsupervised Feature Learning over Noisy, Sparse, and Irregular Clinical Data

Wed 15 March 2017

Lecture 15: Introduction to Boltzmann Machines

[required] Book: Goodfellow -- Chapter 16.7, 20.1, 20.2 -- Structured Probabilistic Models for Deep Learning
[optional] Paper: Montavon, Muller -- Deep Boltzmann Machines and the Centering Trick.
[optional] Paper: Hinton -- A practical guide to training restricted Boltzmann machines.

Mon 20 March 2017

Lecture 16: Unsupervised Time Series Modeling

[required] Paper: Fraccaro et al., Sequential Neural Models with Stochastic Layers
[required] Paper: Längkvist et al., A Review of Unsupervised Feature Learning and Deep Learning for Time Series Modeling
[required] Paper: Bayer and Osendorfer Learning Stochastic Recurrent Networks

Wed 22 March 2017

Midterm Project Presentations

Sampling and Inference Procedures

Mon 27 March 2017

Lecture 17: Sampling Techniques

[required] Book: Goodfellow -- Chapter 17 -- Monte Carlo Methods
[optional] Book: Murphy -- Chapter 23, Section 23.1-23.4 -- Monte Carlo Inference
[optional] Book: Murphy -- Chapter 24, Sections 24.1-24.4 -- Markov Chain Monte Carlo (MCMC) Inference
[optional] Book: Murphy -- Chapter 24, Sections 24.5-24.7 -- Markov Chain Monte Carlo (MCMC) Inference
[optional] Video: Iain Murray -- Markov Chain Monte Carlo
[optional] Video: de Freitas -- Monte Carlo Simulation for Statistical Inference
[optional] Video: Christian Robert -- Markov Chain Monte Carlo Methods

Wed 29 March 2017

Lecture 18: Approximate Inference

[required] Book: Goodfellow -- Chapter 18-19 -- Partition Function and Approximate Inference

Deep Generative Models

Mon 3 April 2017

Lecture 19: Restricted Boltzmann Machine

[required] Book: Goodfellow -- Chapter 20.3-20.5 -- Deep Boltzmann Machines
[optional] Book: Murphy -- Chapter 27, Section 27.7 -- Latent Variable Models for Discrete Data
[optional] Video: Geoffrey Hinton -- Deep Belief Networks
[optional] Video: Yoshua Bengio and Yann LeCun -- Tutorial on Deep Learning Architectures

Wed 5 April 2017

Lecture 20: Recurrent Temporal RBM

[required] Book: Goodfellow -- Chapter 20 -- Deep Generative Models
[required] Paper: Sutskever, Hinton et al. -- The Recurrent Temporal Restricted Boltzmann Machine
[required] Paper: Mittelman et al. -- Structured Recurrent Temporal Restricted Boltzmann Machines
[required] Paper: Chung et al. -- A Recurrent Latent Variable Model for Sequential Data

Mon 10 April 2017

Lecture 21: Helmholtz Machines I

[optional] Paper: Kirby -- A Tutorial on Helmholtz Machines
[optional] Paper: Hinton -- The "wake-sleep" algorithm for unsupervised neural networks
[optional] Paper: Lin and Tegmark -- Why does deep and cheap learning work so well?

Wed 12 April 2017

Lecture 22: Helmholtz Machines II

[optional] Paper: Kirby -- A Tutorial on Helmholtz Machines
[optional] Paper: Bornschein -- Bidirectional Helmholtz Machines
[optional] Paper: Mehta and Schwab -- An exact mapping between the Variational Renormalization Group and Deep Learning.

Deep Learning research

Mon 17 April 2017

Generative Adversarial Networks

Deep Residual Networks

Mon 19 April 2017

Generative Adversarial Networks

Deep Residual Networks

Final Presentations and Reports

Mon 24, Wed 26 April 2017

Final Project Presentations

May 1 2017

Final Project Reports Due

Assignments

Weekly Reports should be documented in one single Google Doc , with clear headings for each week (dated). Please share the link with your instructor.
These assignments include your progress on hands-on coding tutorials, including any obstacles preventing you from successfully completing your tutorials.
Additionally, you're expected to review and critique one paper every week (minimum 300 words).
Finally, your weekly report should include any concepts that you find difficult to understand pertaining to the corresponding week's lectures.

[required] Assignment 1: Feedforward Neural Networks

[required] Assignment 2: Convolutional Neural networks

[required] Assignment 3: Recurrent Neural Networks

[required] Assignment 4: A Practical Application

[required] Assignment 5: Visualization

[required] Assignment 6: Autoencoders (AE)

[required] Assignment 7: Unsupervised Learning

[optional] Assignment 8: Generative Networks

Final Project

Use overleaf to write a journal style report and share the link with your instructor. You may use the NIPS style files available here.

Final Report: Due 1 May 2017

Grading

Weekly Assignments and Reports (lowest dropped): 20%
Midterm Project Report: 30%
Final Project Presentation and Report: 45%
Attendance and Contribution to Class Discussions: 5%

Are there standard techniques for handling missing data? This is an area of active research. For starters, it helps to distinguish between MCAR (missing completely at random), MAR (missing at random), MNAR (missing not at random). You may find the following papers useful (or rather food for thought!):
(1) Rezende et al. -- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
(2) Bengio et al., Recurrent Neural Networks for Missing or Asynchronous Data
(3) Lipton et al. -- Modeling Missing Data in Clinical Time Series with RNNs
(4) Che et al. -- Recurrent Neural Networks For Multivariate Time Series With Missing Values
(5) Uchihashi and Kanemura, Modeling the Propensity Score with Statistical Learning
(6) Beaulieu-Jones et al, Missing Data Imputation In The Electronic Health Record Using Deeply Learned Autoencoders
(7) Leke et al., Missing data and deep learning
What do you recommend for visualizing high-dimensional datasets?
t-SNE is a popular technique. Use it with care!
What are some good references on model comparison?
(1) Rezende et al. -- DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988 Sep 1:837-45.
(2) Bengio et al., Pencina MJ, D'Agostino RB, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Statistics in medicine. 2008 Jan 30;27(2):157-72.
(3) Lipton et al. -- Cook NR, Ridker PM. The use and magnitude of reclassification measures for individual predictors of global cardiovascular risk. Annals of internal medicine. 2009 Jun 2;150(11):795.
(4) Che et al. -- Airola A, Pahikkala T, Waegeman W, De Baets B, Salakoski T. A comparison of AUC estimators in small-sample studies. InMLSB 2010 (pp. 3-13).
(5) Davis and Goadrich -- Davis J, Goadrich M. The relationship between Precision-Recall and ROC curves. InProceedings of the 23rd international conference on Machine learning 2006 Jun 25 (pp. 233-240). ACM.