CS584: Deep Learning
Prof. Shamim Nemati (OH: Mon 1:00pm-2:00pm in BMI (36 Eagle Row, 5th Floor South) 579)
TF: Supreeth Prajwal (OH: Wed 10:15am-11:15am BMI 581)
TF: Ali Ahmadvand (OH: Tue 9am-10am BMI 581)
Time: Monday and Wednesday, 11:30am-12:45pm
Location: MSc E408
Contact: Instructor firstname dot Instructor lastname at emory.edu
Course Website: http://nematilab.info/CS584.html
GitHub: https://github.com/NematiLab/CS584 (permission only)
Announcements
- January 11, 2017: Please bring your laptops for Lecture 2 (Wed 18 Jan 2017). We will have a hands-on demonstration of various computational resources available at Emory for running large scale deep learning computations.
- January 23, 2017: Please complete Assignment 1 by Sunday, 01/29/2017.
- January 30, 2017: Please complete Assignment 2 by Sunday, 02/05/2017.
- February 8, 2017: Please complete Assignment 3 by Sunday, 02/12/2017.
- February 16, 2017: Please complete Assignment 4 by Sunday, 02/26/2017.
- February 20, 2017: Midterm project presentations are on Wed 22 March. Midterm project reports are due Sunday 26th of March.
- March 7, 2017: Optional Assignment 5 has been posted.
- March 15, 2017: Please complete Assignment 6 by Wednesday, 03/29/2017.
- April 3, 2017: Please complete Assignment 7 by Monday, 04/10/2017.
- April 16, 2017: Please complete Assignment 8 by Monday, 04/23/2017.
Schedule
Subject to change.
Feedforward Networks
Mon 23 Jan 2017
Lecture 3: Deep Feedforward Networks
Wed 25 Jan 2017
Lecture 4: Optimization
Mon 30 Jan 2017
Lecture 5: Convolutional Neural Networks
Wed 1 Feb 2017
Lecture 6: Convolutional Neural Networks
Time Series Models
Mon 6 Feb 2017
Lecture 7: Recurrent Neural Networks
Wed 8 Feb 2017
Lecture 8: Dynamic Bayesian Networks
Reinforcement Learning
Mon 13 Feb 2017
Lecture 9: Deep RL, Discrete Action Spaces
Wed 15 Feb 2017
Lecture 10: Deep RL, Continuous Action Spaces
Applications & Practicals
Mon 20 Feb 2017
Lecture 11: Deep Learning in Practice
Wed 22 Feb 2017
Lecture 12: Biomedical Applications
Unsupervised Deep Learning
Monday 27 Feb 2017
Lecture 13: Linear Factor Models
- [required] Book: Goodfellow -- Chapter 13 -- Linear Factor Models
- [required] Book: Murphy -- Chapter 12 -- Latent Linear Models
- [optional] Video: Zoubin Ghahramani -- Graphical Models
- [required] Paper: Sam Roweis and Zoubin Ghahramani. A Unifying Review of Linear Gaussian Models. Neural Computation 11(2), 1999.
Mon 13 March 2017
Lecture 14: Autoencoders
Wed 15 March 2017
Lecture 15: Introduction to Boltzmann Machines
Mon 20 March 2017
Lecture 16: Unsupervised Time Series Modeling
Wed 22 March 2017
Midterm Project Presentations
Sampling and Inference Procedures
Mon 27 March 2017
Lecture 17: Sampling Techniques
- [required] Book: Goodfellow -- Chapter 17 -- Monte Carlo Methods
- [optional] Book: Murphy -- Chapter 23, Section 23.1-23.4 -- Monte Carlo Inference
- [optional] Book: Murphy -- Chapter 24, Sections 24.1-24.4 -- Markov Chain Monte Carlo (MCMC) Inference
- [optional] Book: Murphy -- Chapter 24, Sections 24.5-24.7 -- Markov Chain Monte Carlo (MCMC) Inference
- [optional] Video: Iain Murray -- Markov Chain Monte Carlo
- [optional] Video: de Freitas -- Monte Carlo Simulation for Statistical Inference
- [optional] Video: Christian Robert -- Markov Chain Monte Carlo Methods
Wed 29 March 2017
Lecture 18: Approximate Inference
- [required] Book: Goodfellow -- Chapter 18-19 -- Partition Function and Approximate Inference
Deep Generative Models
Mon 3 April 2017
Lecture 19: Restricted Boltzmann Machine
- [required] Book: Goodfellow -- Chapter 20.3-20.5 -- Deep Boltzmann Machines
- [optional] Book: Murphy -- Chapter 27, Section 27.7 -- Latent Variable Models for Discrete Data
- [optional] Video: Geoffrey Hinton -- Deep Belief Networks
- [optional] Video: Yoshua Bengio and Yann LeCun -- Tutorial on Deep Learning Architectures
Wed 5 April 2017
Lecture 20: Recurrent Temporal RBM
Mon 10 April 2017
Lecture 21: Helmholtz Machines I
Wed 12 April 2017
Lecture 22: Helmholtz Machines II
Deep Learning research
Mon 17 April 2017
Mon 19 April 2017
Final Presentations and Reports
Mon 24, Wed 26 April 2017
- Final Project Presentations
May 1 2017
- Final Project Reports Due
Assignments
Weekly Reports should be documented in one single Google Doc , with clear headings for each week (dated). Please share the link with your instructor.
These assignments include your progress on hands-on coding tutorials, including any obstacles preventing you from successfully completing your tutorials.
Additionally, you're expected to review and critique one paper every week (minimum 300 words).
Finally, your weekly report should include any concepts that you find difficult to understand pertaining to the corresponding week's lectures.
[required] Assignment 1: Feedforward Neural Networks
[required] Assignment 2: Convolutional Neural networks
[required] Assignment 3: Recurrent Neural Networks
[required] Assignment 4: A Practical Application
[required] Assignment 5: Visualization
[required] Assignment 6: Autoencoders (AE)
[required] Assignment 7: Unsupervised Learning
[optional] Assignment 8: Generative Networks
Final Project
Use overleaf to write a journal style report and share the link with your instructor.
You may use the NIPS style files available here.
- Final Report: Due 1 May 2017
Grading
- Weekly Assignments and Reports (lowest dropped): 20%
- Midterm Project Report: 30%
- Final Project Presentation and Report: 45%
- Attendance and Contribution to Class Discussions: 5%
General Deep Learning and Machine Learning Books
Other Resources
Frequently Asked Questions
- Can I collaborate with another student on my term project? I expect a unique midterm report from all students. You may form teams of size n<=3 for the final project with written permission (i.e., email), from your instructor.
- What are the requirements of this course?
Knowledge of linear algebra, multivariate calculus, basic statistics and probability theory, and Machine Learning (CS534) and Artificial Intelligence (CS557). Homework and project will require programming in Python and Matlab.
- Are there standard techniques for handling missing data? This is an area of active research. For starters, it helps to distinguish between MCAR (missing completely at random), MAR (missing at random), MNAR (missing not at random). You may find the following papers useful (or rather food for thought!):
(1) Rezende et al. -- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
(2) Bengio et al., Recurrent Neural Networks for Missing or Asynchronous Data
(3) Lipton et al. -- Modeling Missing Data in Clinical Time Series with RNNs
(4) Che et al. -- Recurrent Neural Networks For Multivariate Time Series With Missing Values
(5) Uchihashi and Kanemura, Modeling the Propensity Score with Statistical Learning
(6) Beaulieu-Jones et al, Missing Data Imputation In The Electronic Health Record Using Deeply Learned Autoencoders
(7) Leke et al., Missing data and deep learning
- What do you recommend for visualizing high-dimensional datasets?
t-SNE is a popular technique. Use it with care!
- What are some good references on model comparison?
(1) Rezende et al. -- DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988 Sep 1:837-45.
(2) Bengio et al., Pencina MJ, D'Agostino RB, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Statistics in medicine. 2008 Jan 30;27(2):157-72.
(3) Lipton et al. -- Cook NR, Ridker PM. The use and magnitude of reclassification measures for individual predictors of global cardiovascular risk. Annals of internal medicine. 2009 Jun 2;150(11):795.
(4) Che et al. -- Airola A, Pahikkala T, Waegeman W, De Baets B, Salakoski T. A comparison of AUC estimators in small-sample studies. InMLSB 2010 (pp. 3-13).
(5) Davis and Goadrich -- Davis J, Goadrich M. The relationship between Precision-Recall and ROC curves. InProceedings of the 23rd international conference on Machine learning 2006 Jun 25 (pp. 233-240). ACM.