About OH Course Work Class Notes Lectures Recitations Assignments Docs & Tools Resources S23 F22
Menu
About
OH
Course Work
Class Notes
Lectures
Recitations
Assignments
Docs & Tools
Resources
S23 F22
11-785 Introduction to Deep Learning
Spring 2023
Class Streaming Link

In-Person Venue: Giant Eagle Auditorium, Baker Hall (A51)

Bulletin and Active Deadlines

Assignment Deadline Description Links
HW0P1 30th Jan, 11:59 PM
Fundamentals of Numpy, Pytorch, Python & OOP Autolab, Handout
HW0P2 30th Jan, 11:59 PM
Dataloaders in Pytorch and AWS Setup Autolab
Project Gallery
Here's an example of a successful project from Fall 2020.
The team developed an AI Limmerick generator,
and compiled a book from the AI Poet's creations.
Project Report, Project Video, Book (Amazon)
This piece is performed by the Chinese Music Institute at
Peking University (PKU) together with PKU's Chinese orchestra.
This is an adaptation of Beethoven: Serenade in D major, Op.25 - 1.
Entrata (Allegro),for Chinese transverse flute (Dizi), clarinet and flute.

The Course

“Deep Learning” systems, typified by deep neural networks, are increasingly taking over all the AI tasks, ranging from language understanding, speech and image recognition, to machine translation, planning, and even game playing and autonomous driving. As a result, expertise in deep learning is fast changing from an esoteric desirable to a mandatory prerequisite in many advanced academic settings, and a large advantage in the industrial job market.

In this course we will learn about the basics of deep neural networks, and their applications to various AI tasks. By the end of the course, it is expected that students will have significant familiarity with the subject, and be able to apply Deep Learning to a variety of tasks. They will also be positioned to understand much of the current literature on the topic and extend their knowledge through further study.

If you are only interested in the lectures, you can watch them on the YouTube channel.

Course description from student point of view

The course is well rounded in terms of concepts. It helps us understand the fundamentals of Deep Learning. The course starts off gradually with MLPs and it progresses into the more complicated concepts such as attention and sequence-to-sequence models. We get a complete hands on with PyTorch which is very important to implement Deep Learning models. As a student, you will learn the tools required for building Deep Learning models. The homeworks usually have 2 components which is Autolab and Kaggle. The Kaggle components allow us to explore multiple architectures and understand how to fine-tune and continuously improve models. The task for all the homeworks were similar and it was interesting to learn how the same task can be solved using multiple Deep Learning approaches. Overall, at the end of this course you will be confident enough to build and tune Deep Learning models.

Prerequisites

  1. We will be using Numpy and PyTorch in this class, so you will need to be able to program in python3.
  2. You will need familiarity with basic calculus (differentiation, chain rule), linear algebra and basic probability.

Units

Courses 11-785 and 11-685 are equivalent 12-unit graduate courses, and have a final project and HW 5 respectively. Course 11-485 is the undergraduate version worth 9 units, the only difference being that there is no final project or HW 5.

Acknowledgments

Your Supporters

Instructors:

TAs:

Pittsburgh Schedule (Eastern Time)

Lecture: Tuesday and Thursday, 12:30 p.m. - 1:50 p.m. - Good times :')

Recitation: Friday, 12:30 p.m. - 1:50 p.m.

Event Calendar: The Google Calendar below ideally contains all events and deadlines for student's convenience. Please feel free to add this calendar to your Google Calendar by clicking on the plus (+) button on the bottom right corner of the calendar below. Any adhoc changes to the schedule will be visible on the calendar first.


OH Calendar: The Google Calendar below contains the schedule for Office Hours. Please feel free to add this calendar to your Google Calendar by clicking on the plus (+) button on the bottom right corner of the calendar below. Any adhoc changes to the schedule, including extra OH, will be visible on the calendar first.


Office hours: This is the schedule for this semester. We will be using OHQueue(11-785) for both zoom and in-person Office hours. Please refer the OH calendar/ Piazza for updated information on OH hours.

Day Time (Eastern Time) TA Zoom/In Person
Monday 9:00AM to 10:00AM Prakruthi Pradeep Zoom
10:00AM to 11:00AM Yooni Choi Zoom
11:00AM to 12:00PM Arjun Chauhan In-person
2:00PM to 3:00PM Yi Yang In-person
8:00PM to 9:00PM Swathi Jadav Zoom
Tuesday 9:30AM to 10:30AM Liangze "Josh" Li In-person
10:30AM to 11:30AM Yonas Charlie Zoom
2:00PM to 3:00PM Eshani Agrawal In-person
3:00PM to 4:00PM Qin Wang In-person
4:00PM to 6:00PM Vish In-person
6:00PM to 7:00PM Harshith Arun Kumar In-person
Wednesday 9:00AM to 10:00AM Prakruthi Pradeep Zoom
10:00AM to 11:00AM Yooni Choi Zoom
11:00AM to 12:00PM Arjun Chauhan In-person
12:00PM to 1:00PM Abu In-person
2:00PM to 3:00PM Eshani Agrawal In-person
6:30 PM to 7:30 PM Swathi Jadav In-person
Thursday 9:30AM to 10:30AM Liangze "Josh" Li In-person
11:00AM to 12:00PM Abu In-person
2:00PM to 3:00PM Vedant Bhasin In-person
3:00PM to 5:00PM Varun Jain Zoom
6:00PM to 7:00PM Harshith Arun Kumar In-person
Friday 10:30AM to 11:30AM Yonas Charlie Zoom
11:30AM to 12:30PM Paul Ewuzie Zoom
2:00PM to 3:00PM Vedant Bhasin In-person
2:00PM to 4:00PM Sarthak Bisht In-person
Saturday 9:00AM to 10:00AM Paul Ewuzie Zoom
10:00AM to 11:00AM Shikhar Agnihotri Zoom
12:00PM to 1:00PM Ruimeng Chang Zoom
Sunday 9:00AM to 11:00AM Shikhar Agnihotri In-person
12:00PM to 1:00PM Ruimeng Chang Zoom
1:00PM to 2:00PM Qin Wang In-person
6:00PM to 8:00PM Aparajith Srinivasan Zoom

Homework Hackathon: During 'Homework Hackathons', students will be assisted with homework by the course staff. It is recommended to come as study groups.
Location: TBA

Course Work

Policy
Breakdown
Score Assignment      Grading will be based on weekly quizzes (24%), homeworks (50%) and a course project (25%). Note that 1% of your grade is assigned to Attendance.
Quizzes
Quizzes      There will be weekly quizzes.
  • We will retain your best 12 out of the remaining 14 quizzes.
  • Quizzes will generally (but not always) be released on Friday and due 48 hours later.
  • Quizzes are scored by the number of correct answers.
  • Quizzes will be worth 24% of your overall score.
Assignments
Assignments There will be five assignments in all. Assignments will include autolab components, where you must complete designated tasks, and a kaggle component where you compete with your colleagues.
  • Autolab components are scored according to the number of correctly completed parts.
  • We will post performance cutoffs for HIGH (90%), MEDIUM (70%), LOW (50%), and VERY LOW (30%) for Kaggle competitions. Scores will be interpolated linearly between these cutoffs.
  • Assignments will have a “preliminary submission deadline”, an “on-time submission deadline” and a “late-submission deadline.”
    • Early submission deadline: You are required to make at least one submission to Kaggle by this deadline. People who miss this deadline will automatically lose 10% of subsequent marks they may get on the homework. This is intended to encourage students to begin working on their assignments early.
    • On-time deadline: People who submit by this deadline are eligible for up to five bonus points. These points will be computed by interpolation between the A cutoff and the highest performance obtained for the HW. The highest performance will get 105.
    • Late deadline: People who submit after the on-time deadline can still submit until the late deadline. There is a 10% penalty applied to your final score, for submitting late.
    • Slack days: Everyone gets up to 10 slack days, which they can distribute across all their homework P2s only. Once you use up your slack days you will fall into the late-submission category by default. Slack days are accumulated over all parts of all homeworks, except HW0, to which no slack applies.
    • Kaggle scoring: We will use max(max(on-time score), max(slack-day score), .0.9*max(late-submission score)) as your final score for the HW. If this happens to be a slack-days submission, slack days corresponding to the selected submission will be counted.
  • Assignments carry 50% of your total score. HW0 is not graded (but is mandatory), while each of the subsequent four are worth 12.5%.
  • A fifth HW, HW5, will be released later in the course and will have the same weight as a course project. Please see Project section below for more details.
Project
Project
  • All students taking a graduate version of the course are required to do a course project. The project is worth 25% of your grade. These points are distributed as follows: 20% - Midterm Report; 35% - Project Video; 5% - Responding to comments on Piazza; 40% - Project report.
  • Note that a Project is mandatory for 11-785/18-786 students. In the event of a catastrophe (remember Spring 2020), the Project may be substituted with HW5. 11-685 Students may choose to do a Project instead of HW5. Either your Project OR HW5 will be graded.
Attendance
Attendance
  • If you are in section A you are expected to attend in-person lectures. We will track attendance.
  • If you are in any of the other (out-of-timezone) sections, you must watch lectures live on zoom. Real-time viewing is mandatory unless you are in inconvenient time zones. Others are required to obtain specific permission to watch the pre-recorded lectures (on MediaServices).
  • If viewed on MediaServices, the lectures of each week must be viewed before 8AM of the Monday following the following week (Otherwise, it doesn’t count)
  • At the end of the semester, we will select a random subset of 50% of the lectures and tabulate attendance
  • If you have attended at least 70% of these (randomly chosen) lectures, you get the attendance point
  • Final grade
    Final grade The end-of-term grade is curved. Your overall grade will depend on your performance relative to your classmates.
    Pass/Fail
    Pass/Fail Students registered for pass/fail must complete all quizzes, HWs and if they are in the graduate course, the project. A grade equivalent to B- is required to pass the course.
    Auditing
    Auditing Auditors are not required to complete the course project, but must complete all quizzes and homeworks. We encourage doing a course project regardless.
    End Policy

    Study groups

    This semester we will be implementing study groups. It is highly recommended that you join a study group; Check piazza for further updates.

    Piazza: Discussion Board

    Piazza is what we use for discussions. You should be automatically signed up if you're enrolled at the start of the semester. If not, please sign up TBA. Also, please follow the Piazza Etiquette when you use the piazza.

    AutoLab: Software Engineering

    AutoLab is what we use to test your understand of low-level concepts, such as engineering your own libraries, implementing important algorithms, and developing optimization methods from scratch.

    Kaggle: Data Science

    Kaggle is where we test your understanding and ability to extend neural network architectures discussed in lecture. Similar to how AutoLab shows scores, Kaggle also shows scores, so don't feel intimidated -- we're here to help. We work on hot AI topics, like speech recognition, face recognition, and neural machine translation.

    MediaServices/YouTube: Lecture and Reciation Recordings

    CMU students who are not in the live lectures should watch the uploaded lectures at MediaServices in order to get attendance credit. Links to individual videos will be posted as they are uploaded.

    Our YouTube Channel is where non-CMU folks can view all lecture and recitation recordings. Videos marked “Old“ are not current, so please be aware of the video title.

    Books and Other Resources

    The course will not follow a specific book, but will draw from a number of sources. We list relevant books at the end of this page. We will also put up links to relevant reading material for each class. Students are expected to familiarize themselves with the material before the class. The readings will sometimes be arcane and difficult to understand; if so, do not worry, we will present simpler explanations in class.

    You can also find a nice catalog of models that are current in the literature here. We expect that you will be in a position to interpret, if not fully understand many of the architectures on the wiki and the catalog by the end of the course.

    Academic Integrity

    You are expected to comply with the University Policy on Academic Integrity and Plagiarism.
    • You are allowed to talk with and work with other students on homework assignments.
    • You can share ideas but not code. You should submit your own code.
    Your course instructor reserves the right to determine an appropriate penalty based on the violation of academic dishonesty that occurs. Violations of the university policy can result in severe penalties including failing this course and possible expulsion from Carnegie Mellon University. If you have any questions about this policy and any work you are doing in the course, please feel free to contact your instructor for help.

    Class Notes

    A book containing class notes is being developed in tandem with this course; check it out.

    Schedule of Lectures

    You can watch the recorded lectures on MediaServices.
    Lecture Date Topics Slides and Video Additional Materials Quiz
    0 -
    • Course Logistics
    • Learning Objectives
    • Grading
    • Deadlines
    Slides (*.pdf)
    Video (YT)
    Quiz 0A
    Quiz 0B
    1 Tuesday,
    17 Jan
    • Introduction
    Slides (*.pdf)
    Video (YT)
    The New Connectionism (1988)
    On Alan Turing's Anticipation of Connectionism
    McCullogh and Pitts paper
    Rosenblatt: The perceptron
    Bain: Mind and body
    Hebb: The Organization Of Behaviour
    Quiz 1
    2 Thursday,
    19 Jan
    • Neural Nets As Universal Approximators
    Slides (*.pdf)
    Video (YT) (F22 in-person lecture link)
    Shannon (1949)
    Boolean Circuits
    On the Bias-Variance Tradeoff
    3 Tuesday,
    24 Jan
    • The problem of learning, Empirical Risk Minimization
    Slides (*.pdf)
    Video (YT)
    Widrow and Lehr (1992)
    Adaline and Madaline
    Convergence of perceptron algorithm
    Threshold Logic
    TC (Complexity)
    AC (Complexity)
    Quiz 2
    4 Thursday,
    26 Jan
    • Empirical risk minimization and gradient descent
    • Training the network: Setting up the problem
    Slides (*.pdf)
    Video (YT)
    Werbos (1990)
    Rumelhart, Hinton and Williams (1986)
    5 Tuesday,
    31 Jan
    • Backpropagation
    • Calculus of Backpropagation
    Slides (*.pdf)
    Video (YT)
    Werbos (1990)
    Rumelhart, Hinton and Williams (1986)
    Quiz 3
    6 Thursday,
    2 Feb
    • Convergence issues
    • Loss Surfaces
    • Momentum
    TBA Backprop fails to separate, where perceptrons succeed, Brady et al. (1989)
    Why Momentum Really Works
    7 Tuesday,
    7 Feb
    • Optimization
    • Batch Size, SGD, Mini-batch, second-order methods
    TBA Momentum, Polyak (1964)
    Nestorov (1983)
    Derivatives and Influences
    Quiz 4
    8 Thursday,
    9 Feb
    • Optimizers and Regularizers
      • Choosing a divergence (loss) function
      • Batch normalization
      • Dropout
    TBA
    Derivatives and Influence Diagrams
    ADAGRAD, Duchi, Hazan and Singer (2011)
    Adam: A method for stochastic optimization, Kingma and Ba (2014)
    9 Tuesday,
    14 Feb
    • Shift invariance and Convolutional Neural Networks
    TBA Quiz 5
    10 Thursday,
    16 Feb
    • Models of vision, Convolutional Neural Networks
    TBA
    11 Tuesday,
    21 Feb
    • Learning in Convolutional Neural Networks
    TBA CNN Explainer Quiz 6
    12 Thursday,
    23 Feb
    • Learning in CNNs
    • Transpose Convolution
    • CNN Stories
    TBA
    13 Tuesday,
    28 Feb
    • Time Series and Recurrent Networks
    TBA Fahlman and Lebiere (1990)
    How to compute a derivative, extra help for HW3P1 (*.pptx)
    Quiz 7
    14 Thursday,
    2 Mar
    • Stability and Memory, LSTMs
    TBA Bidirectional Recurrent Neural Networks
    - Tuesday,
    7 Mar
    • No Class - Spring Break
    - Quiz 8
    - Thursday,
    9 Mar
    • No Class - Spring Break
    -
    15 Tuesday,
    14 Mar
    • Sequence Prediction
    • Alignments and Decoding
    TBA LSTM Quiz 9
    16 Thursday,
    16 Mar
    • Sequence prediction
    • Connectionist Temporal Classification (CTC) - Blanks and Beam-search
    TBA
    17 Tuesday,
    21 Mar
    • Language Models
    • Sequence To Sequence Prediction
    TBA Labelling Unsegmented Sequence Data with Recurrent Neural Networks Quiz 10
    18 Thursday,
    23 Mar
    • Sequence To Sequence Methods
    • Attention
    TBA Attention Is All You Need
    The Annotated Transformer - Attention is All You Need paper, but annotated and coded in pytorch!
    19 Tuesday,
    28 Mar
    • Transformers and GNNs
    TBA A comprehensive Survey on Graph Neural Networks Quiz 11
    20 Thursday,
    30 Mar
    • Learning Representations
    • AutoEncoders
    TBA
    21 Tuesday,
    4 Apr
    • Variational Auto Encoders
    TBA Tutorial on VAEs (Doersch)
    Autoencoding variational Bayes (Kingma)
    Quiz 12
    22 Thursday,
    6 Apr
    • Variational Auto Encoders II
    TBA
    23 Tuesday,
    11 Apr
    • Generative Adversarial Networks, 1
    TBA Quiz 13
    - Thursday,
    13 Apr
    • No Class - Spring Carnival
    - -
    24 Tuesday,
    18 Apr
    • Generative Adversarial Networks, 2
    TBA Quiz 14
    25 Thursday,
    20 Apr
    • Hopfield Nets and Auto Associators
    TBA
    26 Tuesday,
    25 Apr
    • Hopfield Nets and Boltzmann Machines
    TBA
    No Quiz
    27 Thursday,
    27 Apr
    • Boltzmann Machines
    TBA

    Schedule of Recitations

    `
    Recitation Date Topics Materials Videos Instructor
    0A Friday, 13th Jan Python & OOP Fundamentals Notebook (*.zip)

    Video (YT): 1, 2

    Paul Ewuzie
    0B Friday, 13th Jan Fundamentals of NumPy Notebook (*.zip)

    Video (YT): 1, 2, 3, 4, 5, 6, 7, 8

    Prakruthi Pradeep
    0C Friday, 13th Jan PyTorch Tensor Fundamentals Notebook + Slides + Cheatsheet (*.zip)

    Video (YT): 1

    Sarthak Bhisht
    0D Friday, 13th Jan Dataset & DataLoaders Notebook + Slides (*.zip) Video (YT)

    Eshani Agrawal
    0E Friday, 13th Jan Introduction to Google Colab Notebook (*.zip) Video(YT)

    Yooni Choi
    0F Friday, 13th Jan AWS Fundamentals Slides (*.zip) Video (YT): 1, 2, 3, 4

    Ruimeng Chang, Arjun
    0G Friday, 13th Jan Debugging, Monitoring Notebook + Slides (*.zip) Video (YT): 1, 2, 3
    Qin Wang
    0H Friday, 13th Jan Basics of Git Colab Notebook
    Video (YT)

    Varun Jain
    0I Friday, 13th Jan Wandb Video (YT)

    0J Friday, 13th Jan What to do if you're struggling Slides (*.zip) Video (YT)

    Aparajith Srinivasan
    0K Friday, 13th Jan Data Preprocessing Notebook + Slides (*.zip)

    Video (YT)

    Yonas Chanie
    0L Friday, 13th Jan Google Cloud

    TBA

    Vish
    0M Friday, 13th Jan Workflow of a Deeplearning Homework Notebook + Slides (*.zip)

    Video (YT)

    Vedant Bhasin
    1 Friday, 20th Jan Your first MLP Code Slides (*.pdf) , Notebook 1a,
    Notebook 1b, Notebook 1c
    Video (YT) Sarthak Bhisht,
    Yooni Choi
    2 Friday, 27th Jan Network Optimization, Hyperparameter Tuning Slides (*.pdf) Video (YT)
    Sarthak Bhisht,
    Varun Jain
    HW1 Bootcamp Saturday, 28th Jan How to get started with HW1 Slides (*.zip) Video (YT)
    Eshani, Sarthak, Yooni
    3 Friday, 3rd Feb Computing Derivatives & Autograd
    4 Friday, 10th Feb Paper Writing Workshop
    5 Friday, 17th Feb CNN: Basics & Backprop
    6 Friday, 24th Feb CNNs: Classification & Verification
    HW2 Bootcamp Saturday, 25th Feb How to get started with HW2
    7 Friday, 3rd Mar CNNs: Verification, Code
    8 Friday, 10th Mar RNN Basics (Pre-recorded)
    9 Friday, 17th Mar CTC, Beam Search
    HW3 Bootcamp Saturday, 18th Mar How to get started with HW3
    10 Friday, 24th Mar Attention, MT, LAS
    11 Friday, 31st Mar Transformers
    HW4 Bootcamp Saturday, 1st Apr How to get started with HW4
    12 Friday, 7th Apr Graph Neural Networks
    13 Friday, 21st Apr Autoencoders and VAEs
    14 Friday, 28th Apr GANs
    HW5 Bootcamp GANs and How to get started with HW5

    Assignments

    ∑ Ongoing, ∏ Upcoming

    Assignment Release Date (EST) Due Date (EST) Related Materials / Links
    HW0P1 Friday, 13th Jan, 11:59 PM Final: 30th Jan, 11:59 PM AutolabHandout
    (see recitation 0s)
    HW0P2 Friday, 13th Jan, 11:59 PM Final: 30th Jan, 11:59 PM AutolabHandout
    (see recitation 0s)
    HW1P1 Sunday, 22nd Jan, 11:59 PM Early Submission: 11th Feb, 11:59 PM Autolab, Writeup (pdf),
    Handout (.tar)
    Final: 17th Feb, 11:59 PM
    HW1P2 Sunday, 22nd Jan, 11:59 PM Early Submission: 11th Feb, 11:59 PM Kaggle, Writeup (pdf),
    Starter Notebook, MCQ
    Final: 17th Feb, 11:59 PM
    HW1 Bonus Sunday, 22nd Jan, 12:00 AM Friday, 10th Mar, 11:59 PM Autolab
    HW1 Autograd Sunday, 22nd Jan, 12:00 AM Wednesday, 3rd May, 11:59 PM Autolab, Writeup (pdf),
    Handout (.tar)
    Project Proposal Monday, 20th Feb, 11:59 PM
    HW5 Monday, 20th Feb, 11:59 PM Thursday, 27th Apr, 11:59 PM EST
    HW2P1 Friday, 17th Feb, 11:59 PM Early Submission: 4th Mar, 11:59 PM TBA
    Final: 17th Mar, 11:59 PM
    HW2P2 Friday, 17th Feb, 11:59 PM Early Submission: 4th Mar, 11:59 PM TBA
    Final: 17th Mar, 11:59 PM
    HW2 Bonus Friday, 17th Feb, 11:59 PM Friday, 7th Apr, 11:59 PM TBA
    HW2 Autograd Friday, 17th Feb, 11:59 PM Wednesday, 3rd May, 11:59 PM TBA
    Project Midterm Report -
    HW3P1 Friday, 10th Mar, 11:59 PM Early Submission: 25th Mar, 11:59 PM TBA
    Final: 7th Apr, 11:59 PM
    HW3P2 Friday, 17th Mar, 11:59 PM Early Submission: 25th Mar, 11:59 PM TBA
    Final: 7th Apr, 11:59 PM
    HW3 Bonus Friday, 10th Mar, 11:59 PM Wednesday, 3rd May, 11:59 PM TBA
    HW2 Autograd Friday, 10th Mar, 11:59 PM Wednesday, 3rd May, 11:59 PM TBA
    HW4P1 Friday, 24th Mar, 11:59 PM Early Submission: 15th Apr, 11:59 PM TBA
    Final: 28th Apr, 11:59 PM
    HW4P2 Friday, 24th Mar, 11:59 PM Early: 15th Apr, 11:59 PM TBA
    Final: 28th Apr, 11:59 PM
    Final Project Video Presentation & Preiliminary Project Report -
    Project Peer reviews -
    Final Project Report Submission -

    Documentation and Tools

    Textbooks

    This is a selection of optional textbooks you may find useful

    Deep Learning
    Dive Into Deep Learning By Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola PDF, 2020
    Deep Learning
    Deep Learning By Ian Goodfellow, Yoshua Bengio, Aaron Courville Online book, 2017
    Neural Networks and Deep Learning
    Neural Networks and Deep Learning By Michael Nielsen Online book, 2016