IKT724 Deep Learning

Teaching language

English

Prerequisites

Signal processing, Optimization, Probability theory and stochastic processes, Linear algebra, Mathematical analysis and Programming. Students are allowed to use different programming languages, such as: Python (with TensorFlow Library), Matlab, Octave and C.

Recommended prerequisites

Recommended previous knowledge IKT 720 Optimization and IKT 719 Advanced Optimization (or similar courses elsewhere).

Course contents

This course offers an in-depth study on the mathematical and algorithmic foundations of deep neural networks (DNNs).

The course covers the following main topics:

• Part 1: Fundamentals of statistical learning and large-scale optimization for modern machine learning

  • Review of statistical learning theory, empirical risk minimization, estimators.

  • Supervised, unsupervised and reinforcement learning.

  • Limitations of traditional machine learning methods, motivation for deep neural networks (DNNs).

  • Large-scale statistical learning (convex case): batch vs. mini-batch training, stochastic gradient descent, acceleration methods, second-order methods, variance/noise reduction methods.

  • Non-convex optimization for statistical learning: DNN training problem as non-convex problem, successive linearization/convex approximation, block-coordinate descent methods (application to training auto-encoders and factor models), block-successive upper-bound minimization methods (application to training general DNNs), convergence guarantees.

• Part 2: Algorithms and Architectures of Deep Neural Networks

  • Overview of DNN types: feed-forward, convolutional, linear and recurrent.

  • Mathematical models: DNNs as compositions of non-linear operators.

  • Training with back-propagation, the role of regularization, dropout training.

  • Large-scale training of DNNs: challenges (ill-conditioning, inexact gradients, complex loss surface), algorithms for large-scale training with adaptive learning rate (ADAGRAD, ADAM).

  • Common DNN architectures: Deep convolutional neural networks, Deep recurrent neural networks, LTSM networks, Deep reinforcement learning.

  • Factor models: Principal/independent component analysis, Manifold learning. 

• Part 3: Current challenges and Applications of Deep Learning

  • Novel optimization methods for training DNNs (beyond first-order methods).

  • Training DNNs using Alternating Direction Multiplier Methods (ADMMs).

  • Training with convergence guarantees.

  • Adversarial examples and robustness.

  • Applications in data-driven inference and signal processing tasks.

  • Applications in wireless communication and sensor networks.

  • Geometric deep learning and applications to 3D data.

  • Open problems in deep learning research.

Learning outcomes

Upon successful completion of this course, the students should:

- Understand the underlying concepts and properties related to learning by Deep Neural Networks (DNNs), including its formulation as empirical risk minimization problems and the different training methods.

- Be able to evaluate the performance of common DNN architectures/types, such as deep convolutional/recurrent neural networks, as well as factor models, analyzing also the practical issues.

- Know how to formulate and apply the Deep Learning framework to solve several practical problems in different application domains. 

Examination requirements

Compulsory attendance is the only requirement. 

Assessment methods and criteria

The Final Grade: pass (A or B) or fail (based on 60% of Homework grade + 40% of final project grade). Passing the course is contingent upon attending all lectures, successfully finishing homework problems, and the final project. Regarding the final project, its grade will be assessed based on two parts: project report and oral presentation, which will count equally. For both the project report and the oral presentation, the assessment criteria will be technical correctness and clarity of exposition.