On Mathematical Theories of Deep Learning 1 Yuan YAO HKUST

Acknowledgement A following-up course at HKUST: https://deeplearning-math.github.io/

Outline ´ Why mathematical theories of Deep Learning? ´ The tsunami of deep learning in recent years… ´ What Theories Do We Have or Need? ´ Harmonic Analysis: what are optimal representation of functions? ´ Approximation Theory: when deep networks are better than shallow ones? ´ Optimization: what are the landscapes of risk and how to efficiently find a good optimum? ´ Statistics: how deep net models can generalize well?

Reaching Human Performance Level 1997 2004 AlphaGo “LEE” 2016 Deep Blue in 1997 AlphaGo ”ZERO” ¡D ¡Silver ¡ et ¡al. ¡Nature 550, ¡ 354–359 ¡(2017) ¡doi:10.1038/nature24270

ImageNet Dataset 14,197,122 labeled images 21,841 classes Labeling required more than a year of human effort via Amazon Mechanical Turk

ImageNet Top 5 classification error ImageNet (subset): 1.2 million training images 100,000 test images 1000 classes ImageNet large-scale visual recognition Challenge source: https://www.linkedin.com/pulse/must-read-path-breaking-papers-image-classification-muktabh-mayank 13 /

Crowdcomputing: researchers raising the competition record

Depth as function of year [He et al., 2016]

Growth of Deep Learning

New Moore’s Laws CS231n attendance NIPS registrations

"We’re at the beginning of a new day… This is the beginning of the AI revolution.” — Jensen Huang, GTC Taiwan 2017

Some Cold Water: Tesla Autopilot Misclassifies Truck as Billboard Problem: Why? How can you trust a blackbox?

Deep Learning may be fragile in generalization against noise! [Goodfellow et al., 2014] Small but malicious perturbations can result in severe misclassification Malicious examples generalize across different architectures What is source of instability? Can we robustify network?

Kaggle survey: Top Data Science Methods https://www.kaggle.com/surveys/2017 Academic Industry

What type of data is used at work? https://www.kaggle.com/surveys/2017 Academic Industry

What’s wrong with deep learning? Ali Rahimi NIPS’17: Machine (deep) Learning has become alchemy . https://www.youtube.com/watch?v=ORHFOnaEzPc Yann LeCun CVPR’15, invited talk: What’s wrong with deep learning? One important piece: missing some theory ! http://techtalks.tv/talks/whats-wrong-with-deep-learning/61639/

Perceptron: single-layer Invented by Frank Rosenblatt (1957) b x 1 w 1 x 2 w 2 · f ( z ) z = − → w · − → x + b · · w d x d

Locality or Sparsity of Computation Minsky and Papert, 1969 Perceptron can’t do XOR classification Perceptron needs infinite global locality of computation? information to compute connectivity Locality or Sparsity is important: Locality in time? Locality in space?

Multilayer Perceptrons (MLP) and Back-Propagation (BP) Algorithms Rumelhart, Hinton, Williams (1986) Learning representations by back-propagating errors, Nature, 323(9): 533-536 BP algorithms as stochastic gradient descent algorithms ( Robbins–Monro 1950; Kiefer- Wolfowitz 1951 ) with Chain rules of Gradient maps MLP classifies XOR, but the global hurdle on topology (connectivity) computation still exists

On Mathematical Theories of Deep Learning 1 Yuan YAO HKUST - PowerPoint PPT Presentation

On Mathematical Theories of Deep Learning 1 Yuan YAO HKUST Acknowledgement A following-up course at HKUST: https://deeplearning-math.github.io/ Outline Why mathematical theories of Deep Learning? The tsunami of deep learning in

Generalization of Deep Learning 1 Yuan YAO HKUST Some Theories are limited but help:

Bayesian Deep Learning Mohd Adnan Problems With Deep Learning What does a model not know?

Building an Auto-formalization Infrastructure from Mathematical Literature through Deep Learning

Machine learning for lattice theories Real-world lattices 2 Machine learning for lattice

deep learning for natural language processing . Sergey I. Nikolenko 1,2,3 Deep Machine

A Mathematical View of Attention Models in Deep Learning Shuiwang Ji, Yaochen Xie Department of

Making w orking theories visible in teaching and learning: Our w orking theories about w orking

Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

4. Learning in MAS 4.1. What is learning? One definition: Bower, Hilgard: Theories of learning,

Chapter 3 Theories Behind Gamification of Learning and Instruction Overview What learning

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Deep Neural Network Mathematical Mysteries for High Dimensional Learning Stphane Mallat

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin

An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 Deep Residual Learning

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Mocha.jl Deep Learning in Julia Chiyuan Zhang (@pluskid) CSAIL, MIT Deep Learning Learning

Minjie Wang Deep Learning Deep Learning trend in the past 10 years Caffe State-of-art DL

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: State of the Art (2020) Deep Learning Lecture Series https://deeplearning.mit.edu

12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from