introduction to high performance machine learning
play

Introduction to High-Performance Machine Learning: Convolutional - PowerPoint PPT Presentation

Dec 01, 2022 •286 likes •1.12k views

Introduction to High-Performance Machine Learning: Convolutional Neural Networks Valeriu Codreanu SURFsara 1 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks SURFsara History: 1971: Founded

  1. Introduction to High-Performance Machine Learning: Convolutional Neural Networks Valeriu Codreanu SURFsara 1 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  2. SURFsara History: 1971: Founded by the VU, UvA, and CWI 2013: SARA (Stichting Academisch Rekencentrum A’dam) becomes part of SURF Cartesius (Bull supercomputer): 40.960 Ivy Bridge / Haswell cores: 1327 TFLOPS 56GBit/s Infiniband 64 nodes with 2 K40m GPUs each: 210 TFLOPS Broadwell & KNL extension (Nov 2016) 177 BDW and 18 KNL nodes: 284TFLOPS 7.7 PB Lustre parallel file-system Top500 position #45 2014/11 Increasing number of deep learning #142 2017/11 projects! 2 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  3. Outline Today’s lecture: • NNs for Computer vision • High-performance CNN vision models (AlexNet, GoogLeNet, ResNet, …) • High-performance hardware • GPUs, CPUs, special-purpose accelerators • Scaling deep learning vision models • Bottlenecks • Efficiency • Successful results • Today’s hands-on: • Deep learning frameworks • Example: Tensorflow/Horovod and MXNet on Intel and NVIDIA • Example: Keras 3 GPU Programming www.prace-ri.eu

  4. Classical Computer Vision Pipeline CV experts 1.Select / develop features: SURF, HoG, SIFT, RIFT, … 2.Add on top of this Machine Learning for multi-class recognition and train classifier Feature Detection, Extraction: Classification SIFT, HoG... Recognition Classical CV feature definition is domain- specific and time-consuming 4 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  5. Deep Learning–based Vision Pipeline Deep Learning: • Build features automatically based on training data • Combine feature extraction and classification DL experts: define NN topology and train NN Detection, Deep NN ... Deep NN ... Classification Recognition Deep Learning promise: train good feature automatically, same method for different domain 5 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  6. Neural networks 6 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  7. 7 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  8. Object segmentation Figure credit: Dai, He, and Sun, “Instance-aware Semantic Segmentation via Multi-task Network Cascades”, CVPR 2016 8 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  9. Pose estimation Figure credit: Cao et al, “Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields”, arXiv 2016 9 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  10. Image Super-resolution Figure credit: Ledig et al, “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network”, arXiv 2016 10 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  11. Art generation Gatys, Ecker, and Bethge, “Image Style Transfer using Convolutional Neural Networks”, CVPR 2016 (left) Mordvintsev, Olah, and Tyka, “Inceptionism: Going Deeper into Neural Networks” (upper right) Johnson, Alahi, and Fei-Fei: “Perceptual Losses for Real-Time Style Transfer and Super-Resolution”, ECCV 2016 (bottom left) 11 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  12. Neural networks 12 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  13. The neuron model Based on slide from Andrew Ng 13 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  14. 14 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks Parallel Computer Architecture and Programming CMU 15-418/15-618, Spring 2017

  15. 15 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks Parallel Computer Architecture and Programming CMU 15-418/15-618, Spring 2017

  16. 4 easy steps: 1. Sample a batch of data 2. Forward prop it through the graph (network), get loss 3. Backprop to calculate the gradients 4. Update the parameters using the gradient 16 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  17. Fully-connected networks Nonlinearity Matrix Matrix Multiply Multiply x s a ŷ [C 1 ] [C 2 ] [C 2 ] [C 3 ] w 2 w 1 [C 2 ×C 3 ] [C 1 ×C 2 ] 17 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  18. Convolutional neural networks Fully Connected y t i n r a o e i t u n Pooling i l l o n v o n N o C x s a p ŷ C 1 ×H×W C 2 ×H×W C 2 ×H×W C 2 ×H/2×W/2 C 3 w 1 w 2 C 2 × C 1 × k × k C 2 HW/4×C 3 18 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  19. The convolution operator Sobel operator: 19 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  20. Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 20 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  21. Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 21 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  22. Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 22 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  23. Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 23 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  24. Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 24 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  25. Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 25 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  26. Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 26 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  27. Convolution as matrix-matrix multiplication Very important factor motivating early GPU usage for neural network training! 27 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  28. Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 28 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  29. Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 29 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  30. l Subsampling pixels will not change the object bird bird Subsampling Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 30 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  31. Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 31 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  32. LeNet-5 Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 32 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  33. Neural networks 33 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  34. AlexNet 5 convolutional layers 3 fully connected layers + soft-max 650K neurons , 60 M weights 34 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  35. 35 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  36. Depth reduction experiment on AlexNet Rob Fergus, NIPS 2013 36 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  37. Depth reduction experiment on AlexNet 37 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  38. Depth reduction experiment on AlexNet 38 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  39. Depth reduction experiment on AlexNet 39 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  40. Depth reduction experiment on AlexNet 40 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

  41. Feature visualization Stanford CS231n class CS231n: Convolutional Neural Networks for Visual Recognition 41 Introduction to High-Performance Machine Learning: www.prace-ri.eu Convolutional Neural Networks

Recommend

High Performance Machine Learning: Advances, Challenges and Opportunities Eduardo Rodrigues

High Performance Machine Learning: Advances, Challenges and Opportunities Eduardo Rodrigues

High Performance Machine Learning: Advances, Challenges and Opportunities Eduardo Rodrigues Lecture @ ERAD-RS - April 11th, 2019 IBM Research Artificial Intelligence Deep Blue (1997) AI and Machine Learning AI ML

647 views • 63 slides
High Performance and Energy Efficient Machine Learning Accelerators and Variable Precision

High Performance and Energy Efficient Machine Learning Accelerators and Variable Precision

High Performance and Energy Efficient Machine Learning Accelerators and Variable Precision Circuits for Sub-10nm Microprocessors (Invited Paper) Ram Krishnamurthy Senior Principal Engineer & Director of High Performance and Low Voltage

609 views • 28 slides
High-Performance Hardware for Machine Learning U.C. Berkeley October 19, 2016 William Dally

High-Performance Hardware for Machine Learning U.C. Berkeley October 19, 2016 William Dally

High-Performance Hardware for Machine Learning U.C. Berkeley October 19, 2016 William Dally NVIDIA Corporation Stanford University Machine learning is transforming computing Speech Vision Natural Language Understanding Autonomous Vehicles

800 views • 68 slides
1 Why Study Machine Learning? Why Study Machine Learning? Cognitive Science The Time is Ripe

1 Why Study Machine Learning? Why Study Machine Learning? Cognitive Science The Time is Ripe

What is Learning? Herbert Simon: Learning is any process by which a system improves performance from CS 391L: Machine Learning experience. Introduction What is the task? Classification Problem solving / planning /

603 views • 6 slides
Learning Everywhere: Pervasive Machine Learning for Effective High-Performance Computing Geoffrey

Learning Everywhere: Pervasive Machine Learning for Effective High-Performance Computing Geoffrey

Learning Everywhere: Pervasive Machine Learning for Effective High-Performance Computing Geoffrey Fox, many others ., Shantenu Jha* Rutgers University and Brookhaven National Lab. http://radical.rutgers.edu Outline Learning

450 views • 24 slides
High-Performance Communication in Machine Learning Lunch talk at ICL/UTK, Knoxville, TN W ITH

High-Performance Communication in Machine Learning Lunch talk at ICL/UTK, Knoxville, TN W ITH

spcl.inf.ethz.ch @spcl_eth T. H OEFLER High-Performance Communication in Machine Learning Lunch talk at ICL/UTK, Knoxville, TN W ITH CONTRIBUTIONS FROM T AL B EN -N UN , D AN A LISTARH , S HOSHANA J AKOBOVITS , C EDRIC R ENGGLI , AND OTHERS AT

788 views • 53 slides
Measuring model performance or error Introduction to Machine Learning Is our model any good?

Measuring model performance or error Introduction to Machine Learning Is our model any good?

INTRODUCTION TO MACHINE LEARNING Measuring model performance or error Introduction to Machine Learning Is our model any good? Context of task Accuracy Computation time Interpretability 3 types of tasks

1.31k views • 77 slides
High-Performance Communication in Machine Learning Keynote at Austrian HPC Meeting, Feb. 2019,

High-Performance Communication in Machine Learning Keynote at Austrian HPC Meeting, Feb. 2019,

spcl.inf.ethz.ch @spcl_eth T. H OEFLER High-Performance Communication in Machine Learning Keynote at Austrian HPC Meeting, Feb. 2019, Grundlsee W ITH CONTRIBUTIONS FROM T AL B EN -N UN , D AN A LISTARH , S HOSHANA J AKOBOVITS , C EDRIC R ENGGLI ,

606 views • 56 slides
Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 12, 2015 Today: Readings: What is machine learning? The Discipline of ML Decision tree learning Mitchell, Chapter 3

872 views • 29 slides
1 Sample Job Posting Extracted Job Template Subject: US - TN -SOFTWARE PROGRAMMER

1 Sample Job Posting Extracted Job Template Subject: US - TN -SOFTWARE PROGRAMMER

Machine Learning and Text Textual data is ubiquitous and ever-important Text Learning and Information Extraction WWW, digital libraries, LexisNexis, Medline, news, . Machine learning is required for high performance on key tasks

386 views • 5 slides
Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Introduction to Machine Learning 1 / 12 Outline 1 Nearest Neighbor Prediction 2 Complexity Considerations 3 The Voronoi Diagram 4 Overfitting

564 views • 12 slides
High-Performance Machine Learning for Weather Prediction Applications Hatem Ltaief Senior

High-Performance Machine Learning for Weather Prediction Applications Hatem Ltaief Senior

High-Performance Machine Learning for Weather Prediction Applications Hatem Ltaief Senior Research Scientist Extreme Computing Research Center King Abdullah University of Science and Technology NVIDIA GTC at San Jose, CA May 8-11, 2017 H.

467 views • 35 slides
Machine Learning Machine Learning: algorithms that use experience to improve their

Machine Learning Machine Learning: algorithms that use experience to improve their

Machine Learning Machine Learning: algorithms that use experience to improve their performance We use machine learning in situations where it is very challenging (or impossible) to define the rules by hand: e.g. face detection

1.04k views • 28 slides
MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING

MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING

JACOPO DI IORIO, MATTEO FONTANA, DANIELE OCCHIUTO, CLAUDIA VOLPETTI MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING STATISTICAL LEARNING Separate evolutions, but with shared methodologies MERGE

565 views • 55 slides
ECE 5984: Introduction to Machine Learning Topics: Supervised Learning Measuring

ECE 5984: Introduction to Machine Learning Topics: Supervised Learning Measuring

ECE 5984: Introduction to Machine Learning Topics: Supervised Learning Measuring performance Nearest Neighbour Readings: Barber 14 (kNN) Dhruv Batra Virginia Tech TA: Qing Sun PhD candidate at ECE department Research

606 views • 26 slides
Scikit-learn 1 / 13 Machine Learning Learning: using experience to improve performance.

Scikit-learn 1 / 13 Machine Learning Learning: using experience to improve performance.

Scikit-learn 1 / 13 Machine Learning Learning: using experience to improve performance. Machine learning: a class of algorithms that uses data (experience) to improve performance on a task Kinds of Tasks Classification: identify the

568 views • 13 slides
Introduction to Machine Learning Andrea Passerini passerini@disi.unitn.it Machine Learning

Introduction to Machine Learning Andrea Passerini passerini@disi.unitn.it Machine Learning

Introduction to Machine Learning Andrea Passerini passerini@disi.unitn.it Machine Learning Introduction to Machine Learning What is Machine Learning A computer program is said to learn from experience E with respect to some class of tasks T

220 views • 20 slides
Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Introduction to Machine Learning 1 / 18 Outline 1 Classification, Regression, Unsupervised Learning 2 About Dimensionality 3 Drawings and

701 views • 18 slides
Machine Learning for Auto Optimization What is Machine Learning? Definition: Machine

Machine Learning for Auto Optimization What is Machine Learning? Definition: Machine

Machine Learning for Auto Optimization What is Machine Learning? Definition: Machine learning refers to any system where the performance of a machine in performing a task improves by gaining more experience in performing that task .

394 views • 27 slides
Introduction to Machine Learning Introduction Prof. Andreas Krause Institute for Machine

Introduction to Machine Learning Introduction Prof. Andreas Krause Institute for Machine

Introduction to Machine Learning Introduction Prof. Andreas Krause Institute for Machine Learning (las.ethz.ch) What is Machine Learning I: An example Classify email messages as Spam or Non Spam Classical Approach : manual rules

1.27k views • 59 slides
INTRODUCTION TO MACHINE LEARNING Joseph C. Osborn CS 51A Spring 2020 Machine Learning is

INTRODUCTION TO MACHINE LEARNING Joseph C. Osborn CS 51A Spring 2020 Machine Learning is

INTRODUCTION TO MACHINE LEARNING Joseph C. Osborn CS 51A Spring 2020 Machine Learning is Machine learning is about predicting the future based on the past. -- Hal Daume III Machine Learning is Machine learning is about predicting

917 views • 59 slides
Introduction to machine learning COMS 4721 Learning from data Machine learning : the study

Introduction to machine learning COMS 4721 Learning from data Machine learning : the study

Introduction to machine learning COMS 4721 Learning from data Machine learning : the study of computational mechanisms that learn from data in order to make predictions and decisions. Example 1: image classification A birdwatcher

932 views • 14 slides
Advanced Introduction to Machine Learning, CMU-10715 Manifold Learning Barnabs Pczos

Advanced Introduction to Machine Learning, CMU-10715 Manifold Learning Barnabs Pczos

Advanced Introduction to Machine Learning, CMU-10715 Manifold Learning Barnabs Pczos Motivation Find meaningful low-dimensional structures hidden in high-dimensional observations. The human brain confronts the same problem in

1.06k views • 85 slides
Ensemble Learning Machine Learning Introduction 2 In our daily life Asking different

Ensemble Learning Machine Learning Introduction 2 In our daily life Asking different

Machine Learning 1 Ensemble Learning Machine Learning Introduction 2 In our daily life Asking different doctors opinions before undergoing a major surgery Reading user reviews before purchasing a product There are countless

695 views • 22 slides

More recommend