an introduction to neural networks and deep learning
play

An introduction to Neural Networks and Deep Learning Talk given at - PowerPoint PPT Presentation

Dec 26, 2022 •95 likes •749 views

An introduction to Neural Networks and Deep Learning Talk given at the Department of Mathematics of the University of Bologna February 20, 2018 Andrea Asperti DISI - Department of Informatics: Science and Engineering University of Bologna

  1. An introduction to Neural Networks and Deep Learning Talk given at the Department of Mathematics of the University of Bologna February 20, 2018 Andrea Asperti DISI - Department of Informatics: Science and Engineering University of Bologna Mura Anteo Zamboni 7, 40127, Bologna, ITALY andrea.asperti@unibo.it Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 1

  2. A branch of Machine Learning What is Machine Learning? There are problems that are difficult to address with traditional programming techniques: ◮ classify a document according to some criteria (e.g. spam, sentiment analysis, ...) ◮ compute the probability that a credit card transaction is fraudulent ◮ recognize an object in some image (possibly from an inusual viewpoint, in new lighting conditions, in a cluttered scene) ◮ ... Typically the result is a weighted combination of a large number of parameters, each one contributing to the solution in a small degree. Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 2

  3. The Machine Learning approach Suppose to have a set of input-output pairs (training set) {� x i , y i �} the problem consists in guessing the map x i �→ y i The M.L. approach: • describe the problem with a model depending on some parameters Θ (i.e. choose a parametric class of functions) • define a loss function to compare the results of the model with the expected (experimental) values • optimize (fit) the parameters Θ to reduce the loss to a minimum Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 3

  4. Why Learning? Machine Learning problems are in fact optimization problems ! So, why talking about learning? Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 4

  5. Why Learning? Machine Learning problems are in fact optimization problems ! So, why talking about learning? The point is that the solution to the optimization problem is not given in an analytical form (often there is no closed form solution). Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 5

  6. Why Learning? Machine Learning problems are in fact optimization problems ! So, why talking about learning? The point is that the solution to the optimization problem is not given in an analytical form (often there is no closed form solution). So, we use iterative techniques (typically, gradient descent) to progressively approximate the result. Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 6

  7. Why Learning? Machine Learning problems are in fact optimization problems ! So, why talking about learning? The point is that the solution to the optimization problem is not given in an analytical form (often there is no closed form solution). So, we use iterative techniques (typically, gradient descent) to progressively approximate the result. This form of iteration over data can be understood as a way of progressive learning of the objective function based on the experience of past observations. Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 7

  8. Using gradients The objective is to minimize some loss function over (fixed) training samples, e.g. � Θ( w ) = E ( o ( w , x i ) , y i ) i by suitably adjusting the parameters w . See how it changes according to small perturbations ∆( w ) of the parameters w : this is the gradient ∇ w [ θ ] = [ ∂ Θ ∂ w 1 , . . . , ∂ Θ ∂ w n ] of Θ w.r.t. w . The gradient is a vector pointing in the direction of steepest ascent. Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 8

  9. Gradient descent Goal: minimize some loss function Θ( w ) by suitably adjusting the parameters. We can reach a minimal configuration for Θ( w ) by iteratively taking small steps in the direction opposite to the gradient (gradient descent). This is a general technique . Warning: not guaranteed to work: ◮ may end up in local minima ◮ may get lost in plateau Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 9

  10. Next arguments A bit of taxonomy Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 10

  11. Different types of Learning Tasks • supervised learning : inputs + outputs (labels) - classification - regression supervised • unsupervised learning : just inputs - clustering - component analysis - autoencoding unsupervised • reinforcement learning actions and rewards - learning long-term gains - planning reinforcement Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 11

  12. Classification vs. Regression Two forms of supervised learning: {� x i , y i �} Probably a cat! Expected New value input New input classification regression y is discete: y ∈ {• , + } y is (conceptually) continuous Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 12

  13. Many different techniques • Different ways to define the models : Outlook - decision trees Sunny Overcast Rain - linear models Humidity Yes Wind - neural networks High Normal Strong Weak - ... Yes No Yes No decision tree neural net • Different error (loss) functions : - mean squared errors - logistic loss - cross entropy - cosine distance - maximum margin mean squared errors maximum margin - ... Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 13

  14. Next argument Neural Networks Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 14

  15. Neural Network A network of (artificial) neurons Artificial neuron Each neuron takes multiple inputs and produces a single output (that can be passed as input to many other neurons). Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 15

  16. The artificial neuron inputs x 1 activation w 1 function w x 2 2 Σ output b w n bias x n +1 The purpose of the activation function is to introduce a thresholding mechanism (similar to the axon-hillock of cortical neurons). Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 16

  17. Different activation functions The activation function is responsible for threshold triggering. 1 0 1 threshold: if x > 0 then 1 else 0 logistic function: 1+ e − x 1 0 ex − e − x hyperbolic tangent: rectified linear (RELU): if x > 0 then x else 0 ex + e − x Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 17

  18. A comparison with the cortical neuron Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 18

  19. Next argument Networks typology/topology Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 19

  20. Layers A neural network is a collection of artificial neurons connected together. Neurons are usually organized in layers. If there is more than one hidden layer the network is deep, otherwise it is called a shallow network. Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 20

  21. Feed-forward networks If the network is acyclic, it is called a feed-forward network. Feed-forward networks are (at present) the commonest type of networks in practical applications. Important Composing linear transformations makes no sense, since we still get a linear transformation. What is the source of non linearity in Neural Networks? Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 21

  22. Feed-forward networks If the network is acyclic, it is called a feed-forward network. Feed-forward networks are (at present) the commonest type of networks in practical applications. Important Composing linear transformations makes no sense, since we still get a linear transformation. What is the source of non linearity in Neural Networks? The activation function Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 22

  23. Dense networks The most typical feed-forward network is a dense network where each neuron at layer k − 1 is connected to each neuron at layer k . The network is defined by a matrix of parameters (weights) W k for each layer (+ biases). The matrix W k has dimension L k × L k +1 where L k is the number of neurons at layer k . Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 23

  24. Parameters and hyper-parameters The weights W k are the parameters of the model: they are learned during the training phase. The number of layers and the number of neurons per layer are hyper-parameters: they are chosen by the user and fixed before training may start. Other important hyper-parameters govern training such as learning rate, batch-size, number of ephocs an many others. Andrea Asperti Universit` a di Bologna - DISI: Dipartimento di Informatica: Scienza e Ingegneria 24

Recommend

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan

Deep Learning with Neural Networks The Structure and Optimization of Deep Neural Networks Allan Zelener Machine Learning Reading Group January 7 th 2016 The Graduate Center, CUNY Objectives Explain some of the trends of deep learning and

680 views • 44 slides
Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova

Introduction to Artificial Intelligence Neural Networks - Deep Learning for NLP Janyl Jumadinova November 21, 2016 Neural Networks 2/20 Neural Networks 3/20 Neural Networks Neural computing requires a number of neurons , to be connected

813 views • 21 slides
Outline Evolution of neurocomputing Artificial neural networks Feed forward

Outline Evolution of neurocomputing Artificial neural networks Feed forward

Introduction to Neural Networks and Deep Learning Ling Guan References: S. Haykin, Neural Networks , 2 nd Edition, New Jersey: Prentice Hall, 1. 2004. C. M. Bishop, Neural Networks for Pattern Re cognition, New York: 2. Oxford University

451 views • 17 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning objectives What are neural networks? What are deep neural networks? How do we train neural networks? What variants of neural network

331 views • 31 slides
Layer-wise Relevance Propagation in Neural Neural Networks Deep Learning Shortcomings Networks

Layer-wise Relevance Propagation in Neural Neural Networks Deep Learning Shortcomings Networks

LRP Ariyan Zarei Motivation Having More interpretable Layer-wise Relevance Propagation in Neural Neural Networks Deep Learning Shortcomings Networks to have more interpretable Papers and Demo Introduction Machine Learning models

373 views • 35 slides
Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating

Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating

Concise Introduction to Deep Neural Networks Outline: Classification problems Motivating Deep (large) Neural Network (DNN) classifiers Neurons and DNN architectures Numerical training of DNNs (supervised deep learning) Spiking

630 views • 33 slides
Lecture 13: Introduction to Deep Learning Deep Convolutional Neural Networks Aykut Erdem

Lecture 13: Introduction to Deep Learning Deep Convolutional Neural Networks Aykut Erdem

Lecture 13: Introduction to Deep Learning Deep Convolutional Neural Networks Aykut Erdem November 2016 Hacettepe University Administrative Assignment 3 is out! It is due November 30, 2016 You will implement a 2-layer Neural

1.28k views • 114 slides
Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking

Introduction to Deep Neural Networks 0. Logistics Spring 2020 1 Neural Networks are taking over! Neural networks have become one of the major thrust areas recently in various pattern recognition, prediction, and analysis problems In

987 views • 40 slides
CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided Backpropagation, Deep Dream, Deep Art, Fooling Convolutional Neural Networks Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute

1.07k views • 72 slides
AI and Predictive Analytics in Data-Center Environments Neural Networks and Deep Learning Josep

AI and Predictive Analytics in Data-Center Environments Neural Networks and Deep Learning Josep

AI and Predictive Analytics in Data-Center Environments Neural Networks and Deep Learning Josep Ll. Berral @BSC Intel Academic Education Mindshare Initiative for AI Introduction Neural networks attempt to imitate the brain neuron

339 views • 21 slides
Introduction to Artificial Intelligence Deep Learning - Tensor Flow Janyl Jumadinova December 2,

Introduction to Artificial Intelligence Deep Learning - Tensor Flow Janyl Jumadinova December 2,

Introduction to Artificial Intelligence Deep Learning - Tensor Flow Janyl Jumadinova December 2, 2016 Credit: Google Workshop Neural Networks 2/24 Neural Networks 3/24 Neural Networks A fully connected NN layer 4/24 Implementation as

1.2k views • 34 slides
Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning

Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning

Recurrent Neural Networks Deep neural networks have enabled major advances in machine learning and AI y t-1 y t y t+1 Computer vision h t-1 h t h t+1 Language translation Speech recognition h t-1 h t h t+1 Question answering x t-1 x t x t+1

454 views • 43 slides
PATTERN RECOGNITION AND MACHINE LEARNING Slide Set 5: Neural Networks and Deep Learning November

PATTERN RECOGNITION AND MACHINE LEARNING Slide Set 5: Neural Networks and Deep Learning November

PATTERN RECOGNITION AND MACHINE LEARNING Slide Set 5: Neural Networks and Deep Learning November 2019 Heikki Huttunen heikki.huttunen@tuni.fi Signal Processing Tampere University default Traditional Neural Networks Neural networks have

571 views • 21 slides
Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020

Recurrent Neural Networks CS60010: Deep Learning Abir Das IIT Kharagpur Mar 11, 2020 Introduction Recurrent Neural Network LSTM Agenda Get introduced to different recurrent neural architecture e.g. , RNNs, LSTMs, GRUs etc. Get

743 views • 48 slides
6. Convolutional Neural Networks CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

6. Convolutional Neural Networks CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

6. Convolutional Neural Networks CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from Zsolt Kira Quiz coming up Next Monday (2/5) 30 minutes Topics: Optimization Basic neural networks Neural Network

619 views • 41 slides
Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations deeplearning.cce2020@gmail.com Deep Networks Intuition Neural networks with multiple hidden layers - Deep networks [Hinton, 2006] Deep Networks Intuition

1.33k views • 28 slides
Graph Neural Networks Xiachong Feng TG 2019-04-08 Relies heavily on A Gentle Introduction

Graph Neural Networks Xiachong Feng TG 2019-04-08 Relies heavily on A Gentle Introduction

Graph Neural Networks Xiachong Feng TG 2019-04-08 Relies heavily on A Gentle Introduction to Graph Neural Networks (Basics, DeepWalk, and GraphSage) Structured deep models: Deep learning on graphs and beyond Representation Learning

4.09k views • 64 slides
Neural Networks: What can a network represent Deep Learning, Fall 2020 1 Recap : Neural

Neural Networks: What can a network represent Deep Learning, Fall 2020 1 Recap : Neural

Neural Networks: What can a network represent Deep Learning, Fall 2020 1 Recap : Neural networks have taken over AI Tasks that are made possible by NNs, aka deep learning Tasks that were once assumed to be purely in the human domain

1.92k views • 150 slides
Deep Neural Networks CMSC 422 M ARINE C ARPUAT marine@cs.umd.edu Deep learning slides credit:

Deep Neural Networks CMSC 422 M ARINE C ARPUAT marine@cs.umd.edu Deep learning slides credit:

Deep Neural Networks CMSC 422 M ARINE C ARPUAT marine@cs.umd.edu Deep learning slides credit: Vlad Morariu Training (Deep) Neural Networks Computational graphs Improvements to gradient descent Stochastic gradient descent Momentum

814 views • 21 slides
Deep Neural Networks for Data-Driven Turbulence Models @ICERM 2019: Scientific Machine Learning

Deep Neural Networks for Data-Driven Turbulence Models @ICERM 2019: Scientific Machine Learning

Institute of Aerodynamics and Gas Dynamics Andrea Beck David Flad C.-D. Munz Deep Neural Networks for Data-Driven Turbulence Models @ICERM 2019: Scientific Machine Learning Outline 1 Introduction Machine Learning with Neural Networks 2

627 views • 49 slides
Deep Learning Barun Patra Index Convolutional Networks Introduction to Neural Nets

Deep Learning Barun Patra Index Convolutional Networks Introduction to Neural Nets

Deep Learning Barun Patra Index Convolutional Networks Introduction to Neural Nets Inspiration Activations Kernels Sigmoid Idea Tanh As used in NLP Relu (Derivatives) Paper Discussion

806 views • 46 slides
Deep Learning: Theory and Practice 30-04-2019 Recurrent Neural Networks Introduction The

Deep Learning: Theory and Practice 30-04-2019 Recurrent Neural Networks Introduction The

Deep Learning: Theory and Practice 30-04-2019 Recurrent Neural Networks Introduction The standard DNN/CNN paradigms (x,y) - ordered pair of data vectors/images (x) and target (y) Moving to sequence data (x(t),y(t)) where this

780 views • 38 slides
Neural Networks: What can a network represent Deep Learning, Spring 2018 1 Recap : Neural

Neural Networks: What can a network represent Deep Learning, Spring 2018 1 Recap : Neural

Neural Networks: What can a network represent Deep Learning, Spring 2018 1 Recap : Neural networks have taken over AI Tasks that are made possible by NNs, aka deep learning 2 Recap : NNets and the brain In their basic form, NNets

2.14k views • 145 slides

More recommend