Introduction CSCE CSCE 479/879 479/879 Good for data with a - PDF document

Introduction CSCE CSCE 479/879 479/879 Good for data with a grid-like topology Lecture 4: Lecture 4: Convolutional Convolutional Image data Neural Neural CSCE 479/879 Lecture 4: Networks Networks Time-series data Stephen Scott Convolutional Neural Networks Stephen Scott We’ll focus on images Based on the use of convolutions and pooling Introduction Introduction Feature extraction Outline Outline Invariance to transformations Convolutions Stephen Scott Convolutions Parameter-efficient CNNs CNNs Example Example Parallels with biological primary visual cortex Architectures Architectures Use of simple cells for low-level detection Each has a local receptive field covering a small region of the visual field Each tends to respond to specific patterns , e.g., vertical lines sscott@cse.unl.edu Use of complex cells for invariance to transformations 1 / 26 2 / 26 Outline Convolutions CSCE CSCE A convolution is an operation that computes a 479/879 479/879 Lecture 4: Lecture 4: weighted average of a data point and its neighbors Convolutional Convolutional Neural Neural Networks Networks Weights provided by a kernel Stephen Scott Stephen Scott Convolutions Introduction Introduction CNNs Outline Outline Pooling Convolutions Convolutions Applications: Examples CNNs Completing the network Use in Feature De-noising Extraction Example Example architectures CNNs Architectures Edge detection Example Architectures Image blurring Image sharpening 3 / 26 4 / 26 Convolutions Convolutions Example: Edge Detection in Images Example [Image from Kenneth Dwain Harrelson] CSCE CSCE Example: Sobel operator for edge detection 479/879 479/879 Lecture 4: Lecture 4: Convolutional Convolutional Neural Neural G x G y Networks Define a small, 2-dimensional kernel over the image I Networks − 1 0 + 1 + 1 + 2 + 1 Stephen Scott Stephen Scott At image pixel I i , j , multiply I i � 1 , j � 1 by kernel value K 1 , 1 , 0 + 2 0 − 2 0 0 and so on, and add to get output I 0 Introduction Introduction i , j − 1 0 + 1 − 1 − 2 − 1 Outline Outline Convolutions Convolutions 0 + 1 Examples − 1 Examples Pass G x and G y over image and add gradient results Use in Feature Use in Feature Extraction Extraction − 2 0 + 2 CNNs CNNs 0 + 1 − 1 Example Example Architectures Architectures This kernel measures the image gradient in the x direction 5 / 26 6 / 26

Convolutions Convolutions Example: Image Blurring Use in Feature Extraction CSCE CSCE 479/879 479/879 A box blur kernel computes uniform average of neighbors Lecture 4: Lecture 4: Convolutional Convolutional Use of pre-defined kernels has been common in Neural Neural Networks Networks feature extraction for image analysis 1 1 1 Stephen Scott Stephen Scott User specified kernels, applied them to input image, 1 1 1 and processed results into features for learning Introduction 1 1 1 Introduction algorithm Outline Outline Convolutions Convolutions But how do we know if our pre-defined kernels are best Apply same approach and divide by 9: Examples Examples for the specific learning task? Use in Feature Use in Feature Extraction Extraction CNNs CNNs Convolutional nodes in a CNN will allow the network to Example Example learn which features are best to extract Architectures Architectures We can also have the network learn which invariances are useful 7 / 26 8 / 26 Convolutions Basic Convolutional Layer Connectivity CSCE CSCE Imagine kernel represented as weights into a hidden 479/879 479/879 Lecture 4: Lecture 4: layer Convolutional Convolutional Neuron at row i , column j connects to previous layer’s rows i Neural Neural Output of a linear unit is exactly the kernel output Networks Networks to i + f h − 1 and columns j to j + f w − 1 If instead use, e.g., ReLU, get nonlinear transformation Stephen Scott Stephen Scott of kernel Introduction Introduction Outline Outline Convolutions Convolutions CNNs CNNs Basic Convolutional Basic Convolutional Layer Layer Pooling Pooling Complete Network Complete Network Example Example Architectures Architectures Apply zero padding at boundary Note that, unlike other network architectures, do not have complete connectivity ⇒ Many fewer parameters to tune 9 / 26 10 / 26 Convolutions Basic Convolutional Layer Downsampling: Stride Convolutional Stack CSCE CSCE 479/879 479/879 Can reduce size of layers by downsampling with a stride Lecture 4: Lecture 4: Convolutional Convolutional parameter Often use multiple convolutional layers in a convolutional Neural Neural Networks Networks stack Stephen Scott Stephen Scott Introduction Introduction Outline Outline Convolutions Convolutions CNNs CNNs Basic Convolutional Basic Convolutional Layer Layer Pooling Pooling Complete Network Complete Network Example Example Architectures Architectures Extends a higher-layer node’s receptive field Neuron at row i , column j connects to previous layer’s rows is h to is h + f h − 1 and columns js w to js w + f w − 1 11 / 26 12 / 26

Basic Convolutional Layer Basic Convolutional Layer Parameter Sharing Multiple Sets of Kernels CSCE CSCE Weight sharing forces the convolution layer to learn a 479/879 479/879 Lecture 4: Lecture 4: specific feature extractor Convolutional Convolutional Neural Neural To learn multiple extractors simultaneously, can have Networks Networks multiple convolution layers Stephen Scott Stephen Scott Sparse connectivity from input to hidden greatly Each is independent of the other reduces paramters Introduction Introduction Each uses its own weight sharing Outline Outline Can further reduce model complexity via parameter Convolutions Convolutions sharing (aka weight sharing ) CNNs CNNs Basic Convolutional E.g., weight w 1 , 1 that multiplies the upper-left value of Basic Convolutional Layer Layer Pooling Pooling the window is the same for all applications of kernel Complete Network Complete Network Example Example Architectures Architectures 13 / 26 14 / 26 Basic Convolutional Layer Basic Convolutional Layer Multiple Sets of Kernels Multiple Sets of Kernels CSCE CSCE 479/879 479/879 Let z ijk be output of node at row i , column j , feature map Can also span multiple channels (e.g., color planes) Lecture 4: Lecture 4: Convolutional Convolutional k of current layer ` Neural Neural Networks Networks Let s h and s w be strides, receptive field be f h × f w , and A neuron’s receptive field Stephen Scott Stephen Scott let f n 0 be number of feature maps in layer ` − 1 now spans all feature Introduction Introduction Let x i 0 j 0 k 0 be output of layer- ( ` − 1 ) node in row i 0 , maps of previous layer Outline Outline column j 0 , feature map (channel) k 0 Neuron at row i , column j Convolutions Convolutions of feature map k of layer Let b k be bias term for feature map k and w uvk 0 k be CNNs CNNs Basic Convolutional Basic Convolutional ` connects to layer weight connecting any node in feature map k 0 , position Layer Layer Pooling Pooling ( ` − 1 ) ’s rows is h to ( u , v ) , layer ` − 1 , to feature map k in layer ` Complete Network Complete Network Example is h + f h − 1 and columns Example Architectures Architectures f h � 1 f w � 1 f n 0 � 1 js w to js w + f w − 1 , X X X z ijk = b k + x i 0 j 0 k 0 w uvk 0 k spanning all feature maps of layer ` − 1 u = 0 v = 0 k 0 = 0 where i 0 = is h + u and j 0 = js w + v 15 / 26 16 / 26 Pooling Pooling CSCE CSCE 479/879 479/879 Lecture 4: Lecture 4: Convolutional Convolutional To help achieve Neural Neural Networks Networks translation invariance Stephen Scott Stephen Scott and reduce complexity, can feed Introduction Introduction output of neighboring Outline Outline Convolutions convolution nodes Convolutions CNNs CNNs into a pooling node Basic Convolutional Basic Convolutional Layer Layer Pooling function Pooling Pooling Typically pool each channel independently (reduce Complete Network Complete Network typically unweighted dimension, not depth), but can also pool over depth and Example Example Architectures max or average of Architectures keep dimension fixed inputs 17 / 26 18 / 26