Neural Nets for Adaptive Filter and Adaptive Neural Nets as - PowerPoint PPT Presentation

Neural Nets for Adaptive Filter and Adaptive Pattern Recognition Brian Young Article Context Neural Nets for Adaptive Filter and Adaptive Neural Nets as Adaptive Filters Pattern Recognition Adaptive Filters Min. Disturb. and LMS Adalines and Madalines Brian Young MRII Solution Algorithm btyoung@gmail.com Example Implementation CSCE 636 Conclusions 10 February 2010

Neural Nets for Outline Adaptive Filter and Adaptive Pattern Recognition Brian Young Article Context Article Context Neural Nets as Adaptive Filters Neural Nets as Adaptive Filters Adaptive Filters Min. Disturb. and Adaptive Combiners and Filters LMS Adalines and Madalines Minimal Disturbance and the LMS Algorithm MRII Adalines and Madalines Solution Algorithm Example Implementation Conclusions Madaline Rule II (MRII) Solution Algorithm Example Implementation Conclusions

Neural Nets for Article Context Adaptive Filter and Adaptive Pattern Recognition Brian Young Article Context Neural Nets as Adaptive Filters Adaptive Filters ◮ Published 1988 in IEEE Journals Min. Disturb. and LMS ◮ Bernard Widrow and Rodney Winter Adalines and Madalines ◮ Widrow was an EE professor at Stanford MRII Solution Algorithm ◮ Background in adaptive filtering and control Example Implementation ◮ Developed the LMS Algorithm Conclusions ◮ Specific algorithm isn’t referenced often.

Neural Nets for Adaptive Combiners and Filters Adaptive Filter and Adaptive Pattern The Adaptive Linear Combiner (ALC) Recognition Brian Young Article Context Neural Nets as Adaptive Filters Adaptive Filters Min. Disturb. and LMS Adalines and Madalines MRII Solution Algorithm Example Implementation Conclusions ◮ Weighted inputs are summed w T x = y ◮ Adaptive algorithm works to minimize ǫ = d − y error ǫ ◮ Essentially an SLP with a single linear output

Neural Nets for Adaptive Combiners and Filters Adaptive Filter and Adaptive Pattern The Adaptive Filter (AF) Recognition Brian Young Article Context ◮ Digitizes a single input to Neural Nets as Adaptive Filters feed into an ALC Adaptive Filters Min. Disturb. and ◮ Requires some knowledge LMS Adalines and Madalines of the required output MRII ◮ In general this requires Solution Algorithm Example Implementation some data that can be Conclusions correlated to the unknown output ◮ Well-used in industrial applications – not a ‘toy’ technique

Neural Nets for Adaptive Combiners and Filters Adaptive Filter and Adaptive Pattern Adaptive Filter example - Noise removal Recognition ◮ Adaptive filter attempts Brian Young to minimize output noise Article Context from an external source Neural Nets as Adaptive Filters ◮ Filter algorithm tries to Adaptive Filters Min. Disturb. and minimize total output LMS Adalines and Madalines square magnitude MRII ◮ Filter only has Solution Algorithm Example Implementation information on noise, so Conclusions can only reduce the noise E [ ǫ 2 ] E [ s 2 ] + 2 E [ s ( n 0 − y )] + E [( n 0 − y ) 2 ] = E [ s 2 ] + E [( n 0 − y ) 2 ] = ◮ n 0 is related to n 1 , but not necessarily the same ◮ The filter finds the the mapping from n 0 to n 1

Neural Nets for Minimal Disturbance and the LMS Algorithm Adaptive Filter and Adaptive Pattern Minimimal Disturbance Principle Recognition Brian Young Article Context Neural Nets as Adaptive Filters Adaptive Filters ◮ Consider a system with more variables than constraints Min. Disturb. and LMS Adalines and ◮ Infinitely many solutions that can fulfill constraint Madalines MRII ◮ How do you pick a solution? Solution Algorithm Example Implementation Assuming that the current set of parameters is not too far Conclusions from the solution, the best choice is the one that minimizes the change in the adaptive parameters.

Neural Nets for Minimal Disturbance and the LMS Algorithm Adaptive Filter and Adaptive Pattern Example - Smartphone accelerometer calibration Recognition Brian Young 2 3 2 3 2 3 2 3 x ˆ S 11 S 12 S 13 x b 1 3 Sources of Error: 5 = 5 + Article Context ˆ y 0 S 22 S 23 y b 2 4 4 5 4 4 5 z ˆ 0 0 S 33 z b 3 ◮ Scaling error Neural Nets as Adaptive Filters ◮ Offset error Subject to: Adaptive Filters x 2 + ˆ y 2 + ˆ z 2 − 1 = 0 ◮ Non-orthogonality f (ˆ x ) = ˆ Min. Disturb. and LMS Adalines and Madalines Now we minimize MRII Solution Algorithm Example X X ∆ S 2 ∆ b 2 H = ij + i + λ f (ˆ x + ∆ˆ x ) Implementation Conclusions Taking partials ∂ H ` ´ = 2∆ S ij + 2 λ x i x j + ∆ˆ ˆ x j = 0 ∆ S ij ≈ − λ x i ˆ x j → ∂ ∆ S ij ∂ H = 2∆ b i + 2 λ (ˆ x i + ∆ˆ x i ) = 0 ∆ b i ≈ − λ ˆ x i → ∂ ∆ b i

Neural Nets for Minimal Disturbance and the LMS Algorithm Adaptive Filter and Adaptive Pattern Example - Smartphone accelerometer calibration Recognition Brian Young Now solve for the λ that sets f (ˆ x + ∆ˆ x ) to zero: Article Context Neural Nets as x T ∆ ˆ x T ∆ ˆ λ 2 ∆ ˆ x T ˆ Adaptive Filters x + 2 λ ˆ x + ˆ x = 0 Adaptive Filters Min. Disturb. and LMS This is a quadratic equation and easily solved. Apply a multiplicative learning Adalines and Madalines factor α and make changes: MRII Solution Algorithm Example Implementation Conclusions

Neural Nets for Minimal Disturbance and LMS Algorithm Adaptive Filter and Adaptive Pattern LMS Algorithm Recognition Brian Young Generalizing this for the system y = Wx , with teaching Article Context vector d , we define a cost function Neural Nets as Adaptive Filters ∆ W 2 + ǫ T ǫ Adaptive Filters � � J = Min. Disturb. and LMS Adalines and Madalines ǫ = d − ( W + ∆ W ) x MRII Solution Algorithm Example Taking the gradient WRT ∆ W : Implementation Conclusions ∂ J ∂ ∆ W = 2∆ W + 2 ǫ x T = 0 Applying a learning rate of α yields the delta rule : ∆ W = − α ǫ x T

Neural Nets for Adalines Adaptive Filter and Adaptive Pattern Adaptive Channel Equalizer Recognition Brian Young ◮ Transmission lines Article Context ‘smear’ signals with an Neural Nets as Adaptive Filters unknown impulse Adaptive Filters response Min. Disturb. and LMS Adalines and ◮ Directly quantizing the Madalines MRII sample signal would not Solution Algorithm Example yield the right result Implementation Conclusions ◮ Placing a quantizer outside of an adaptive filter can improve the Sample Signal: precision ◮ Quadruples data rate ◮ Looking even more like a neural net

Neural Nets for Adalines Adaptive Filter and Adaptive Pattern Recognition If we strip away all of the application specific components of Brian Young the Adaptive Channel Equalizer, we get the Adaptive Linear Neuron (Adaline): Article Context Neural Nets as Adaptive Filters Adaptive Filters Min. Disturb. and LMS Adalines and Madalines MRII Solution Algorithm Example Implementation Conclusions ◮ Simply an ALC with a quantizer ◮ Note that errors off of y , not q ◮ Allows it to become “more” correct

Neural Nets for Madalines Adaptive Filter and Adaptive Pattern Recognition Combining multiple Adalines is a natural extension, creating Brian Young a Madaline: Article Context Neural Nets as Adaptive Filters Adaptive Filters Min. Disturb. and LMS Adalines and Madalines MRII Solution Algorithm Example Implementation Conclusions Note how the second layer is fixed

Neural Nets for Madalines Adaptive Filter and Adaptive Pattern Recognition Brian Young This creates multiple intersecting hyperplanes. In a 2-D Article Context boolean space: Neural Nets as Adaptive Filters Adaptive Filters Min. Disturb. and LMS Adalines and Madalines MRII Solution Algorithm Example Implementation Conclusions

Neural Nets for Madalines Adaptive Filter and Adaptive Pattern Not a far leap to an MLP Recognition Brian Young Article Context Neural Nets as Adaptive Filters Adaptive Filters Min. Disturb. and LMS Adalines and Madalines MRII Solution Algorithm Example Implementation Conclusions “Invented” a neural network without mentioning the brain!

Neural Nets for The Hammer Principle Adaptive Filter and Adaptive Pattern When you have a hammer... Recognition Brian Young Article Context Neural Nets as Adaptive Filters ◮ Here we have derived a multi-layer perceptron from an Adaptive Filters Min. Disturb. and adaptive filter LMS Adalines and Madalines ◮ AFs are human-engineered solutions rather than a MRII biological model Solution Algorithm Example Implementation ◮ However, this approach leads the author to Conclusions over-engineer applications, as we will see Also note that minimal disturbance is a new take on least-squares minimization.

Neural Nets for Madaline Rule II Adaptive Filter and Adaptive Pattern Another problem with backpropogation Recognition Brian Young Article Context Neural Nets as Adaptive Filters ◮ The backpropogation algorithm requires a differentiable Adaptive Filters Min. Disturb. and threshold function. LMS Adalines and Madalines ◮ This can make digital implementation difficult MRII Solution Algorithm ◮ Paper proposes a method that allows training of hidden Example Implementation units with an ideal step function Conclusions This already begs the question “Is this necessary” given that backpropogation has been well implemented and is the de facto standard.