Processing Expectation Maximization Mixture Models Bhiksha Raj - PowerPoint PPT Presentation

Machine Learning for Signal Processing Expectation Maximization Mixture Models Bhiksha Raj Class 10. 3 Oct 2013 3 Oct 2011 11755/18797 1

Administrivia • HW2 is up – A final problem will be added – You have four weeks – It’s a loooooong homework – About 12-24 hours of work • Does everyone have teams/project proposals • Begin working on your projects immediately.. 3 Oct 2011 11755/18797 2

A Strange Observation The pitch of female Indian playback singers • A trend is on an ever-increasing trajectory 800 Alka Yangnik, Dil Ka Rishta Lata Mangeshkar, Anupama Peak: 740 Hz Pitch (Hz) Peak: 570 Hz 600 400 Shamshad Begum, Patanga Peak 310 Hz 1949 1966 2003 Year (AD)  Mean pitch values: 278Hz, 410Hz, 580Hz 3 Oct 2011 11755/18797 3

I’m not the only one to find the high-pitched stuff annoying • Sarah McDonald (Holy Cow): “.. shrieking…” • Khazana.com: “.. female Indian movie playback singers who can produce ultra high frequncies which only dogs can hear clearly.. ” • www.roadjunky.com : “ .. High pitched female singers doing their best to sound like they were seven years old .. ” 3 Oct 2011 11755/18797 4

A Disturbing Observation The pitch of female Indian playback singers • A trend is on an ever-increasing trajectory Glass Shatters 800 Alka Yangnik, Dil Ka Rishta Lata Mangeshkar, Anupama Peak: 740 Hz Pitch (Hz) Peak: 570 Hz 600 400 Shamshad Begum, Patanga Average Female Peak 310 Hz Talking Pitch 1949 1966 2003 Year (AD)  Mean pitch values: 278Hz, 410Hz, 580Hz 3 Oct 2011 11755/18797 5

Lets Fix the Song • The pitch is unpleasant • The melody isn’t bad • Modify the pitch, but retain melody • Problem: – Cannot just shift the pitch: will destroy the music • The music is fine, leave it alone – Modify the singing pitch without affecting the music 3 Oct 2011 11755/18797 6

“Personalizing” the Song • Separate the vocals from the background music – Modify the separated vocals, keep music unchanged • Separation need not be perfect – Must only be sufficient to enable pitch modification of vocals – Pitch modification is tolerant of low-level artifacts • For octave level pitch modification artifacts can be undetectable. 3 Oct 2011 11755/18797 7

Separation example Dayya Dayya original (only vocalized regions) Dayya Dayya separated music Dayya Dayya separated vocals 3 Oct 2011 11755/18797 8

Some examples  Example 1: Vocals shifted down by 4 semitonesExample 2: Gender of singer partially modified 3 Oct 2011 11755/18797 9

Some examples  Example 1: Vocals shifted down by 4 semitones  Example 2: Gender of singer partially modified 3 Oct 2011 11755/18797 10

Techniques Employed • Signal separation – Employed a simple latent-variable based separation method • Voice modification – Equally simple techniques • Separation: Extensive use of Expectation Maximization 3 Oct 2011 11755/18797 11

Learning Distributions for Data • Problem: Given a collection of examples from some data, estimate its distribution • Solution: Assign a model to the distribution – Learn parameters of model from data • Models can be arbitrarily complex – Mixture densities, Hierarchical models. • Learning must be done using Expectation Maximization • Following slides: An intuitive explanation using a simple example of multinomials 3 Oct 2011 11755/18797 12

A Thought Experiment 6 3 1 5 4 1 2 4 … • A person shoots a loaded dice repeatedly • You observe the series of outcomes • You can form a good idea of how the dice is loaded – Figure out what the probabilities of the various numbers are for dice • P(number) = count(number)/sum(rolls) • This is a maximum likelihood estimate – Estimate that makes the observed sequence of numbers most probable 3 Oct 2011 11755/18797 13

The Multinomial Distribution • A probability distribution over a discrete collection of items is a Multinomial  ( : belongs to a discrete set ) ( ) P X X P X • E.g. the roll of dice – X : X in (1,2,3,4,5,6) • Or the toss of a coin – X : X in (head, tails) 3 Oct 2011 11755/18797 14

Maximum Likelihood Estimation n 2 n 4 n 1 n 5 n 6 n 3 p 6 p 3 p 4 p 1 p 2 p 2 p 4 p 5 p 1 p 5 p 6 p 3 • Basic principle: Assign a form to the distribution – E.g. a multinomial – Or a Gaussian • Find the distribution that best fits the histogram of the data 3 Oct 2011 11755/18797 15

Defining “Best Fit” • The data are generated by draws from the distribution – I.e. the generating process draws from the distribution • Assumption: The world is a boring place – The data you have observed are very typical of the process • Consequent assumption: The distribution has a high probability of generating the observed data – Not necessarily true • Select the distribution that has the highest probability of generating the data – Should assign lower probability to less frequent observations and vice versa 3 Oct 2011 11755/18797 16

Maximum Likelihood Estimation: Multinomial • Probability of generating (n 1 , n 2 , n 3 , n 4 , n 5 , n 6 )   n ( , , , , , ) P n n n n n n Const p i 1 2 3 4 5 6 i i • Find p 1 ,p 2 ,p 3 ,p 4 ,p 5 ,p 6 so that the above is maximized • Alternately maximize        log ( , , , , , ) log( ) log P n n n n n n Const n p 1 2 3 4 5 6 i i i – Log() is a monotonic function – argmax x f(x) = argmax x log(f(x)) • Solving for the probabilities gives us EVENTUALLY n – Requires constrained optimization to  i p ITS JUST  i ensure probabilities sum to 1 n COUNTING! j j 3 Oct 2011 11755/18797 17

Segue: Gaussians   1  X  m Q    m Q  m 1 T ( ) ( ; , ) exp 0 . 5 ( ) ( ) P X N X X  Q d ( 2 ) | | • Parameters of a Gaussian: – Mean m , Covariance Q 3 Oct 2011 11755/18797 18

Maximum Likelihood: Gaussian  Given a collection of observations ( X 1 , X 2 ,…), estimate mean m and covariance Q   1      m Q  m 1 T ( , ,...) exp 0 . 5 ( ) ( ) P X X X X 1 2 i i  Q d ( 2 ) | | i           Q   m Q  m 1 T log ( , ,...) 0 . 5 log | | ( ) ( ) P X X C X X 1 2 i i i • Maximizing w.r.t m and Q gives us ITS STILL 1 1      JUST m  Q   m  m T X X X i i i COUNTING! N N i i 3 Oct 2011 11755/18797 19

Laplacian  m   1 | | x  m     ( ) ( ; , ) exp P x L x b   2 b b • Parameters: Mean m , scale b ( b > 0) 3 Oct 2011 11755/18797 20

Maximum Likelihood: Laplacian  Given a collection of observations ( x 1 , x 2 ,…), estimate mean m and scale b  m | |   x     i log ( , ,...) log( ) P x x C N b 1 2 b i • Maximizing w.r.t m and b gives us 1 1   m    m | | x b x i i N N i i 3 Oct 2011 11755/18797 21

Dirichlet (from wikipedia) log of the density as we change α from α=(0.3, 0.3, 0.3) to (2.0, 2.0, 2.0), keeping all the individual αi's equal to each other.  K =3. Clockwise from top left:  a ( ) α =(6, 2, 2), (3, 7, 5), (6, 2, 6), (2, 3, 4) i  a   a  1 ( ) ( ; ) i P X D X x i   i   a   i • Parameters are a s i   i – Determine mode and curvature • Defined only of probability vectors – X = [x 1 x 2 .. x K ] , S i x i = 1, x i >= 0 for all i 3 Oct 2011 11755/18797 22

Maximum Likelihood: Dirichlet  Given a collection of observations ( X 1 , X 2 ,…), estimate a                 a    a   a   log ( , ,...) ( 1 ) log( ) log log P X X X N N   1 2 , i j i i i     j i i i • No closed form solution for a s. – Needs gradient ascent • Several distributions have this property: the ML estimate of their parameters have no closed form solution 3 Oct 2011 11755/18797 23

Continuing the Thought Experiment 6 3 1 5 4 1 2 4 … 4 4 1 6 3 2 1 2 … • Two persons shoot loaded dice repeatedly – The dice are differently loaded for the two of them • We observe the series of outcomes for both persons • How to determine the probability distributions of the two dice? 3 Oct 2011 11755/18797 24

Estimating Probabilities 6 4 5 1 2 3 4 5 2 2 1 4 3 4 6 2 1 6 … • Observation: The sequence of numbers from the two dice – As indicated by the colors, we know who rolled what number 3 Oct 2011 11755/18797 25

Estimating Probabilities 6 4 5 1 2 3 4 5 2 2 1 4 3 4 6 2 1 6 … • Observation: The sequence of numbers from the two dice – As indicated by the colors, we know who rolled what number 4 1 3 5 2 4 4 2 6.. 6 5 2 4 2 1 3 6 1.. • Segregation: Separate the blue Collection of “blue” Collection of “red” observations from the red numbers numbers 3 Oct 2011 11755/18797 26