Lecture Maximization Expectation 12 . Variational Inference - PowerPoint PPT Presentation

Lecture Maximization Expectation 12 . Variational Inference Scribes Daniel Zeiberg : Alesia Chernihova

6mm Maximum Likelihood Estimation in " observed " Easy Estimate for t : n = & - § acy Am log fly 2- ply o ) 77 ) , , ? Problem pcyly Need marginalize to : over Z ) = ldtplu.tt/y1=n7!/dznpCyn.7nly7oQylogpCy1yl=ptqy,fqnI ldznexplyitly , aint - , throws Integral worms in spanner

Maximization Expectation * .gg?gxlogpCy11u.E,n * n.ME ) Objective : * a = Repeat trnhynynt ' unchanged ) convergence until ( objective For 1 Ni 7 in n . . . - / dznpcznlyn.io , ) ) IlZn=h Ilan - h ] El ) yuh = - ie he Points # cluster in For W Ki 2. in I , . . . N N Empirical I Nh E t Mu yn ruh yuh :-. = = µ h ' Mean h h 't ! & - pryuht Ih Empirical = , Covariance h Nh IN cluster Fraction nu = in

Maximization Expectation Example : Iteration O :

Maximization Expectation Example : Iteration 1 :

Maximization Expectation Example : 2 Iteration :

Maximization Expectation Example : 3 Iteration :

Maximization Expectation Example : 4 Iteration :

Maximization Expectation Example : 5 Iteration :

Maximization Expectation Example : 6 Iteration :

Jensen Inequality Intermezzo : 's , Functions Convex Area above fix f- ( tx t ) ) fkn Ci . • t xz curve is a - , . . , set convex • fix t ) flat t s It fix , ) ' t . g - X Xz , Functions Concave Area below •,←¥f f- ( fix flat t ) tx Ci ) t ve is a xz cu - - , set convex - fix , , fix t ) flat t 7 It , ) + s - Xz X , Random Variables Corrolary : t.ci#xnl:Efii.i:iit::::.

Lowen Likelihoods Bounds on Use to Bound Lower Jensen define Idea inequality 's : Marginal fax fax Gaussian 4¥ , 2- goal E ) aix six , = i - = - lost slog It # . I I , 14 L . tog 't E. * : - - - ⇐ " [ Lower bound 7 boy on Mixture Model ; y , PgY¥ = Id 't I 2- I O ) :O ) pig . 't at ;o , act pig ; = = , 7 ;D ) pig log , I log Ez £10,81 , ) pay ol s ; :-. .mg

Leibler Kullback Divergence Intermezzo lax : - - I 9¥ KL( qcxsll MIX ) ) Measures how by much DX 941 : ' " n deviates from MIN , ,× a Properties ( Positive femi ) ) - definite ) ( a call I KL mix 3 o . " 9¥ KL ( g MIN ) ⇐ galley "g , ) E. kill leg guy - = = , ± log ( E⇐g*l" gift ) lil log o - = - KL ( q C x ) 11171×1 ) 91×1 Mk ) 2 o = a = . I 9kt lax 171×7 , log 94¥ leg dxqix Mix → , o = , ,

KL Bound Lower divergence vs - ;o ) pcyit = :O 'P P 's 't 's :o) Eagan , lloypggjt.sc ] £10,21 = + leg Plaything ) #z~q , ;y)|log :o) pig = on £ Kttdiv depend does not rewrite as log an ;n|log9pYftTo , ] # :o) = pig - ← KL ( ;o ) ) log H :O) paly ply qhsy ) - = a \ Does depend Depends not y on on y Maximizing £ ( O ,y ) Implication equivalent : y is wrt O ) ) to ( KL 11 minimizing ; g) pcttly 917 ; o .

Algorithm Generalized Expectation Maximization : Egj Lto Objective .hr : . , 9175g ) A ; Initialize £10 Repeat , y ) until unchanged : .gg/loyPlY'tt-9/slogpcy;o7 Expectation Step 1 . Computes expected I ( O 8 ) y ang mate = sufficient statistics , r Step Maximization 2 . Maximizes O given L( O 0,8 ) anymore = computed statistics

Algorithm Generalized Expectation Maximization : Egj Lto Objective Hi : , . , 9175g ) O qc.zi-hsyl-ku-pf2-n-hlyn.cl : Initialize £10,8 ) Repeat until unchanged " EE ) ) q moments ttt .gg/loyPlY'tt-9/slogpcy;o7 Expectation Step distribution 1 determine , ) ) 8 I I o angginkhfgct.gl//pC71yiO r ) = = angngax , Step Maximization 2 . LIO D= r ) anymore ,

! yall ainu Mtf Parameters Update Maximization Step : . .gs/logPg::?IT ] . rt a Ea , n = tdyiz ) K µ f \ , E leg , thlynl Ithih ] ) Iftar ) ) ply I it y = - htt ) I = Ig go.gg/bgpisitin r ) 9. = ? N fthlynl-g-YEgn.nl#zn=hH=n.Etulyn1rnu ) Nh 844yd -

acyu Mt Parameters Update Maximization Step : . ,nflogPgY÷ ] . rt a Ea , n = K µ E f. leg , tulynl Ithih ) ) Iftar ) g) ply I it = - htt = ( I ? tulyn ) r ) run ) Cly Iq Nh 844yd o = - , h Maximum Match Likelihood to suff stats moments expected : ftp..mn/thl5tY--oa-M--fuE?tulynlrnu bye

Algorithm Generalized Expectation Maximization : Egj Lto Objective Hi : , . , ; y ) 917 O qc.zi-hsyl-ku-pftn-hlyn.cl : Initialize £10,8 ) Repeat until unchanged " EE ) ) q Moments ttt .gg/loyPlYitt-9/slogpcy;o7 Expectation Step distribution 1 determine , ) ) 8 I angginkhfgct.gl//pC71yiO r ) C a = = angngax , Step Maximization z . suff use computed stats - . update to parameters L ( O D= r ) moments by anymore matching ,

Variational Inference Approximate by posterior Idea maximizing , bound lower Ptt variational Ely ply ? ) a , , * go.o.cn/eogPgY.if?T ) " 9) = llogpqlcz.co#)--legpcys-KL/9lZ.osoDllpcz.o1ys 7,0 ly log ply ) = Eg t o ; ) q L Maximizing lol ) the log is s pay , KL minimizing Same as

: Minimizing Intuition KL divergences pcx PCYIX , ,×z ) Ply . ) ) × ,X ,xz / = , , , g) Z ;µ 2) Norm ( qcx ) x = , , ,×z qc ,× , q(× )q( , × ÷ . , /× , ,6 ? ) Norm , ) qcx ;µ ÷ , Normkn ;µz,6i ) qkz ) := approximates LC EaaflogPlgYIlI@ply.x ) aim :-. KL ( q( ) ,×z ) ) , leg Hpcx , ,×zly ply ) ,×ziy ) ,x pcyipix , ) x = = - , , Intuition KL divergence : under variance

: Minimizing Intuition KL divergences 1- > ) pcylx ,×up( P( ) , ,x Yixi ,×z g = , g) z [ ) p ( Norm a. ( ) x x ; = , , ,×z qk qkz qc ,× , ) , , ÷ pagan Norm ( Propagate , ) ,6 × ;µ qlx , ) :-. , , µorm( , G) 91×21 ;µz Xz :-. ,xz ) ) KL( plx 119k . ,xzly ) * , = |d× 9k I ,×z ) ) , KL( 1 log Hpcx dx - ) qlx ,xz ,xzly qk ,xz , ,Xz1y ) . . plx , , , leg 'f ligng log ph lipm , a q o as = = . Intuition whenever ( × , ,xz : q ,xe ) ly )→0 → o PC x ,

Variational Algorithm : Expectation Maximization g C O 's 90 ) ; g) Define 't , O gets ) qcz : Of = "q¥to :L ' I slog , do ) flog Clot Et Objective ply , = qiao , - , 00 ) I left ( change Repeat until smaller threshold ) converges than some Expectation Step 1 . ( oligo ) lot L Analogous to EM argy.mx = step for j Step Maximization z . Updates distribution £ ( 97,010 ) 010 ; go ) angurax instead of = glo 40 estimate O point

( Simplified ) Gaussian Example Mixture : f ! si is :/ I Generative Model Norm ( pro ,d huh ) , d So ~ , EI . ,Yk ) ( YK Discrete 2- - n . . , Norml ) ynl7n=h pea n ,

" Margined Average likelihood Model Selection µ " livelihood Evidence log ldtdoply.z.io ) log £ log pigs I ply ) = I • K=2 I Clusters of Number by keeping Can Intuition model fitting avoid : over highest £ with

Gaussian Mixture Derivation Updates of : of of I E go.ngimm.si/bsPgIYIYgTp..m,s 9 9 Ltr , ) miss = . ) E = gcaayuillogpcy.7.ms/-EgmlloylgiIY-Eqyuslloglgii-j depends depends depends on S y m , s an y on m , , KL ( que ) llpqul ) KL ( 11pm ) gets - - I Solve E step a = : - or ¥ M Solve step Ofm 0 = o = - : ,

Gaussian Mixture Derivation Updates of : Idea Exploit Families Exponential : , y ) ) Eaczsgcy , flog algal ) guy It ) ply - hiyqexpfyuttlyn , I &!£!IHn=h ) log pcynlzi-h.lu ) ] Eta = # Eam , feign ) ) Ego ,fIha=hD crplyittlyni - = . 017=8 I T T 0/7=4,5 ) depends depends on on

( Simplified ) Gaussian , EI Example Mixture ! ! : " s :/ I Generative Model " f Variational i Distribution Norm ( pro ,d µ ) h , d So 9441917 ) ~ 7) gyu = , , . ,Yk ) ( 11k Discrete 2- 9171 917187 n = n . . , .NL/uuimu,5u I ? Norml ) ynl7n=h ) 9 ( in pea n = h - step E plynltn-h.MY ) !µ , I log exp I Eg ✓ a uh M i step - N Tywyn si 1- mo mu t = = ( ÷ . ) frm mi t