Project Wednesday Apr 25 : Apr Project Reports : Friday 27 - - PowerPoint PPT Presentation

SLIDE 1 Lecture 22 : Review Homework 4 : Due Friday Exam : Next Wednesday ( 18 Apr ) Project Presentations : Wednesday 25 Apr Project Reports : Friday 27 Apr Lecture Notes : Will release today ( if submitted )

SLIDE 2 https://course.ccs.neu.edu/cs7140sp18/exam-topics.html Exam Topic List

SLIDE 3 Review : Minimizing KL Divergences 2

f

Target Density :

)

Pc × ' 1×2 ' Y ) × MY '×' 1×21 Approximation : q ( × , ,×z I ¢ ) ( under = µ ( x ; ¢ " , ¢£ ) approximates variance ) arg min 1<2 ( qcx , ,×z;¢ ) 11 pcx , ,×z1 y ) ) ¢ EPILBP arg min KL ( PC × , ,×zly ) H qk , ,xz :O) ) (

• ven
• ¢

approximates variance )

SLIDE 4 Review : Minimizing KL Divergences 2

f

Target Density :

)

Pc × ' 1×2 1 Y ) × MY '×' 1×21 Approximation : q ( × , ,×z 1 4 ) ( under =

• g. ( ×

, ;¢ , ) qkz ;¢z ) approximates variance ) argmin K2( qcx , ,×z ;¢ ) 11 pcx , ,×z1y ) ) ¢ EPILBP arg min KL ( PC × , ,×zly ) H qk , ,xz :O) ) (

• ven
• ¢

approximates variance )

SLIDE 5 Review : Vaniaticnal Inference vs Max Likelihood Expectation maximization p(×,z

;g

) L ( 01,01 = #

an

;¢,[log

qcz ;¢ , ] = log pc .

9)

• KL(q(

7 ;¢ ) 11 pczlx ;O ) ) = log PC x ;D ) When ql 7 ;¢ ) = PAK ;O ) Variation Inference ( true after M

• step )

LC10,41 = # [ log PK 't

't

' ' ] 9C 7,0 ;¢ ) ql 7,0 ;¢ ) In Do not

• ptimize

= leg

plxsd

)

• KL(917,0

;¢ ) A plot ,Olx :D ) )

SLIDE 6 Review : Vaniaticnal Inference vs Max Likelihood Example Question : ( Hand )

• •
• Stretch

the EM bound

• and

The VBEM bound

• as

a function

• f

K EM VBEM c ) L 1 2 34 56 789 k 1 2 34 56 789 k

SLIDE 7 Review : Vaniaticnal Inference vs Max Likelihood Example Question : ( Hand )

• •
• Stretch

the EM bound

• and

the VBEM bound

• as

a function

• f

K EM VBEM c ) L

+t#t

t 1 2 34 56 789 K 1 2 34 56 789 k

SLIDE 8 Aside ' . Maximum Likelihood as KL minimization Data ; Xn ~ PDATA ( x ) n = n , . . . , N Model : plx :O ) = |d7 pcx , 7 :O ) KL divergence ;

• KL

( pot 't ( x ) 11 pcx ;o ) ) = E poatak , [ log pcx :o)

• log

pPHA(× , ] = I, §Y log pcxn ;o )

+

H . # Ix ]

SLIDE 9 Review : Minimizing KL Divergences KL ( a 11 p ) KL .( pllq ) KL( PDATAHP )

• Variation

al EM

• Loopy

belief propagation

• Stochastic

VI

• Expectation

propagation

• Black
• box

VI

• Expectation

maximization

• VAES

( max likelihood ) ( encoder ) . ✓ AES ( decoder )

SLIDE 10 Review : Conjugate Priors Likelihood : pcxiy ) = hcxl exp 1 yttcx )

• acy

) ] Conjugate prior : pcyli ) = hc g) eipfytd , . acrpiz . au )] Joint : pl

• x. y )

x exp[ ] = heap

,c

yi E) exp{ acts

• at

) ] Posterior pcylx ) = pcy 15) Marginal Likelihood pcx ) = exp [ ac 5)

• AN

) ]

SLIDE 11 VBEM Updates Using Functional Derivatives Functional Derivatives : Differentiate an integral with respect to a function L[ acx ) , bk ) , ccxi ) = |dx ( acx > bcx , + c ( x ) ) Idea : " Differentiate " integrand wnt functions 81 i = bcx ,

• h

= and

• k

= 1 fak , Jbcx ) Ek ,

SLIDE 12 VBEM Updates Using Functional Derivatives Joint log plx , 7,0 ) = log

pcxiz

,O ) + log

pczlo

) + log pl O ) Objective PCX , 7,0 ) LGHI ,a( d) :

Eqa

, qco , [ log

aca

) = E qa

,q(m[

log

pcx

, 7,0 ) ]

• #

an

,[ leg

got , ] Functional Derivatives

• #

a ,• , [ log qco ) ] §§µ = Eta ,•)[ log PK , 7,0 ) ]

• leg

qczs

• i

=

SLIDE 13 VBEM Updates Using Functional Derivatives Lcqataca) : Eamon . ,[ log PgYI'}Yh- ) = Eg , , ,q,o , [ log

pcxitt

) ]

• Eqn

. ,[ loggia ] Functional Derivatives

• #

qc a) [ log qco ) ] §§µ = # a ,o , [ log plat ,0 ) ]

• log

qh . )

• 1

=

• 8¥ .

=

Etqcz

, [ leg

plx

, 7,0 ) )

• log

9101

. 1

SLIDE 14 VBEM Updates Using Functional Derivatives Functional Derivatives §§µ = # a ,o , [ log pcx , 7,0 ) ]

• log

qh . )

• 1

=

• fgffm

= El qc 7) [ leg plx , 7,9 ) )

• log

910 )

• 1

= c Updates gas x exp [ Etqco ) [ log pc x. 7,0 ) ] ] qlo ) × exp [ # qn . , I log pix , 7,9 ) ] ]