Lecture Scribe Graphs Stochastic Computation 22 : Heiko Zimmermann :
Auto encoding Variational General Methods View : - weighted Ulp Auto encoding & Importance auto encoders SMC : EI [ IT Replace unbiased estimator Cx ) with pg W = £194 ) ftp.#qczixsllogwDsEp..llogpdxiIw=Po&I9pl71X - - ) n ↳ lower Replace band Wake methods sleep : - bound log with ) upper > , poles
Gradient Variational Inference Estimation in Reinforce Style - d t ) ) go , czixsl log # Pok Z ) 2%5×3 wo.pk ICO wo.pk , of ,zs= , a ) = , ddqo 94171×3 9¢CZgk,x ! ( Ig leg & & a) ( log At tbh ) Wo # dd-plogwo.pk " ith ) Z aaczks = go - , , Normal x ) µ¢lxlt6¢X E. ) E 2-41 Re : = parameterized . ) E ) ) ddp Epee , flog Pok dido L ( O C x. to ) , woo , Wo ,¢c× = = , , ) IX ) K If duplet , eh ) = K k - I -
Stochastic Computation Graphs ( 3 ago ) MNIST VAE far Structured Example lectures : ) Lionel Oo 9. = , o Data Decoder pocx 9,7 ) Encoder .gl#gcxifEh*.axflogPf;.1 golly 71×7 qcx ) , , -
Stochastic Computation Graphs ( 3 ago ) MNIST VAE far Structured Example lectures : To .gl#gcxsfEtqcy.zixflog+zx Do ( x ,7 ) ) ) , y £10,91 To ,o , = Represent loss graph computation Idea stochastic as : h y X , 9 Not parameterized : re ¢ ¢ , ) ) ( h Discrete , ( x y n ,
Stochastic Computation Graphs ( 3 ago ) MNIST VAE far Structured Example lectures : To .gl#gcxsfEtqcy.zixflog+zx Do ( x ,7 ) ) ) , y To ,qL( 0,91 = Represent loss graph computation Idea stochastic as : = 0¥ - log ""- Eh ' 0¥ en = 1h , ) 24 q ly , , 9 h y X , 9 Not parameterized : re ¢ , to , ) ) ( h Discrete , ( x y n
Stochastic Computation Graphs ( 3 ago ) MNIST VAE far Structured Example lectures : #\ To .gl#gcxsfEtqcy.zixflog+zx Do ( x ,7 ) ) ) , y £10,91 To ,o , = Represent loss graph computation Idea stochastic as : 9 pepanametvi red hz h y 2- : X , 2- ( hzlx.y.dz ) E ) , & to E p ( Es E n
Stochastic Computation Graphs ( 3 ago ) MNIST VAE far Structured Example lectures : 4/2 em To .gl#gcxsfEtqcy.zixflog+zx Do ( x ,7 ) ) ) , y £10,91 To ,o , = Represent loss graph computation Idea stochastic as : - log er e - - . dl¥z= , . . , 9 9 h ghq y x ; , d & Q E
Stochastic Computation Graphs ( 3 ago ) MNIST VAE far Structured Example lectures : y htt em To .gl#gcxsfEtqcs.zixflog+zx Do ( x ,7 ) ) ) , y £10,91 To ,o , = Represent loss graph computation Idea stochastic as : - log er e - - . . dl¥z= , 9 9 0¥ h ghq × ; , d z% +9¥ 19 Q E 092
Stochastic Computation Graphs ( 3 ago ) MNIST VAE far Structured Example lectures : (a) To .gl#gcxsfEtqcy.zixflog+zx Do ( x ,7 ) ) ) , y £10,91 To ,o , = Represent loss graph computation Idea stochastic as : h . / dot dlg I 9 g. n ha h z y X , , & Q E
Stochastic Computation Graphs ( 3 ago ) MNIST VAE far Structured Example lectures : #\ n it To .gl#gcxsfEtqcy.zixflog+zx Do ( x ,7 ) ) ) , y £10,91 To ,o , = Represent loss graph computation Idea stochastic as : Problem I Cannot : l take derivative . , value discrete it t w n . . . 9 9 :& ¥ of ?%÷¥ n s × . . & Q E
Stochastic Computation Graphs ( 3 ago ) MNIST VAE far Structured Example lectures : #) To .gl#gcxsfEtqcy.zixflog+zx Do ( x ,7 ) ) ) , y £10,91 To ,o , = i foil Represent loss graph computation Idea stochastic as : Problem :* :* I Cannot : l tame derivative . , value discrete t w - . . 9 9 t 1) 8¥ d daff hz h y 2- × , Problem 2 & : Q E tale Cannot stochastic value of derivative
Stochastic Computation Graphs ( 3 ago ) MNIST VAE far Structured Example lectures : To .gl#gcxsfEtqcy.zixflog+zx Do ( x ,7 ) ) ) , y #)#dl× £10,91 To ,o , = Represent loss graph computation Idea stochastic as : - leg paths l×= ) / 9 9 q - log ↳ he h h y 2- X = , , , - log ly %) 19 to a - E - 1h , ) 9 ly
Stochastic Computation Graphs ( 3 ago ) MNIST VAE far Structured Example lectures : ) ) ) To .gl#gcxsfEtlqo,eyixspcesfllxtlztly To ,qL( 0,9 ) = Represent loss graph computation Idea stochastic as : ,#dl× - leg paths l×= ) / 9 9 q - log ↳ he h h y 2- X = , , , - log ly %) Q to a - E - 1h , ) qty
Gradients Stochastic Computation Graphs of Idea Re Combine Reinforce gradients parameterized stale : and - E acxiaocsixspcail data I ÷ = - - leg paths l×= ) × / 9 9 q - log ↳ he h h y 2- X = , , , - log ly %) 19 to a - E - qlyih , )
Gradients Stochastic Computation Graphs of Idea Re Combine Reinforce gradients parameterized stale : and - E gcxiqocyixspcaif To , ) DL dlx .is#pcqfdel-oxz+dd-ozy } = - - re Parameterized DO , 1¥ # = - gag )dl× - leg paths l×= ) / 9 9 q - log ↳ he h h y 2- × = , , - log ly %) 9h to a - E - qlyih , )
Gradients Computation Graphs of Idea Re gradients Combine Reinforce parameterized stale : and - P' af To , ) dlx daff E = - gcxioocsix ' Reinforce Eacxiqocsixspcaildoztddoz style - Stochastic l ] d daff I = - , 9ksqcsixspceifddo.bg , ) ) ( lyte , ) ] dq text 1h DL # 9cg = - ,#dl× - leg paths l×= ) / 9 9 q - log ↳ he h h y 2- × = , , - log ly 1£ oh a to - E - qlyih , )
Gradients Stochastic Computation Graphs of Question ? What happens if add extra edge we an ; ftp. ! gcxiqocsixspcaifddqlogqcyih lxtlytl , ) ] , ) ) ( E - #)dl× - leg paths l×= ) / 9 9 q - log ↳ he h h y 2- × = , , - - log ly %) 19 a to - E - qlyih , )
Gradients Stochastic Computation Graphs of Question ? What happens if add extra edge we an ; gcxsqcsixspcafddo.bg lxtlytl , ) ) ( , ) Fp qcylh # = - , +8,9 +9¥ ) )dl× - leg paths l×= ) / 9 9 q - log ↳ he h h y 2- × = , , - - log ly %) 9h a to - E - qlyih , )
Gradients Stochastic Computation Graphs of Question ? What happens if add extra edge we an ; y ) ( Paths Sum Ideal paths over f . ftp.t#yt1(pathsaroId4 through gcxiqocsixspcaifddo.bg lxtlytl , ) ) ( , ) ¥p qcylh # = - , + ) )de× - leg paths l×= ) / 9 9 q - log ↳ he h h y 2- × = , , - - log ly %) 19 a to - E - qlyih , )
Gradients Stochastic Computation Graphs of General We loss compute derivatives of rules expected : can an to parameters from 1 Sum derivatives all paths loss nodes over . " bloch Stochastic " 2 gradients nodes . - style blocked 3 Use for paths reinforce derivative . 4 Use derivative for repanahreberised paths other , Vz 03 e.me#l3 V3 / 9 9 q l , tlztl du L des d dz V , = , - Oz O ,
Gradients 03 Stochastic Computation Graphs of IT Sun " " blocked paths V3 Vz IL ] over d ( E = Veith PLY Sum " " unblocked paths , over ; Epa I ' .mn/j..o?y.ddoibg9hil/Lta..?aeuado;ea)#.eTl3 / 9 9 q da L l , tlztl des d dz V , = , , → Oz O ,
Gradients Stochastic Computation Graphs of IT Sum " " blocked paths IL ] over d ( E = Veith PLY Sum " " unblocked paths , over , .mn/j..o?sy.dd-oibg9hil/Lta..?aeuad-oieu I ] E per . Variational Policy Search Inference RL ) ( model likelihood based ( ) and ' and max - Oi I Generative Policy Oi inference parameters : : RL ) model . based parameters would I model t Latent Vj variables States actions : Vj and : lie la Log Incremental weights incremental : rewards :
Stochastic Computation Graphs Credit Assignment in IT Sum " " blocked paths IL ] over d ( E = Veith PLY Sum " " unblocked paths , over .mn/j..o?sy.dd-oibg9hil/Lta..?aeuadoieu ; I ] ⇐ per . = & eh Assignment The loss L Credit depends : { Vj } all If Vsnpcv variables ) sample on we . s " that eihely " Then Vj samples good some are L ? Ls " " bad increase samples whereas decrease , As " " bad get result credit samples may a .
Stochastic Computation Graphs Credit Assignment Th IT Sum " " blocked paths IL ] over d ( E = Veith PLY Sum " " unblocked paths , over .mn/j..o?sy.dd-oibg9hil/Lta..?aeuadoieu , I ] ⇐ per . I Lj Replace L Variable Idea loss cific with Spe : a - that that the expected gradient such remains but has lower the same variance a ,
Stochastic Computation Graphs Credit Assignment in IL ) IE Lj = PCY.k.vn do ; m ] E per . .mn/j..o?y.ddoibg9hilllQiB;lta..?aeuadoieu ÷ " T.i.a.e.su . . . - Qj ) ] - Blachwellizatian ) E I adobe.gg/vj ) ( L ( Rao which o = Add / Ftfddobogglvj baseline Bj ) 2 Bj subtract ) any with o = . ( ) known craniate also control a as
Recommend
More recommend
