Advanced Machine Learning CS 7140 - Spring 2018 Lecture 13: Project - PowerPoint PPT Presentation

Advanced Machine Learning CS 7140 - Spring 2018 Lecture 13: Project Discussion Jan-Willem van de Meent

Schedule • Next 3 lectures: SVI, BBVI, Variational Autoencoders 
 (BP, Junction Tree, EP later) • Homework 4: Fri 30 Mar • Homework 5: Fri Ap 13 • Final Exam: Wed 18 Apr • Project Presentations: Wed 25 Apr • Project: Due Fri 27 Apr

Project • Goal: Implement and test one 
 state-of-the-art method • Group size 2-3 members • Amount of work should be equivalent 
 to ~2 homework assignments • 20% of Grade

Grading Homework: 30% • Scribing: 20% • Exams: 30% • Project: 20% •

Example: LDA Download one or more datasets • 20 Newsgroups, NY Times, Wikipedia Implement and compare algorithms • Gibbs Sampling, • Stochastic Variational Inference 
 [Hoffman JMLR 2013] Test your Implementations • Geweke Style Tests • Reference Implementations Evaluate results • Visualize Topics • Perplexity & Coherence Measures

Example: HMC Basic: Implement HMC • Use: https://github.com/HIPS/autograd • Implement Geweke Style Tests • Calculate Effective Sample Size Basic: Test your Implementations • Geweke Style Tests • Calculate Effective Sample Size Advanced: Implement the No-Uturn Sampler • See: [Hoffman et al. JMLR 2014]

Example: Automatic Statistician Goal: Search over Kernels for GP Regression [Duvenaud et al. ICML 2013]

Example: Automatic Statistician 0 0 quadratic locally Lin × Lin SE × Per functions periodic 0 0 periodic periodic Lin + Per SE + Per with trend with noise 0 0 increasing growing Lin × SE Lin × Per variation amplitude Goal: Search over Kernels for GP Regression [Duvenaud et al. ICML 2013]

Example: Automatic Statistician Basic: Automatic Statistician “Light” • Do basis function regression 
 (with lots of basis functions) Basic: Test on Standard Datasets • Airline, CO2, etc… Advanced: Full Implementation • Use GPFlow, perform kernel search Advanced: Real-world Datasets • Analyze Uber Data Very Advanced: Model Ensembles • Use MCMC to sample distribution 
 over possible solutions

Example: Time Series Analysis Goal: Model commonalities between many short time series. [van de Meent et al. ICML 2013]

Example: Time Series Analysis Basic: Variational Inference • Implement VBEM for Hidden Markov Models Basic: Test on Synthetic Datasets • Can provide these Advanced: Stochastic Gradient Version • Implement Stochastic Variational Inference Advanced: Full Implementation • Maximize prior hyperparameters

Example: Variational Autoencoders Input Hidden Mean 
 Encoding Hidden Reconstructed 
 Images Units Std Dev (random) Units Images 784 256 2-50 256 784 (28 x 28) (28 x 28)

Example: Variational Autoencoders 2-dimensional 50-dimensional (TSNE)

Example: Variational Autoencoders Basic: Semi-supervised Learning • Reproduce [Kingma et al NIPS 2014] Advanced: Autoencoding LDA • Reproduce [Kingma et al NIPS 2014] • Reproduce [Miao et al ICML 2016] 
 or [Srivastava ICLR 2017] Super Advanced: Grammar VAEs • Reproduce [Kushner et al. ICML 2017] Super Advanced: Structured VAEs • Reproduce [Johnson et al. NIPS 2016] Super Advanced: Disentangled Representations • (talk to Babak)

References Stochastic Variational Inference for LDA • Hoffman, M. D., Blei, D. M., Wang, C. & Paisley, J. Stochastic variational inference. Journal of Machine Learning Research 14, 1303–1347 (2013). No-Uturn Sampler • Hoffman, M. D. & Gelman, A. The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo. JMLR (2014). Automatic Statistican • Duvenaud, D., Lloyd, J. R., Grosse, R., Tenenbaum, J. B. & Ghahramani, Z. Structure discovery in nonparametric regression through compositional kernel search. ICML (2013). Time Series Analysis • van de Meent, J.-W., Bronson, J. E., Wood, F., Gonzalez, R. L. & Wiggins, C. H. Hierarchically-coupled hidden Markov models for learning kinetic rates from single- molecule data. in International Conference on Machine Learning 28, 361–369 (2013).

References Semi-Supervised VAEs • Kingma, D. P., Rezende, D. J., Mohamed, S. & Welling, M. Semi-Supervised Learning with Deep Generative Models. NIPS (2014). Auto-encoding LDA • Miao, Y., Yu, L. & Blunsom, P. Neural variational inference for text processing. in ICML 1727–1736 (2016). • Srivastava, A. & Sutton, C. Autoencoding variational inference for topic models. ICLR (2017). Grammar VAEs • Kusner, M. J., Paige, B. & Hernández-Lobato, J. M. Grammar variational autoencoder. ICML (2017). Structured VAEs • Johnson, M., Duvenaud, D. K., Wiltschko, A., Adams, R. P. & Datta, S. R. Composing graphical models with neural networks for structured representations and fast inference. in Advances in neural information processing systems 2946–2954 (2016).