Analysis-by-Synthesis a.k.a Generative Modeling Tejas D Kulkarni - PowerPoint PPT Presentation

Analysis-by-Synthesis a.k.a Generative Modeling Tejas D Kulkarni (tejask@mit.edu) Monday, October 19, 15

Traditional paradigm for AI research • Traditional machine learning and pattern recognition has been remarkable successful at questions like: “ what is where ” Krizhevsky et. al • Decades of progress driven by a single experimental paradigm! • Train/Test/Validation set split • Learn parameters using train set and evaluate on the test set Monday, October 19, 15

But infant learning looks like this ... source: https://www.youtube.com/watch?v=3f3rOz0NzPc Monday, October 19, 15

What is the ‘right’ way to think about AI? • Agent learns an internal model of the world (generative model) given sensory states • Agent uses learnt model to mentally hypothesize plans and pick actions that maximizes expected future rewards • While exploring, agents pick actions that minimizes prediction error of it’s internal model (i.e. minimizes entropy) Monday, October 19, 15

Analysis-by-Synthesis in Perception Geoff Hermann Von Helmholtz Karl Friston Hinton et al The general rule determining the ideas Boltzmann Machines of vision that are formed whenever an The free-energy principle Helmholtz Machine impression is made on the eye, is that says that any self-organizing such objects are always imagined as system that is at equilibrium being present in the field of vision as with its environment must would have to be there in order to minimize its free energy produce the same impression on the nervous mechanism (1865) (1885, 1995) (2010) Monday, October 19, 15

Analysis-by-Synthesis in Perception Kersten, NIPS 1998 Tutorial on Computational Vision Monday, October 19, 15

Analysis-by-Synthesis in Perception S(t) S(t+1) S(t+T) I(t) I(t+1) . . . I(t+T) Goal: P ( S | I ) ∝ P ( I | S ) P ( S ) Monday, October 19, 15

Are probabilities necessary? • Our internal models of reality are often incomplete. Therefore we need a mathematical language to handle uncertainty • Probability theory is a framework to extend logic to include reasoning on uncertain information • Probability need not have anything to do with randomness. Probabilities do not describe reality -- only our information about reality - E.T. Jaynes • Bayesian statistics describes epistemological ( study of the nature and scope of knowledge ) uncertainty using the mathematical language of probability • Start with prior beliefs and update these using data to give posterior beliefs Monday, October 19, 15

Are probabilities necessary? Assumptions violated: Assumptions satisfied: broad posterior narrower posterior superimposed sampled (d) Monday, October 19, 15

Probabilistic 3D Face Analysis face-id Eyes Eyes Nose Nose Light Mouth Mouth Outline Outline Shape Texture Shading Affine Inference Problem: Simulator P ( S, T, L, A | I ) ∝ P ( I | S, T, L, A ) P ( L ) P ( S ) P ( T ) P ( A ) Y ∝ N ( I − O ; 0 , 0 . 1) P ( L ) P ( A ) P ( S i ) P ( T i ) Image i Monday, October 19, 15

Probabilistic 3D Face Analysis face-id Eyes Eyes Nose Nose Light Mouth Mouth Outline Outline Shape Texture Shading Affine Random Draw Simulator Image Monday, October 19, 15

Probabilistic 3D Face Analysis face-id Eyes Eyes Nose Nose Light Mouth Mouth Outline Outline Shape Texture Shading Affine Random Draw Simulator Image Monday, October 19, 15

Probabilistic 3D Face Analysis face-id Eyes Eyes Nose Nose Light Mouth Mouth Outline Outline Shape Texture Shading Affine Random Draw Simulator Image Monday, October 19, 15

Probabilistic 3D Face Analysis face-id Eyes Eyes Nose Nose Light Mouth Mouth Outline Outline Shape Texture Shading Affine Random Draw Simulator Image Monday, October 19, 15

Probabilistic 3D Face Analysis Monday, October 19, 15

Probabilistic 3D Face Analysis Inferred model Inferred model Observed Inferred re-rendered with re-rendered with Image (reconstruction) novel poses novel lighting Monday, October 19, 15

3D Human Pose Test Image Inference Trajectory Monday, October 19, 15

3D Shape Program Test Image Inference Trajectory Monday, October 19, 15

Inference • Many inference strategies for generative models: MCMC, Variational, Message Passing, Particle filtering etc. • Today we will discuss an algorithm that is simple and general (not necessarily efficient) Monday, October 19, 15

Inference • Simplest MCMC Algorithm: Metropolis Hastings • For simplicity, let us fix light and affine variables. S Nose ∼ randn (50) T Nose ∼ randn (50) . Nose Nose . Eyes Eyes . Light Outline Outline Mouth Mouth S Mouth ∼ randn (50) Shape Texture T Mouth ∼ randn (50) Shading P ( I | S, T ) ∝ Normal( O − R ; 0 , σ 0 ) Affine Simulator Image Monday, October 19, 15

MCMC • Simplest MCMC Algorithm: Metropolis Hastings • For simplicity, let us fix light and affine variables. S Nose ∼ randn (50) Repeat until convergence: T Nose ∼ randn (50) . . . (1) Let x be either S i or T i S Mouth ∼ randn (50) We sample new x 0 ∼ randn (50) T Mouth ∼ randn (50) (2) r = p ( x 0 ) q ( x | x 0 ) p ( x ) q ( x 0 | x ) P ( I | S, T ) ∝ Normal( O − R ; 0 , σ 0 ) (3) Accept x’ with probability: α = min { 1 , r } Otherwise, x’=x Monday, October 19, 15

How do we make inference faster? Monday, October 19, 15

Combine Probabilistic Inference with Neural Nets Monday, October 19, 15

Combine Probabilistic Inference with Neural Nets • Inference often gets stuck in local minima and the only way out is if large set of variables are changed at once Monday, October 19, 15

Combine Probabilistic Inference with Neural Nets • Inference often gets stuck in local minima and the only way out is if large set of variables are changed at once • Helmholtz machine: Wake-Sleep Alg (Dayan, 1995) Monday, October 19, 15

Combine Probabilistic Inference with Neural Nets • Inference often gets stuck in local minima and the only way out is if large set of variables are changed at once • Helmholtz machine: Wake-Sleep Alg (Dayan, 1995) • Informed Sampler (Jampani, 2015) Monday, October 19, 15

Combine Probabilistic Inference with Neural Nets • Inference often gets stuck in local minima and the only way out is if large set of variables are changed at once • Helmholtz machine: Wake-Sleep Alg (Dayan, 1995) • Informed Sampler (Jampani, 2015) • Use an external long-term memory to cache “hallucinations” synthesized from your generative models ( sleep ) and use them during perception ( wake) Monday, October 19, 15

Combine Probabilistic Inference with Neural Nets Monday, October 19, 15

Combine Probabilistic Inference with Neural Nets Generative Model Monday, October 19, 15

Combine Probabilistic Inference with Neural Nets Generative Model Unconditional Runs Monday, October 19, 15

Combine Probabilistic Inference with Neural Nets Hallucinated Hallucinated Hallucinated Hallucinated Hallucinated Data Generative Hallucinated Data Data Data Data { ρ i , I i Model R } Data Unconditional Hallucinated Data Runs (Sleep) Monday, October 19, 15

Combine Probabilistic Inference with Neural Nets Hallucinated Hallucinated Hallucinated Hallucinated Hallucinated Data Generative Hallucinated Data Data Data Data { ρ i , I i Model R } Data Unconditional Hallucinated Data Runs (Sleep) ν ( . ) (Krizhevsky et al.) Monday, October 19, 15

Combine Probabilistic Inference with Neural Nets Hallucinated Hallucinated Hallucinated Hallucinated Hallucinated Data Generative Hallucinated Data Data Data Data { ρ i , I i Model R } Data Unconditional Hallucinated Data Runs (Sleep) Learning ν ( . ) (Krizhevsky et al.) Monday, October 19, 15

Combine Probabilistic Inference with Neural Nets Hallucinated Hallucinated Hallucinated Hallucinated Hallucinated Data Generative Hallucinated Data Data Data Data { ρ i , I i Model R } Data Unconditional Hallucinated Data Runs (Sleep) ν ( . ) Learning ν ( . ) (Krizhevsky et al.) Monday, October 19, 15

Combine Probabilistic Inference with Neural Nets Hallucinated Hallucinated Hallucinated Hallucinated Hallucinated Data Generative Hallucinated Data Data Data Data { ρ i , I i Model R } Data Unconditional Hallucinated Data Runs (Sleep) ν ( . ) Learning Long-term Memory ν ( . ) (Krizhevsky et al.) Monday, October 19, 15

Combine Probabilistic Inference with Neural Nets Conditional Density Estimator Probabilistic Program q ( S ρ ← S 0 ρ | I D ) CNN ν ( I D ) Test Data I D Now run inference Long-term Memory 90%: Data-driven (Pattern Matching) (Sleep) 10%: Sampling/Search (Reasoning) Monday, October 19, 15

3D Face Analysis With Data-driven Proposals Without Data-driven Proposals Monday, October 19, 15

Learning parametrized generative models Tijmen Tielemen (Thesis, 2014) Monday, October 19, 15

Learning parametrized generative models Tijmen Tielemen (Thesis, 2014) Monday, October 19, 15