MPE inference in Conditional Linear Gaussian Networks Antonio Salmern - PowerPoint PPT Presentation

MPE inference in Conditional Linear Gaussian Networks Antonio Salmerón 1 Rafael Rumí 1 Helge Langseth 2 Anders L. Madsen 3 , 4 Thomas D. Nielsen 4 1 Dept. Mathematics, University of Almería, Spain 2 Dept. Computer and Information Science. Norwegian University of Science and Technology, Trondheim, Norway 3 Hugin Expert A/S, Aalborg, Denmark 4 Dept. Computer Science, Aalborg University, Denmark ECSQARU 2015, Compiegne, July 17, 2015 1

Introduction ◮ The AMiDST project: Analysis of MassIve Data STreams http://www.amidst.eu ECSQARU 2015, Compiegne, July 17, 2015 2

Introduction ◮ The AMiDST project: Analysis of MassIve Data STreams http://www.amidst.eu ◮ Large number of variables ◮ Queries to be answered in real time ◮ Hybrid Bayesian networks (involving discrete and continuous variables) ◮ Conditional linear Gaussian networks ECSQARU 2015, Compiegne, July 17, 2015 3

Conditional Linear Gaussian networks A Conditional Linear Gaussian (CLG) network is a hybrid Bayesian network where ◮ The conditional distribution of each discrete variable X D given its parents is a multinomial ◮ The conditional distribution of each continuous variable Z with discrete parents X D and continuous parents X C , is p ( z | X D = x D , X C = x C ) = N ( z ; α ( x D ) + β ( x D ) T x C , σ ( x D )) for all x D and x C , where α and β are the coefficients of a linear regression model of Z given X C , potentially different for each configuration of X D . ECSQARU 2015, Compiegne, July 17, 2015 4

Conditional Linear Gaussian networks. Example P ( Y ) = ( 0 . 5 , 0 . 5 ) P ( S ) = ( 0 . 1 , 0 . 9 ) Y f ( w | Y = 0 ) = N ( w ; − 1 , 1 ) f ( w | Y = 1 ) = N ( w ; 2 , 1 ) f ( t | w , S = 0 ) = N ( t ; − w , 1 ) W S f ( t | w , S = 1 ) = N ( t ; w , 1 ) f ( u | w ) = N ( u ; w , 1 ) U T ECSQARU 2015, Compiegne, July 17, 2015 5

Querying a Bayesian network (I) ◮ Probabilistic inference: Computing the posterior distribution of a target variable: � � p ( x , x E ) d x C x C x D p ( x i | x E ) = � � p ( x , x E ) d x C i x Ci x Di ECSQARU 2015, Compiegne, July 17, 2015 6

Querying a Bayesian network (II) ◮ Maximum a posteriori (MAP): For a set of target variables X I , the goal is to compute x ∗ I = arg max p ( x I | X E = x E ) x I where p ( x I | X E = x E ) is obtained by first marginalizing out from p ( x ) the variables not in X I and not in X E ECSQARU 2015, Compiegne, July 17, 2015 7

Querying a Bayesian network (II) ◮ Maximum a posteriori (MAP): For a set of target variables X I , the goal is to compute x ∗ I = arg max p ( x I | X E = x E ) x I where p ( x I | X E = x E ) is obtained by first marginalizing out from p ( x ) the variables not in X I and not in X E ◮ Most probable explanation (MPE): A particular case of MAP where X I includes all the unobserved variables ECSQARU 2015, Compiegne, July 17, 2015 8

MPE in CLG networks ◮ Can be carried out using bucket elimination (Dechter, 1999): ◮ A bucket containing probability functions is kept for each variable. ◮ Initially, an ordering of the variables in the network is established, and each conditional distribution in the network is assigned to the bucket corresponding to the variable in its domain holding the highest rank. ◮ Afterwards, the buckets are processed in a sequence opposite to the initial ordering of the variables. ◮ Each bucket is processed by combining all the functions it contains and by marginalizing the main variable in that bucket by maximization. ECSQARU 2015, Compiegne, July 17, 2015 9

Example P ( Y ) = ( 0 . 5 , 0 . 5 ) Y P ( S ) = ( 0 . 1 , 0 . 9 ) f ( w | Y = 0 ) = N ( w ; − 1 , 1 ) f ( w | Y = 1 ) = N ( w ; 2 , 1 ) W S f ( t | w , S = 0 ) = N ( t ; − w , 1 ) f ( t | w , S = 1 ) = N ( t ; w , 1 ) f ( u | w ) = N ( u ; w , 1 ) U T Elimination order: Y , S , W , T , U ECSQARU 2015, Compiegne, July 17, 2015 10

Example Elimination order: Y , S , W , T , U P ( Y ) h S ( Y ) B Y : P ( Y ) = ( 0 . 5 , 0 . 5 ) P ( S ) = ( 0 . 1 , 0 . 9 ) h W ( Y , S ) B S : P ( S ) f ( w | Y = 0 ) = N ( w ; − 1 , 1 ) f ( w | Y = 1 ) = N ( w ; 2 , 1 ) f ( w | Y ) h T ( w , S ) h U ( w ) B W : f ( t | w , S = 0 ) = N ( t ; − w , 1 ) B T : f ( t | w , S ) f ( t | w , S = 1 ) = N ( t ; w , 1 ) f ( u | w ) = N ( u ; w , 1 ) f ( u | w ) B U : ECSQARU 2015, Compiegne, July 17, 2015 11

Example Elimination order: Y , S , W , T , U P ( Y ) h S ( Y ) B Y : P ( Y ) = ( 0 . 5 , 0 . 5 ) P ( S ) = ( 0 . 1 , 0 . 9 ) h W ( Y , S ) B S : P ( S ) f ( w | Y = 0 ) = N ( w ; − 1 , 1 ) f ( w | Y = 1 ) = N ( w ; 2 , 1 ) 1 f ( w | Y ) h T ( w , S ) √ B W : 2 π f ( t | w , S = 0 ) = N ( t ; − w , 1 ) B T : f ( t | w , S ) f ( t | w , S = 1 ) = N ( t ; w , 1 ) f ( u | w ) = N ( u ; w , 1 ) f ( u | w ) B U : ECSQARU 2015, Compiegne, July 17, 2015 14

Example Elimination order: Y , S , W , T , U P ( Y ) h S ( Y ) B Y : P ( Y ) = ( 0 . 5 , 0 . 5 ) P ( S ) = ( 0 . 1 , 0 . 9 ) h W ( Y , S ) B S : P ( S ) f ( w | Y = 0 ) = N ( w ; − 1 , 1 ) f ( w | Y = 1 ) = N ( w ; 2 , 1 ) 1 f ( w | Y ) h T ( w , S ) √ B W : 2 π f ( t | w , S = 0 ) = N ( t ; − w , 1 ) B T : f ( t | w , S ) f ( t | w , S = 1 ) = N ( t ; w , 1 ) f ( u | w ) = N ( u ; w , 1 ) f ( u | w ) B U : ECSQARU 2015, Compiegne, July 17, 2015 15

Example Elimination order: Y , S , W , T , U P ( Y ) h S ( Y ) B Y : P ( Y ) = ( 0 . 5 , 0 . 5 ) P ( S ) = ( 0 . 1 , 0 . 9 ) h W ( Y , S ) B S : P ( S ) f ( w | Y = 0 ) = N ( w ; − 1 , 1 ) f ( w | Y = 1 ) = N ( w ; 2 , 1 ) 1 1 f ( w | Y ) √ √ B W : 2 π 2 π f ( t | w , S = 0 ) = N ( t ; − w , 1 ) B T : f ( t | w , S ) f ( t | w , S = 1 ) = N ( t ; w , 1 ) f ( u | w ) = N ( u ; w , 1 ) f ( u | w ) B U : ECSQARU 2015, Compiegne, July 17, 2015 16

Example Elimination order: Y , S , W , T , U P ( Y ) h S ( Y ) B Y : P ( Y ) = ( 0 . 5 , 0 . 5 ) P ( S ) = ( 0 . 1 , 0 . 9 ) h W ( Y , S ) B S : P ( S ) f ( w | Y = 0 ) = N ( w ; − 1 , 1 ) f ( w | Y = 1 ) = N ( w ; 2 , 1 ) 1 1 f ( w | Y ) √ √ B W : 2 π 2 π f ( t | w , S = 0 ) = N ( t ; − w , 1 ) B T : f ( t | w , S ) f ( t | w , S = 1 ) = N ( t ; w , 1 ) f ( u | w ) = N ( u ; w , 1 ) f ( u | w ) B U : ECSQARU 2015, Compiegne, July 17, 2015 17

Example Elimination order: Y , S , W , T , U P ( Y ) h S ( Y ) B Y : P ( Y ) = ( 0 . 5 , 0 . 5 ) P ( S ) = ( 0 . 1 , 0 . 9 ) 1 2 π ) 3 ( B S : P ( S ) √ f ( w | Y = 0 ) = N ( w ; − 1 , 1 ) f ( w | Y = 1 ) = N ( w ; 2 , 1 ) 1 1 f ( w | Y ) √ √ B W : 2 π 2 π f ( t | w , S = 0 ) = N ( t ; − w , 1 ) B T : f ( t | w , S ) f ( t | w , S = 1 ) = N ( t ; w , 1 ) f ( u | w ) = N ( u ; w , 1 ) f ( u | w ) B U : ECSQARU 2015, Compiegne, July 17, 2015 18

Example Elimination order: Y , S , W , T , U P ( Y ) h S ( Y ) B Y : P ( Y ) = ( 0 . 5 , 0 . 5 ) P ( S ) = ( 0 . 1 , 0 . 9 ) 1 2 π ) 3 ( B S : P ( S ) √ f ( w | Y = 0 ) = N ( w ; − 1 , 1 ) f ( w | Y = 1 ) = N ( w ; 2 , 1 ) 1 1 f ( w | Y ) √ √ B W : 2 π 2 π f ( t | w , S = 0 ) = N ( t ; − w , 1 ) B T : f ( t | w , S ) f ( t | w , S = 1 ) = N ( t ; w , 1 ) f ( u | w ) = N ( u ; w , 1 ) f ( u | w ) B U : ECSQARU 2015, Compiegne, July 17, 2015 19

Example Elimination order: Y , S , W , T , U 1 2 π ) 3 0 . 9 ( B Y : P ( Y ) √ P ( Y ) = ( 0 . 5 , 0 . 5 ) P ( S ) = ( 0 . 1 , 0 . 9 ) 1 2 π ) 3 ( √ B S : P ( S ) f ( w | Y = 0 ) = N ( w ; − 1 , 1 ) 1 1 f ( w | Y = 1 ) = N ( w ; 2 , 1 ) B W : f ( w | Y ) √ √ 2 π 2 π f ( t | w , S = 0 ) = N ( t ; − w , 1 ) f ( t | w , S ) B T : f ( t | w , S = 1 ) = N ( t ; w , 1 ) f ( u | w ) = N ( u ; w , 1 ) B U : f ( u | w ) ECSQARU 2015, Compiegne, July 17, 2015 20

Example Elimination order: Y , S , W , T , U 1 2 π ) 3 0 . 9 ( B Y : P ( Y ) √ P ( Y ) = ( 0 . 5 , 0 . 5 ) P ( S ) = ( 0 . 1 , 0 . 9 ) 1 2 π ) 3 ( √ B S : P ( S ) f ( w | Y = 0 ) = N ( w ; − 1 , 1 ) 1 1 f ( w | Y = 1 ) = N ( w ; 2 , 1 ) B W : f ( w | Y ) √ √ 2 π 2 π f ( t | w , S = 0 ) = N ( t ; − w , 1 ) f ( t | w , S ) B T : f ( t | w , S = 1 ) = N ( t ; w , 1 ) f ( u | w ) = N ( u ; w , 1 ) B U : f ( u | w ) The MPE configuration is obtained tracing back the steps ECSQARU 2015, Compiegne, July 17, 2015 21

What we have learnt so far ◮ Marginalizing continuous variables is easy if they are the first to be marginalized out ◮ The price to pay is that, in the worst case, a function containing all the discrete variables would be created ◮ This complexity blow-up can be avoided in many cases by allowing orderings for constructing the buckets where discrete and continuous variables can be arranged with no restrictions ◮ But then a new problem arises, as the maximization operation becomes more complex ECSQARU 2015, Compiegne, July 17, 2015 22

MPE inference in Conditional Linear Gaussian Networks Antonio Salmern - PowerPoint PPT Presentation

MPE inference in Conditional Linear Gaussian Networks Antonio Salmern 1 Rafael Rum 1 Helge Langseth 2 Anders L. Madsen 3 , 4 Thomas D. Nielsen 4 1 Dept. Mathematics, University of Almera, Spain 2 Dept. Computer and Information Science.

8 Approximate inference in switching linear dynamical systems using Gaussian mixtures David

& Exact inference for Gaussian networks Probabilistic Graphical Models Sharif University of

Factor analysis & Exact inference for Gaussian networks Probabilistic Graphical Models

Scalable Training of Inference Networks for Gaussian-Process Models Jiaxin Shi Tsinghua

Characterization of Conditional Independence and Weak Realizations of Multivariate Gaussian

Overview Independence Belief Networks Conditional Independence Belief networks Chris

Conditional Heteroscedasticity (CH) So far, our models are for the conditional mean . For

On the Conditional Mutual I nformation in Gaussian- Markov Structured Grids Hanie Sedghi &

Case Study: Approximate Bayesian Inference for Latent Gaussian Models by Using Integrated Nested

1 Gaussian Fun Facts Well add to these as we go along! First, consider a Gaussian random

Modern Gaussian Processes: Scalable Inference and Novel Applications (Part II-b) Approximate

Inference in Bayesian Networks CE417: Introduction to Artificial Intelligence Sharif University

Inference in Bayesian Networks CE417: Introduction to Artificial Intelligence Sharif University

Approximate Bayesian inference for latent Gaussian models avard Rue 1 H Department of

Structured Inference Networks for Nonlinear State Space Models Rahul G. Krishnan, Uri Shalit,

2 Forward Elimination Factored Portion Factored Portion Si

Polynomial Chaos Acceleration for the Bayesian Inference of Random Fields with Gaussian Priors and

Efficient sampling for Gaussian linear regression with arbitrary priors P. Richard Hahn (Arizona

Inference in Bayesian networks Chapter 14.45 Chapter 14.45 1 Outline Exact inference

M odels for Inexact Reasoning Fuzzy Logic Lesson 6 Inference from Conditional Fuzzy

Inference in Bayesian networks Chapter 14.45 Chapter 14.45 1 Outline Exact inference

Tutorial 2 the outline Example-1 from linear algebra Conditional probability Example 2:

Modern Gaussian Processes: Scalable Inference and Novel Applications (Part III) Applications,

Extended Variational Inference for Non-Gaussian Statistical Models Zhanyu Ma

MPE inference in Conditional Linear Gaussian Networks Antonio Salmern - PowerPoint PPT Presentation

MPE inference in Conditional Linear Gaussian Networks Antonio Salmern 1 Rafael Rum 1 Helge Langseth 2 Anders L. Madsen 3 , 4 Thomas D. Nielsen 4 1 Dept. Mathematics, University of Almera, Spain 2 Dept. Computer and Information Science.

8 Approximate inference in switching linear dynamical systems using Gaussian mixtures David

&amp; Exact inference for Gaussian networks Probabilistic Graphical Models Sharif University of

Factor analysis &amp; Exact inference for Gaussian networks Probabilistic Graphical Models

Scalable Training of Inference Networks for Gaussian-Process Models Jiaxin Shi Tsinghua

Characterization of Conditional Independence and Weak Realizations of Multivariate Gaussian

Overview Independence Belief Networks Conditional Independence Belief networks Chris

Conditional Heteroscedasticity (CH) So far, our models are for the conditional mean . For

On the Conditional Mutual I nformation in Gaussian- Markov Structured Grids Hanie Sedghi &amp;

Case Study: Approximate Bayesian Inference for Latent Gaussian Models by Using Integrated Nested

1 Gaussian Fun Facts Well add to these as we go along! First, consider a Gaussian random

Modern Gaussian Processes: Scalable Inference and Novel Applications (Part II-b) Approximate

Inference in Bayesian Networks CE417: Introduction to Artificial Intelligence Sharif University

Inference in Bayesian Networks CE417: Introduction to Artificial Intelligence Sharif University

Approximate Bayesian inference for latent Gaussian models avard Rue 1 H Department of

Structured Inference Networks for Nonlinear State Space Models Rahul G. Krishnan, Uri Shalit,

2 Forward Elimination Factored Portion Factored Portion Si

Polynomial Chaos Acceleration for the Bayesian Inference of Random Fields with Gaussian Priors and

Efficient sampling for Gaussian linear regression with arbitrary priors P. Richard Hahn (Arizona

Inference in Bayesian networks Chapter 14.45 Chapter 14.45 1 Outline Exact inference

M odels for Inexact Reasoning Fuzzy Logic Lesson 6 Inference from Conditional Fuzzy

Inference in Bayesian networks Chapter 14.45 Chapter 14.45 1 Outline Exact inference

Tutorial 2 the outline Example-1 from linear algebra Conditional probability Example 2:

Modern Gaussian Processes: Scalable Inference and Novel Applications (Part III) Applications,

Extended Variational Inference for Non-Gaussian Statistical Models Zhanyu Ma

& Exact inference for Gaussian networks Probabilistic Graphical Models Sharif University of

Factor analysis & Exact inference for Gaussian networks Probabilistic Graphical Models

On the Conditional Mutual I nformation in Gaussian- Markov Structured Grids Hanie Sedghi &