Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem P. Sunehag (with T. Sears) ANU/NICTA July 2007 P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 1 / 14
Graphical Models Section Graphical Models 1 Generalizing the exponential family 2 Generalized Factorization 3 Conclusions 4 P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 2 / 14
Graphical Models Graphical Models Graph G = ( V , E ), vertices (or nodes), V , and edges E . P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 3 / 14
Graphical Models Graphical Models Graph G = ( V , E ), vertices (or nodes), V , and edges E . A clique, c ⊂ V is a fully connected subgraph of G . P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 3 / 14
Graphical Models Graphical Models Graph G = ( V , E ), vertices (or nodes), V , and edges E . A clique, c ⊂ V is a fully connected subgraph of G . Vertices, { x i } M i =1 correspond to random variables. P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 3 / 14
Graphical Models Graphical Models Graph G = ( V , E ), vertices (or nodes), V , and edges E . A clique, c ⊂ V is a fully connected subgraph of G . Vertices, { x i } M i =1 correspond to random variables. Missing edges represent conditional independence assumptions. P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 3 / 14
Graphical Models Factorization Theorem (Hammersley-Clifford) The density of a joint probability distributions satisifies the conditional indepence assumptions represented by a graph G if and only if there are local clique densities ψ c such that f ( x ) = 1 � ψ c ( x c ) Z c where x is the full random vector, x c the restriction of x to c, c runs over all cliques and Z is a normalization constant. It is common to use ψ c of an exponential family form ψ c ( x c ) = exp( η c · T ( x c ) − Z c ( η c )). With such ψ c the joint density is also an exponential family distribution. P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 4 / 14
Graphical Models Limitations of the Exponential Family Exponential family distributions have thin tails and full support. P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 5 / 14
Graphical Models Limitations of the Exponential Family Exponential family distributions have thin tails and full support. Phenomena with heavy tailed power-law distributions are abundant. P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 5 / 14
Graphical Models Limitations of the Exponential Family Exponential family distributions have thin tails and full support. Phenomena with heavy tailed power-law distributions are abundant. We are also interested in situations where an unlikely event for one variable increases the probability for otherwise unlikely events for other variables that under more normal circumstaces behave as if they were close to independent. P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 5 / 14
Graphical Models Limitations of the Exponential Family Exponential family distributions have thin tails and full support. Phenomena with heavy tailed power-law distributions are abundant. We are also interested in situations where an unlikely event for one variable increases the probability for otherwise unlikely events for other variables that under more normal circumstaces behave as if they were close to independent. Example: A portfolio of equities is considered to be more correlated in the face of steep market decline. P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 5 / 14
Graphical Models Limitations of the Exponential Family Exponential family distributions have thin tails and full support. Phenomena with heavy tailed power-law distributions are abundant. We are also interested in situations where an unlikely event for one variable increases the probability for otherwise unlikely events for other variables that under more normal circumstaces behave as if they were close to independent. Example: A portfolio of equities is considered to be more correlated in the face of steep market decline. The generalizations we consider here can help us model that. P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 5 / 14
Generalizing the exponential family Section Graphical Models 1 Generalizing the exponential family 2 Generalized Factorization 3 Conclusions 4 P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 6 / 14
Generalizing the exponential family Deformed logarithms based on φ -logarithms � p 1 log( p ) = x dx Usual construction 1 P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 7 / 14
Generalizing the exponential family Deformed logarithms based on φ -logarithms � p 1 log φ ( p ) = φ ( x ) dx Deformed Log 1 P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 7 / 14
Generalizing the exponential family Deformed logarithms based on φ -logarithms � p 1 log φ ( p ) = φ ( x ) dx Deformed Log 1 Positive increasing φ . P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 7 / 14
Generalizing the exponential family Deformed logarithms based on φ -logarithms � p 1 log φ ( p ) = φ ( x ) dx Deformed Log 1 Positive increasing φ . φ ( x ) = x q yields the q -logarithm from the non-extensive thermodynamics literature. log q ( x ) := x 1 − q − 1 1 − q P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 7 / 14
Generalizing the exponential family Deformed logarithms based on φ -logarithms � p 1 log φ ( p ) = φ ( x ) dx Deformed Log 1 Positive increasing φ . φ ( x ) = x q yields the q -logarithm from the non-extensive thermodynamics literature. log q ( x ) := x 1 − q − 1 1 − q The inverse of the q -logarithm is 1 1 − q exp q ( v ) = (1 + (1 − q ) v ) . + P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 7 / 14
Generalizing the exponential family The Generalized Exponential Map q-Exponential Examples exp Φ � exp q 10 q > 1 naturally gives fat q � 1 8 tails. q � 1.5 q < 1 truncates the tail. q � 0.5 6 exp Φ � exp q 1 0.8 4 0.6 q � 0 0.4 2 0.2 � Asymptote � � 3 � 2.5 � 2 � 1.5 � 1 � 0.5 q � 1.5 � 3 � 2 � 1 1 2 3 P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 8 / 14
Generalized Factorization Section Graphical Models 1 Generalizing the exponential family 2 Generalized Factorization 3 Conclusions 4 P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 9 / 14
Generalized Factorization q-multiplication Problem: If q � = 1, then � exp q ( t i ) is not equal to exp q ( � t i ). P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 10 / 14
Generalized Factorization q-multiplication Problem: If q � = 1, then � exp q ( t i ) is not equal to exp q ( � t i ). Solution: Define a new operation, ⊗ q , such that exp q ( t 1 ) ⊗ q exp q ( t 2 ) = exp q ( t 1 + t 2 ) . P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 10 / 14
Generalized Factorization q-multiplication Problem: If q � = 1, then � exp q ( t i ) is not equal to exp q ( � t i ). Solution: Define a new operation, ⊗ q , such that exp q ( t 1 ) ⊗ q exp q ( t 2 ) = exp q ( t 1 + t 2 ) . ⊗ q is defined by 1 x ⊗ q y = exp q (log q ( x ) + log q ( y )) = ( x 1 − q + y 1 − q − 1) 1 − q + P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 10 / 14
Generalized Factorization q-multiplication Problem: If q � = 1, then � exp q ( t i ) is not equal to exp q ( � t i ). Solution: Define a new operation, ⊗ q , such that exp q ( t 1 ) ⊗ q exp q ( t 2 ) = exp q ( t 1 + t 2 ) . ⊗ q is defined by 1 x ⊗ q y = exp q (log q ( x ) + log q ( y )) = ( x 1 − q + y 1 − q − 1) 1 − q + It is associative, commutative and has 1 as its neutral element. P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem MaxEnt 10 / 14
Recommend
More recommend
