Analysis of MAP in CRP Normal-Normal model ukasz Rajkowski Faculty - PowerPoint PPT Presentation

Analysis of MAP in CRP Normal-Normal model Łukasz Rajkowski Faculty of Mathematics, Informatics and Mechanics University of Warsaw l.rajkowski@mimuw.edu.pl November 28, 2016 Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

Chinese Restaurant Process Chinese Restaurant Process with parameter α can be viewed as a distribution on the space of partitions of a finite set. Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

Chinese Restaurant Process Chinese Restaurant Process with parameter α can be viewed as a distribution on the space of partitions of a finite set. � { 1 , 2 , 4 , 6 } , { 3 } , { 5 , 7 } � ? What is the probability of Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

Chinese Restaurant Process Chinese Restaurant Process with parameter α can be viewed as a distribution on the space of partitions of a finite set. � { 1 , 2 , 4 , 6 } , { 3 } , { 5 , 7 } � ? What is the probability of 7 6 5 4 3 2 1 . . . 0 0 0 0 P (new table) ∝ α P (join table) ∝ # sitting there Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

Chinese Restaurant Process Chinese Restaurant Process with parameter α can be viewed as a distribution on the space of partitions of a finite set. � { 1 , 2 , 4 , 6 } , { 3 } , { 5 , 7 } � ? What is the probability of 7 6 5 4 3 2 . . . 1 0 0 0 P (new table) ∝ α P (join table) ∝ # sitting there P = α α Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

Chinese Restaurant Process Chinese Restaurant Process with parameter α can be viewed as a distribution on the space of partitions of a finite set. � { 1 , 2 , 4 , 6 } , { 3 } , { 5 , 7 } � ? What is the probability of 7 6 5 4 3 . . . 1 2 0 0 0 P (new table) ∝ α P (join table) ∝ # sitting there P = α 1 α · 1 + α Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

Chinese Restaurant Process Chinese Restaurant Process with parameter α can be viewed as a distribution on the space of partitions of a finite set. � { 1 , 2 , 4 , 6 } , { 3 } , { 5 , 7 } � ? What is the probability of 7 6 5 4 . . . 1 2 3 0 0 P (new table) ∝ α P (join table) ∝ # sitting there P = α 1 α α · 1 + α · 2 + α Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

Chinese Restaurant Process Chinese Restaurant Process with parameter α can be viewed as a distribution on the space of partitions of a finite set. � { 1 , 2 , 4 , 6 } , { 3 } , { 5 , 7 } � ? What is the probability of 7 6 5 . . . 1 2 4 3 0 0 P (new table) ∝ α P (join table) ∝ # sitting there P = α 1 α 2 α · 1 + α · 2 + α · 3 + α Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

Chinese Restaurant Process Chinese Restaurant Process with parameter α can be viewed as a distribution on the space of partitions of a finite set. � { 1 , 2 , 4 , 6 } , { 3 } , { 5 , 7 } � ? What is the probability of 7 6 . . . 1 2 4 3 5 0 P (new table) ∝ α P (join table) ∝ # sitting there P = α 1 α 2 α α · 1 + α · 2 + α · 3 + α · 4 + α Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

Chinese Restaurant Process Chinese Restaurant Process with parameter α can be viewed as a distribution on the space of partitions of a finite set. � { 1 , 2 , 4 , 6 } , { 3 } , { 5 , 7 } � ? What is the probability of 7 . . . 1 2 4 6 3 5 0 P (new table) ∝ α P (join table) ∝ # sitting there P = α 1 α 2 α 3 α · 1 + α · 2 + α · 3 + α · 4 + α · 5 + α Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

Chinese Restaurant Process Chinese Restaurant Process with parameter α can be viewed as a distribution on the space of partitions of a finite set. � { 1 , 2 , 4 , 6 } , { 3 } , { 5 , 7 } � ? What is the probability of . . . 1 2 4 6 3 5 7 0 P (new table) ∝ α P (join table) ∝ # sitting there P = α 1 α 2 α 3 1 α · 1 + α · 2 + α · 3 + α · 4 + α · 5 + α · 6 + α Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

Chinese Restaurant Process Chinese Restaurant Process with parameter α can be viewed as a distribution on the space of partitions of a finite set. � { 1 , 2 , 4 , 6 } , { 3 } , { 5 , 7 } � ? What is the probability of . . . 1 2 4 6 3 5 7 0 P (new table) ∝ α P (join table) ∝ # sitting there P = α 1 α 2 α 3 1 α · 1 + α · 2 + α · 3 + α · 4 + α · 5 + α · 6 + α Definition J ∼ CRP n ( α ) means that P ( J = J ) = α |J | � J ∈J ( | J | − 1 )! , α ( n ) where α ( n ) = α ( α + 1 ) . . . ( α + n − 1 ) . Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

CRP Normal-Normal model n observations of dimension d may be modelled as follows J ∼ CRP ( α ) n iid θ = ( θ J ) J ∈J | J ∼ N ( � µ, T ) iid x J = ( x j ) j ∈ J | J , θ ∼ N ( θ J , Σ) for J ∈ J Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

CRP Normal-Normal model n observations of dimension d may be modelled as follows J ∼ CRP ( α ) n iid θ = ( θ J ) J ∈J | J ∼ N ( � µ, T ) iid x J = ( x j ) j ∈ J | J , θ ∼ N ( θ J , Σ) for J ∈ J � � J = { 1 , 2 , 4 , 6 } , { 3 } , { 5 , 7 } Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

CRP Normal-Normal model n observations of dimension d may be modelled as follows J ∼ CRP ( α ) n iid θ = ( θ J ) J ∈J | J ∼ N ( � µ, T ) iid x J = ( x j ) j ∈ J | J , θ ∼ N ( θ J , Σ) for J ∈ J � � J = { 1 , 2 , 4 , 6 } , { 3 } , { 5 , 7 } Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

CRP Normal-Normal model n observations of dimension d may be modelled as follows J ∼ CRP ( α ) n iid θ = ( θ J ) J ∈J | J ∼ N ( � µ, T ) iid x J = ( x j ) j ∈ J | J , θ ∼ N ( θ J , Σ) for J ∈ J � � J = { 1 , 2 , 4 , 6 } , { 3 } , { 5 , 7 } Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

CRP Normal-Normal model n observations of dimension d may be modelled as follows J ∼ CRP ( α ) n iid θ = ( θ J ) J ∈J | J ∼ N ( � µ, T ) iid x J = ( x j ) j ∈ J | J , θ ∼ N ( θ J , Σ) for J ∈ J � � J = { 1 , 2 , 4 , 6 } , { 3 } , { 5 , 7 } Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

CRP Normal-Normal model n observations of dimension d may be modelled as follows J ∼ CRP ( α ) n iid θ = ( θ J ) J ∈J | J ∼ N ( � µ, T ) iid x J = ( x j ) j ∈ J | J , θ ∼ N ( θ J , Σ) for J ∈ J � � J = { 1 , 2 , 4 , 6 } , { 3 } , { 5 , 7 } Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

CRP Normal-Normal model n observations of dimension d may be modelled as follows J ∼ CRP ( α ) n iid θ = ( θ J ) J ∈J | J ∼ N ( � µ, T ) iid x J = ( x j ) j ∈ J | J , θ ∼ N ( θ J , Σ) for J ∈ J � � J = { 1 , 2 , 4 , 6 } , { 3 } , { 5 , 7 } Advantage: No need to define a priori limit on number of clusters Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

CRP Normal-Normal model n observations of dimension d may be modelled as follows J ∼ CRP ( α ) n iid θ = ( θ J ) J ∈J | J ∼ N ( � µ, T ) iid x J = ( x j ) j ∈ J | J , θ ∼ N ( θ J , Σ) for J ∈ J � � J = { 1 , 2 , 4 , 6 } , { 3 } , { 5 , 7 } The ’true’ partition is not known Advantage: No need to define a priori limit on number of clusters Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

CRP Normal-Normal model n observations of dimension d may be modelled as follows J ∼ CRP ( α ) n iid θ = ( θ J ) J ∈J | J ∼ N ( � µ, T ) iid x J = ( x j ) j ∈ J | J , θ ∼ N ( θ J , Σ) for J ∈ J � � J = { 1 , 2 , 4 , 6 } , { 3 } , { 5 , 7 } The ’true’ partition is not known Advantage: Task: No need to Compute the define a priori distribution of limit on number J provided of clusters observation x Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

CRP Normal-Normal model The Posterior For µ = � 0, T = τ 2 I , Σ = σ 2 I � 1 � � 2 � � α � |J | � | J | · | J | ! � x J � � P ( J | x ) ∝ · exp � 2 σ 2 1 + τ 2 / | J | τ | J | σ 2 + 1 | J | J ∈J J ∈J σ 2 τ 2 � �� � =: Q x ( J ) Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

CRP Normal-Normal model The Posterior For µ = � 0, T = τ 2 I , Σ = σ 2 I � 1 � � � 2 � α � |J | � | J | · | J | ! � x J � � P ( J | x ) ∝ · exp � 2 σ 2 1 + τ 2 / | J | τ | J | σ 2 + 1 | J | J ∈J J ∈J σ 2 τ 2 � �� � =: Q x ( J ) The MAP The Maximal Posterior Partition (MAP) is defined by ˆ J ( x ) = argmax J P ( J | x ) Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

MAP properties (in Normal model) Property 1 ˆ J MAP ( x ) is a convex partition with respect to x . Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

MAP properties (in Normal model) Property 1 ˆ J MAP ( x ) is a convex partition with respect to x . Convex and lovely Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

MAP properties (in Normal model) Property 1 ˆ J MAP ( x ) is a convex partition with respect to x . Convex but not lovely Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model

MAP properties (in Normal model) Property 1 ˆ J MAP ( x ) is a convex partition with respect to x . Not convex and disastrous Łukasz Rajkowski Analysis of MAP in CRP Normal-Normal model