Compound Random Measures Jim Griffin (joint work with Fabrizio - PowerPoint PPT Presentation

Compound Random Measures Jim Griffin (joint work with Fabrizio Leisen) University of Kent

Introduction: Two clinical studies CALGB8881 CALGB9160 3 3 2 2 β 1 β 1 1 1 0 0 −1 −1 −5 0 5 −5 0 5 β 0 β 0

Infinite mixture models This data could be analysed using two infinite mixture models ∞ w ( 1 ) � CALGB8881: f 1 ( y ) = k ( y | θ j ) j j = 1 and ∞ w ( 2 ) � CALGB9160: f 2 ( y ) = k ( y | θ j ) j j = 1 where • w ( k ) j = 1 w ( k ) > 0 for j = 1 , 2 , . . . and � ∞ = 1 for k = 1 , 2. j j • k ( y | θ ) is a p.d.f. for y with parameter θ . We need to put a prior on the w ( 1 ) , w ( 2 ) and θ (random probability measure).

Some dependent random probability measures: stick-breaking θ are i.i.d. and � � w ( k ) = V ( k ) 1 − V ( k ) � j j i i < j • Hierarchical Dirichlet Process (Teh et al , 2006): V ( k ) � � �� 1 − � j ∼ Be α 0 β j , α 0 l = 1 β l , j j − 1 � β ′ β j = β ′ ( 1 − β ′ j ∼ Be ( 1 , γ ) , l ) , j l = 1 � � V ( 1 ) , V ( 2 ) • Probit stick-breaking processes, etc.: are j j � � V ( 1 ) , V ( 2 ) correlated and independent of for i � = j . i i

Completely random measures µ is a completely random measure (CRM) on Θ if, for any ˜ disjoint subsets A 1 , . . . , A n , ˜ µ ( A 1 ) . . . , ˜ µ ( A n ) are mutually independent. We concentrate on completely random measures (CRM’s) which can be represented in terms of jump sizes J i and jump locations θ i as ∞ � ˜ µ = J i δ θ i i = 1 where δ is Dirac’s delta function and have Lévy-Khintchine representation � [ 1 − e − sf ( θ ) ] α ( d θ ) ρ ( ds ) � ∞ � µ ( d θ ) � � e − f ( θ )˜ = e − E 0 � where α and ρ are measures for which α ( d θ ) < ∞ .

Completely random measures Poisson process with intensity α ( d θ ) ρ ( ds ) . 1.5 1 J 0.5 0 0 0.2 0.4 0.6 0.8 1 3

Examples of CRM’s Many processes that we use in Bayesian nonparametrics are CRM’s • Gamma process - ρ ( ds ) = s − 1 exp {− s } ds . • Beta process - ρ ( ds ) = β s − 1 ( 1 − s ) β − 1 ds . or can be derived from CRM’s • Normalizing a Gamma process, i.e. taking ˜ p = ˜ µ/ ˜ µ (Θ) , leads to a Dirichlet process. • A beta process prior for p 1 , p 2 , . . . can be used to define an Indian buffet process.

Vectors of CRMs It is useful to define d related CRM’s. Suppose that ˜ µ 1 , . . . , ˜ µ d are CRM’s on Θ with marginal Lévy intensities ¯ ν j ( ds , d θ ) = ν j ( ds ) α ( d θ ) Then ˜ µ 1 , . . . , ˜ µ d are a vector of CRM’s if there is a Lévy-Khintchine representation of the form � µ d ( f d ) � = e − ψ ⋆ ρ, d ( f 1 ,..., f d ) e − ˜ µ ( f 1 ) −···− ˜ E where � � � 1 − e − s 1 f 1 ( θ ) −···− s d f d ( θ ) � ψ ⋆ ρ, d ( f 1 , . . . , f d ) = α ( d θ ) ρ d ( ds 1 , . . . ds d ) ( R + ) d and � ν j ( ds ) = ρ d ( ds 1 , . . . , ds d ) .

Compound Random Measures: Definition A compound random measure (CoRM) is a vector of CRM’s with intensity � � s 1 z , . . . , s d � z − d h ds 1 . . . ds d ν ⋆ ( dz ) ρ d ( ds 1 , . . . , ds d ) = z where • s 1 , . . . , s d are called scores. • H is a score distribution with density h . • ν ⋆ is the Lévy intensity of a directing Lévy process. which satisfies the condition � s 1 z , . . . , s d min ( 1 , � s � ) z − d h � � ν ⋆ ( dz ) < ∞ where � s � is the z Euclidean norm of the vector s = ( s 1 , . . . , s d ) .

A representation of a CoRM Realizations of a CoRM can be expressed as ∞ � µ j = ˜ m j , i J i δ θ i i = 1 where i . i . d . • m 1 , i , . . . , m d , i ∼ H η = � ∞ i = 1 J i δ θ i is a CRM with Lévy intensity ν ⋆ ( ds ) α ( d θ ) . • ˜

CoRMs with independent gamma scores We will concentrate on the class of CoRMs for which d � h ( s 1 / z , . . . , s d / z ) = f ( s j / z ) j = 1 where f is the p.d.f. of a gamma distribution with shape φ , Γ( φ ) x φ − 1 exp {− x } . 1 f ( x ) =

Properties of CoRMs with independent score distributions • The Lévy copula can be expressed as a univariate integral. e ts z − 1 f ( s / z ) ds be the moment generating • Let M f � z ( t ) = function of z − 1 f ( s / z ) then � � � 1 − e − s 1 λ 1 −···− s d λ d � ψ ρ, d ( λ 1 , . . . , λ d ) = ρ d ( ds 1 , . . . ds d ) ( R + ) d �   d �  ν ⋆ ( z ) dz M f = ψ ρ, d ( λ 1 , . . . , λ d ) =  1 − z ( − λ j ) j = 1 • This expression can be used to calculate quantities such as Corr (˜ µ k ( A ) , ˜ µ m ( A )) .

CoRMs with gamma distributed scores Consider a CoRM process with independent Ga ( φ, 1 ) distributed scores. If the CoRM process has gamma process marginals then ( � d j = 1 s j ) φ − 1 | s | − d φ + 1 2 e − | s | 2 W ( d − 2 ) φ + 1 ρ d ( s 1 , . . . , s d ) = 2 ( | s | ) , − d φ [Γ( φ )] d − 1 2 (1) where | s | = s 1 + · · · + s d and W is the Whittaker function. If the CoRM process has σ -stable process marginals then ( � d j = 1 s j ) φ − 1 σ Γ( σ + d φ ) Γ( σ + φ )Γ( 1 − σ ) | s | − σ − d φ . ρ d ( s 1 , . . . , s d ) = (2) [Γ( φ )] d − 1

CoRMs with exponentially distributed scores Consider a CoRM process with independent exponentially distributed scores. If the CoRM has gamma process marginals we recover the multivariate Lévy intensity of Leisen et al (2013), d − 1 ( d − 1 )! � ( d − 1 − j )! | s | − j − 1 e −| s | . ρ d ( s 1 , . . . , s d ) = j = 0 Otherwise, if σ -stable marginals are considered then we recover the multivariate vector introduced in Leisen and Lijoi (2011) and Zhu and Leisen (2014), ( σ ) d Γ( 1 − σ ) | s | − σ − d . ρ d ( s 1 , . . . , s d ) =

CoRMs with independent gamma scores: specific marginals The Lévy intensity of ˜ µ j � z − 1 f ( s / z ) ds ν ⋆ ( dz ) = ν ( ds ) . ν j ( ds ) = If we have independent gamma scores, the directing Lévy intensity ν ⋆ is linked to the marginal Lévy intensity by � Γ( φ ) � 1 � � = t 2 − φ L − 1 ν ⋆ s φ − 1 ν ( s ) ( t ) t where L − 1 is the inverse Laplace transform.

CoRMs with independent gamma scores: specific marginals The intensity of the directing Lévy process is ν ⋆ ( z ) = z − 1 ( 1 − z ) φ − 1 , 0 < z < 1 leads to a marginal gamma process for which ν ( s ) = s − 1 exp {− s } , s > 0 Remarks • ν ⋆ is the the Lévy intensity of a beta process. • If ν ⋆ is the Lévy intensity of a Stable-Beta process (Teh and Görür, 2009), the marginal process is a generalized gamma process.

NCoRM: Gamma marginal, φ = 1 DLP 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Dim. 1 Dim. 2 Dim. 3 5 5 5 4.5 4.5 4.5 4 4 4 3.5 3.5 3.5 3 3 3 2.5 2.5 2.5 2 2 2 1.5 1.5 1.5 1 1 1 0.5 0.5 0.5 0 0 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

NCoRM: Gamma marginal, φ = 10 DLP 0.01 0.009 0.008 0.007 0.006 0.005 0.004 0.003 0.002 0.001 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Dim. 1 Dim. 2 Dim. 3 0.12 0.12 0.12 0.1 0.1 0.1 0.08 0.08 0.08 0.06 0.06 0.06 0.04 0.04 0.04 0.02 0.02 0.02 0 0 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

NCoRM: Gamma marginal, φ = 50 DLP −3 x 10 8 7 6 5 4 3 2 1 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Dim. 1 Dim. 2 Dim. 3 0.5 0.5 0.5 0.45 0.45 0.45 0.4 0.4 0.4 0.35 0.35 0.35 0.3 0.3 0.3 0.25 0.25 0.25 0.2 0.2 0.2 0.15 0.15 0.15 0.1 0.1 0.1 0.05 0.05 0.05 0 0 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Marginal beta process A CoRM with beta process marginals ( ν ( s ) = β s − 1 ( 1 − s ) β − 1 ) can be constructed using • A beta score distribution with parameters α and 1 • A directing Lévy intensity ν ⋆ ( z ) = β z − 1 ( 1 − z ) β − 1 + β ( β − 1 ) ( 1 − z ) β − 2 α i.e. a superposition of a beta process and a compound Poisson process with beta jump distribution.

Links to other processes Other processes can be expressed as CoRM’s: • Superpositions/Thinning: e.g. Griffin et al (2013), Chen et al (2014), Lijoi and Nipoti (2014), Lijoi et al (2014a, b) using mixture score distributions h ( s ) = πδ s = 0 + ( 1 − π ) h ⋆ ( s ) . • Lévy copulae: e.g. Leisen and Lijoi (2011), Leisen et al (2013), Zhu and Leisen (2014).

Normalized Compound Random Measures (NCoRM) A vector of random probability measures can be defined by normalizing each dimension of the CoRM so that ∞ µ k ˜ w ( k ) � p k = µ k (Θ) = δ θ j . j ˜ j = 1

CoRMs on more general spaces For a more general space X , we define ˜ µ ( · ; x ) to be a completely random measure for x ∈ X . The collection { ˜ µ ( · ; x ) | x ∈ X } can be given a CoRM prior with ∞ � µ ( · ; x ) = ˜ m j ( x ) J j δ θ j j = 1 where m k ( x ) is a realisation of a random process on X . Example X = R p , m k ( x ) = exp { r k ( x ) } where r k ( x ) is given a zero-mean Gaussian process prior (see Ranganath and Blei, 2015).