Efficient Sparse Approximations for Convolution Processes Mauricio - PowerPoint PPT Presentation

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Efficient Sparse Approximations for Convolution Processes Mauricio A. Álvarez Joint work with Neil Lawrence, David Luengo and Michalis Titsias University of Manchester

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Introduction: covariances for multiple outputs D 1 = { ( x 1 i , y 1 i ) | i = 1 , . . . , N 1 }

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Introduction: covariances for multiple outputs D 1 = { ( x 1 i , y 1 i ) | i = 1 , . . . , N 1 }

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Introduction: covariances for multiple outputs k 1 ( x i , x j ) D 1 = { ( x 1 i , y 1 i ) | i = 1 , . . . , N 1 }

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Introduction: covariances for multiple outputs k 1 ( x i , x j ) D 1 = { ( x 1 i , y 1 i ) | i = 1 , . . . , N 1 }

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Introduction: covariances for multiple outputs K 1 = k 1 ( x i , x j ) D 1 = { ( x 1 i , y 1 i ) | i = 1 , . . . , N 1 }

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Introduction: covariances for multiple outputs D 1

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Introduction: covariances for multiple outputs D 1 D 2

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Introduction: covariances for multiple outputs D 1 D 2 D 3

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Introduction: covariances for multiple outputs D 1 K 1 = D 2 K 2 = D 3 K 3 =

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Introduction: covariances for multiple outputs D 1 D 2 K 1 K 2 = D 3 K 3 =

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Introduction: covariances for multiple outputs D 1 D 2 K 1 K 2 D 3 K 3 =

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Introduction: covariances for multiple outputs D 1 D 2 K 1 K 2 K 3 D 3

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Introduction: covariances for multiple outputs D 1 D 2 K 1 K 2 K = K 3 D 3

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Introduction: covariances for multiple outputs D 1 Joint covariance D 2 ? ? K 1 ? K 2 ? K = ? ? K 3 D 3 K be a valid covariance matrix

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Some approaches Linear model of coregionalization. ❑ Intrinsic coregionalization model. ❑ Multitask kernels. ❑ Convolution of covariances. ❑ Convolution of processes or convolution process. ❑

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Convolution Process A convolution process is a moving-average construction that ❑ guarantees a valid covariance function. Consider a set of functions { f d ( x ) } D d = 1 . ❑ Each function can be expressed as ❑ � f d ( x ) = G d ( x − z ) u ( z ) d z = G d ( x ) ∗ u ( x ) . X Influence of more than one latent function, { u q ( z ) } Q q = 1 and inclusion of ❑ an independent process w d ( x ) � Q � y d ( x ) = f d ( x ) + w d ( x ) = G d , q ( x − z ) u q ( z ) d z + w d ( x ) . X q = 1

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data A pictorial representation u(x) u(x): latent function.

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data A pictorial representation G (x) 1 u(x) G (x) 2 u(x): latent function. G(x): smoothing kernel.

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data A pictorial representation G (x) 1 f x) ( 1 u(x) f x) ( 2 G (x) 2 u(x): latent function. G(x): smoothing kernel. f(x): output function.

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data A pictorial representation G (x) 1 w x) ( f x) ( 1 1 u(x) f x) ( w x) ( 2 2 G (x) 2 u(x): latent function. G(x): smoothing kernel. f(x): output function. w(x): independent process.

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data A pictorial representation y x) ( 1 G (x) 1 w x) ( f x) ( 1 1 u(x) f x) ( w x) ( 2 2 G (x) 2 y x) ( 2 u(x): latent function. y(x): noisy output function. G(x): smoothing kernel. f(x): output function. w(x): independent process.

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Covariance of the output functions. The covariance between y d ( x ) and y d ′ ( x ′ ) is given as cov [ y d ( x ) , y d ′ ( x ′ )] = cov [ f d ( x ) , f d ′ ( x ′ )] + cov [ w d ( x ) , w d ′ ( x ′ )] δ d , d ′ where � � G d ′ ( x ′ − z ′ ) cov [ u ( z ) , u ( z ′ )] d z ′ d z cov [ f d ( x ) , f d ′ ( x ′ )] = G d ( x − z ) X X

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Different forms of covariance for the output functions. Input Gaussian process ❑ � � G d ′ ( x ′ − z ′ ) k u , u ( z , z ′ ) d z ′ d z cov [ f d , f d ′ ] = G d ( x − z ) X X Input white noise process ❑ � G d ( x − z ) G d ′ ( x ′ − z ) d z cov [ f d , f d ′ ] = X Covariance between output functions and latent functions ❑ � cov [ f d , u ] = G d ( x − z ′ ) k u , u ( z ′ , z ) d z ′ X

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Likelihood of the full Gaussian process. The likelihood of the model is given by ❑ p ( y | X , φ ) = N ( 0 , K f , f + Σ ) � � ⊤ is the set of output functions, K f , f covariance where y = y ⊤ 1 , . . . , y ⊤ D matrix with blocks cov [ f d , f d ′ ] , Σ matrix of noise variances, φ is the set of parameters of the covariance matrix and X = { x 1 , . . . , x N } is the set of input vectors. Learning from the log-likelihood involves the inverse of K f , f + Σ , which ❑ grows with complexity O ( N 3 D 3 )

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Predictive distribution of the full Gaussian process. Predictive distribution at X ∗ ❑ p ( y ∗ | y , X , X ∗ , φ ) = N ( µ ∗ , Λ ∗ ) with µ ∗ = K f ∗ , f ( K f , f + Σ ) − 1 y Λ ∗ = K f ∗ , f ∗ − K f ∗ , f ( K f , f + Σ ) − 1 K f , f ∗ + Σ Prediction is O ( DN ) for the mean and O ( D 2 N 2 ) for the variance, for ❑ one test point. Storage is O ( D 2 N 2 ) .

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Partial independence 1 Fully Independence 2 Variational Approximation 3 Variational Inducing Kernels 4 Case study: a dynamic model for financial data 5

Outline Partial independence Fully Independence Variational Approximation Variational Inducing Kernels Case study: a dynamic model for financial data Partial independence 1 Fully Independence 2 Variational Approximation 3 Variational Inducing Kernels 4 Case study: a dynamic model for financial data 5