Block I Connections between multivariate and FDA Beatriz - PowerPoint PPT Presentation

Block I Connections between multivariate and FDA Beatriz Bueno-Larraz Real Eyes Universidad Aut´ onoma de Madrid IWAFDA 2019

“It turns out, in my opinion, that reproducing kernel Hilbert spaces are the natural setting in which to solve problems of statistical inference on time processes”. Parzen (1962) B. Bueno-Larraz Block III: Connections between multivariate and FDA IWAFDA 1 / 43

1. Introduction to RKHS’s 2. Logistic regression Berrendero, J. R., Bueno-Larraz, B., Cuevas, A. (2018). On func- tional logistic regression via RKHS’s. arXiv:1812.00721 . 3. Mahalanobis distance Berrendero, J. R., Bueno-Larraz, B., Cuevas, A. (2018). On Maha- lanobis distance in functional settings. arXiv:1803.06550 . 4. Binary classification Berrendero, J. R., Cuevas, A., Torrecilla, J.L. (2017). On the use of reproducing kernel Hilbert spaces in functional classification. Jour- nal of the American Statistical Association Delaigle, A., Hall, P. (2012). Achieving near perfect classification for functional data. Journal of the Royal Statistical Society: Series B B. Bueno-Larraz Block III: Connections between multivariate and FDA IWAFDA 2 / 43

Our setting Our sample is made of trajectories x ∈ L 2 [ 0 , 1 ] , drawn from a second order process X ( s ) with • mean function m ( s ) = E [ X ( s )] , • covariance function K ( s , t ) = cov ( X ( s ) , X ( t )) . Standard Brownian motion: m ( s ) = 0 K ( s , t ) = min ( s , t ) B. Bueno-Larraz Block III: Connections between multivariate and FDA IWAFDA 3 / 43

RKHS via the covariance operator The functional analogue of the cov. matrix is the covariance operator. � 1 x = Σ y − → K f ( t ) = K ( t , s ) f ( s ) d s . 0 Definition in Peszat and Zabczyk (2007) The RKHS associated with K is defined as H ( K ) = {K 1 / 2 f , f ∈ L 2 [ 0 , 1 ] } . λ 1 ≥ λ 2 . . . > 0 and { e j } are the eigenvalues and eigenfunctions of K , then for f ∈ H ( K ) , ∞ � f , e i � 2 K = �K − 1 / 2 f � 2 � � f � 2 2 2 = , λ i j = 1 since { e j } is an ONB in L 2 [ 0 , 1 ] . B. Bueno-Larraz Block III: Connections between multivariate and FDA IWAFDA 4 / 43

RKHS via the covariance operator The functional analogue of the cov. matrix is the covariance operator. � 1 x = Σ y − → K f ( t ) = K ( t , s ) f ( s ) d s . 0 Definition in Peszat and Zabczyk (2007) The RKHS associated with K is defined as H ( K ) = {K 1 / 2 f , f ∈ L 2 [ 0 , 1 ] } . λ 1 ≥ λ 2 . . . > 0 and { e j } are the eigenvalues and eigenfunctions of K , then for f ∈ H ( K ) , ∞ � f , e i � 2 K = �K − 1 / 2 f � 2 � � f � 2 2 2 = , λ i j = 1 since { e j } is an ONB in L 2 [ 0 , 1 ] . B. Bueno-Larraz Block III: Connections between multivariate and FDA IWAFDA 4 / 43

Finite dimensional case R d L 2 [ 0 , 1 ] Matrix Σ Operator K H (Σ) = { Σ 1 / 2 a , a ∈ R d } H ( K ) = {K 1 / 2 f , f ∈ L 2 [ 0 , 1 ] } = { Σ a , a ∈ R d } � x � 2 Σ = (Σ − 1 / 2 x ) ′ (Σ − 1 / 2 x ) � g � 2 K = �K − 1 / 2 g � 2 2 In the functional case: when x is the trajectory of a Gaussian process, x �∈ H ( K ) with probability 1. ( Luki´ c and Beder (2001)) Brownian Motion: f | f ( 0 ) = 0 , f absolutely continuous, f ′ ∈ L 2 [ 0 , 1 ] � � H ( K ) = . B. Bueno-Larraz Block III: Connections between multivariate and FDA IWAFDA 5 / 43

Finite dimensional case R d L 2 [ 0 , 1 ] Matrix Σ Operator K H (Σ) = { Σ 1 / 2 a , a ∈ R d } H ( K ) = {K 1 / 2 f , f ∈ L 2 [ 0 , 1 ] } = { Σ a , a ∈ R d } � x � 2 Σ = (Σ − 1 / 2 x ) ′ (Σ − 1 / 2 x ) � g � 2 K = �K − 1 / 2 g � 2 2 In the functional case: when x is the trajectory of a Gaussian process, x �∈ H ( K ) with probability 1. ( Luki´ c and Beder (2001)) Brownian Motion: f | f ( 0 ) = 0 , f absolutely continuous, f ′ ∈ L 2 [ 0 , 1 ] � � H ( K ) = . B. Bueno-Larraz Block III: Connections between multivariate and FDA IWAFDA 5 / 43

Classical definition of RKHS A function r : [ 0 , 1 ] × [ 0 , 1 ] → R is a reproducing kernel of the Hilbert space H ⊂ L 2 [ 0 , 1 ] if and only if it satisfies: 1. r ( s , · ) ∈ H , ∀ s ∈ [ 0 , 1 ] . 2. Reproducing property: � f , r ( s , . ) � = f ( s ) , ∀ f ∈ H , ∀ s ∈ [ 0 , 1 ] . A Hilbert space of real-valued functions with a reproducing kernel is called a reproducing kernel Hilbert space (RKHS) . B. Bueno-Larraz Block III: Connections between multivariate and FDA IWAFDA 6 / 43

Characterizing reproducing kernels A reproducing kernel is always positive semidefinite, that is, for all { s 1 , . . . , s p } ∈ [ 0 , 1 ] p , the matrix ( r ( s i , s j )) p i , j = 1 is positive semidefinite. Moore-Aronszajn theorem Every positive semidefinite function is the kernel of a unique RKHS. Every stochastic process has a natural associated RKHS, H ( K ) , whose kernel is the covariance function of the process, K ( s , t ) . B. Bueno-Larraz Block III: Connections between multivariate and FDA IWAFDA 7 / 43

RKHS via finite linear combinations If K ( s , t ) is the covariance function, we define ( H 0 ( K ) , �· , ·� ) by n � � � H 0 ( K ) ≡ f : f ( s ) = a j K ( s , t j ) , a j ∈ R , t j ∈ [ 0 , 1 ] , n ∈ N , j = 1 � f , g � K = � i , j a i b j K ( s i , t j ) , where f ( · ) = � i a i K ( · , s i ) and g ( · ) = � j b j K ( · , t j ) . The RKHS associated with K is defined as the completion of H 0 ( K ) . B. Bueno-Larraz Block III: Connections between multivariate and FDA IWAFDA 8 / 43

Loève’s isometry L ( X ) is the L 2 -completion of the linear span of X ( s ) . Loéve’s Isometry The RKHS H ( K ) is an isometric copy of L ( X ) , since � p p �� � � � Ψ X a i X ( t i ) − m ( t i ) = a i K ( t i , · ) , ∀ a i ∈ R i = 1 i = 1 defines a congruence. We identify � β, X � K ≡ Ψ − 1 X ( β ) . B. Bueno-Larraz Block III: Connections between multivariate and FDA IWAFDA 9 / 43

Introduction to RKHS’s 1 Logistic regression 2 Mahalanobis distance 3 Binary classification 4 B. Bueno-Larraz Block III: Connections between multivariate and FDA IWAFDA 10 / 43

Logistic regression problem Used for problems with categorical response (typically Y ∈ { 0 , 1 } ). � � is linear in x ∈ R d , where It is assumed that log p ( x ) / ( 1 – p ( x )) p ( x ) = P ( Y = 1 | x ) , which leads to: 1 α 0 ∈ R and α ∈ R d . P ( Y = 1 | x ) = 1 + exp { – α 0 – α ′ x } , It holds when X 0 , X 1 are Gaussian with common covariance matrix. B. Bueno-Larraz Block III: Connections between multivariate and FDA IWAFDA 11 / 43

Goals of this part To analyze the relationship between two functional extensions of the multivariate model. To carefully examine whether ML estimators exist. To see how to circumvent the non-existence issue using multivariate problems. B. Bueno-Larraz Block III: Connections between multivariate and FDA IWAFDA 12 / 43

Functional logistic regression The standard functional extension for x ∈ L 2 [ 0 , 1 ] is, 1 β 0 ∈ R and β ∈ L 2 [ 0 , 1 ] . P ( Y = 1 | x ) = 1 + exp { – β 0 – � β, x � 2 } , The RKHS model would be, 1 P ( Y = 1 | x ) = 1 + exp { – β 0 – � β, x � K } , β 0 ∈ R and β ∈ H ( K ) . B. Bueno-Larraz Block III: Connections between multivariate and FDA IWAFDA 13 / 43

Relationship with the finite dimensional model (I) For Σ invertible, writing α ∈ H (Σ) as d � α = Σ a = a i Σ i , for Σ i the i-th column of Σ , i = 1 we have � ′ � Σ − 1 / 2 Σ a Σ − 1 / 2 x = a ′ x . � � � α, x � K = Thus, the standard logistic regression model is a particular case of the RKHS one: �� − 1 = �� − 1 . – α 0 – a ′ x � � � � P ( Y = 1 | x ) = 1 +exp – α 0 – � α, x � K 1 +exp B. Bueno-Larraz Block III: Connections between multivariate and FDA IWAFDA 14 / 43

Relationship with the finite dimensional model (II) Whenever the slope function β ∈ H ( K ) ⊂ L 2 [ 0 , 1 ] has the form p � β ( · ) = β j K ( t j , · ) , i = 1 the RKHS model reduces to, p �� − 1 � � � P ( Y = 1 | x ) = 1 + exp – β 0 – β j x ( t j ) . j = 1 B. Bueno-Larraz Block III: Connections between multivariate and FDA IWAFDA 15 / 43

Conditional Gaussian distributions Let X 0 ( s ) , X 1 ( s ) be Gaussian processes with continuous trajectories, continu- ous mean functions m 0 , m 1 and continuous covariance function K (equal for both classes). Let P 0 and P 1 be the probability measures on C [ 0 , 1 ] (or L 2 [ 0 , 1 ] ) induced by the processes X 0 , X 1 respectively. (a) if m 0 , m 1 ∈ H ( K ) , then the RKHS model holds with β := m 1 – m 0 and β 0 := ( � m 0 � 2 K – � m 1 � 2 K ) / 2 – log(( 1 – p ) / p ) . (b) if m 1 – m 0 ∈ K ( L 2 ) = {K ( f ) : f ∈ L 2 [ 0 , 1 ] } , then L 2 model holds. (c) if m 1 – m 0 �∈ K ( L 2 ) , L 2 model is never recovered, but different sit- uations are possible. B. Bueno-Larraz Block III: Connections between multivariate and FDA IWAFDA 16 / 43