Smoothing sample extremes: the mixed model approach Francesco Pauli Fabrizio Laurini Dept of Statistical Sciences, Dept of Economics, University of Padova University of Parma fpauli@stat.unipd.it flaurini@stat.unipd.it EVA 2005 F. Pauli & F. Laurini - p. 1/34
Outline of this talk ■ Smoothing semipararametric tools & splines Basics Outline EVA 2005 F. Pauli & F. Laurini - p. 2/34
Outline of this talk ■ Smoothing semipararametric tools & splines Basics Outline ■ Generalized linear mixed models (GLMM) for spline estimation EVA 2005 F. Pauli & F. Laurini - p. 2/34
Outline of this talk ■ Smoothing semipararametric tools & splines Basics Outline ■ Generalized linear mixed models (GLMM) for spline estimation ■ GLMM for smoothing sample extremes EVA 2005 F. Pauli & F. Laurini - p. 2/34
Outline of this talk ■ Smoothing semipararametric tools & splines Basics Outline ■ Generalized linear mixed models (GLMM) for spline estimation ■ GLMM for smoothing sample extremes ■ Some bits of Bayesian inference and MCMC EVA 2005 F. Pauli & F. Laurini - p. 2/34
Outline of this talk ■ Smoothing semipararametric tools & splines Basics Outline ■ Generalized linear mixed models (GLMM) for spline estimation ■ GLMM for smoothing sample extremes ■ Some bits of Bayesian inference and MCMC ■ Simulations and applications to pollutants EVA 2005 F. Pauli & F. Laurini - p. 2/34
Parametric modeling Popular models for response y i , i = 1 , . . . , n have form Smoothing modeling y i = f ( x i ) + error i Regression stick ■ f can be any parametric function spline knots placement ■ many way to characterize error; examples: penalized splines ◆ Gaussian iid (simple regression) optimization generalization ◆ Correlated errors LMM ◆ Exponential family (generalized linear models) spline & LMM extensions breath EVA 2005 F. Pauli & F. Laurini - p. 3/34
Regression Take simple linear regression Smoothing modeling f ( x ) = β 0 + β 1 x Regression stick Smoothing features: spline ■ f is smooth; often unsuited to model real data knots placement penalized splines ■ f is a linear combination of basis functions optimization generalization � � LMM 1 x spline & LMM extensions breath EVA 2005 F. Pauli & F. Laurini - p. 4/34
Regression Take simple linear regression Smoothing modeling f ( x ) = β 0 + β 1 x Regression stick Smoothing features: spline ■ f is smooth; often unsuited to model real data knots placement penalized splines ■ f is a linear combination of basis functions optimization generalization � � LMM 1 x spline & LMM extensions An extension is polynomial regression breath f ( x ) = β 0 + β 1 x + β 2 x 2 + . . . + β p x p Smoothing features: ■ f is smooth and suited to model non-linearity ■ f is a linear combination of basis functions � x p � x 2 · · · 1 x EVA 2005 F. Pauli & F. Laurini - p. 4/34
Broken linear stick modeling Linear model for a “structural change” at time κ τ (broken Smoothing modeling stick) Regression stick f ( x ) = β 0 + β 1 x + b τ ( x − κ τ ) + () + the positive part of ( x − κ τ ) spline knots placement penalized splines Smoothing features: optimization ■ f is rough and suited to explain structural changes generalization LMM ■ f is a linear combination of basis functions spline & LMM extensions � � 1 x ( x − κ τ ) + breath ( x − κ τ ) + is a linear spline basis function EVA 2005 F. Pauli & F. Laurini - p. 5/34
Linear spline modeling Extension to linear spline regression by adding knots Smoothing modeling κ 1 , . . . , κ K Regression K stick � b k ( x − κ k ) + spline f ( x ) = β 0 + β 1 x + knots placement k =1 penalized splines Smoothing features: optimization generalization ■ f is piecewise linear, and more flexible than broken stick LMM spline & LMM ■ f is a linear combination of basis functions extensions � � breath 1 x ( x − κ 1 ) + . . . ( x − κ K ) + EVA 2005 F. Pauli & F. Laurini - p. 6/34
Linear spline modeling Extension to linear spline regression by adding knots Smoothing modeling κ 1 , . . . , κ K Regression K stick � b k ( x − κ k ) + spline f ( x ) = β 0 + β 1 x + knots placement k =1 penalized splines Smoothing features: optimization generalization ■ f is piecewise linear, and more flexible than broken stick LMM spline & LMM ■ f is a linear combination of basis functions extensions � � breath 1 x ( x − κ 1 ) + . . . ( x − κ K ) + “Definition” ■ The set of functions { ( x − κ j ) + } , j = 1 , . . . , K is a linear spline basis ■ A linear combination of such basis functions is a piecewise linear function ■ Commonly called spline with knots at κ 1 , . . . , κ K EVA 2005 F. Pauli & F. Laurini - p. 6/34
How do we choose knots? Knots selection and their placement have drawbacks Smoothing modeling ■ Somehow ad hoc solution Regression ■ Overfitting stick spline ■ Might be time consuming knots placement penalized splines optimization generalization LMM spline & LMM extensions breath EVA 2005 F. Pauli & F. Laurini - p. 7/34
Penalized spline regression Let β = ( β 0 , β 1 , b 1 , . . . , b K ). Smoothing modeling Wiggly fit is avoided by constraints on b k such as Regression K stick � b 2 spline k < C finite C knots placement k =1 penalized splines Minimization is written as optimization generalization LMM β T Dβ ≤ C minimize � y − Xβ � 2 subject to spline & LMM extensions where D is a squared positive matrix with K + 2 rows breath � � 0 2 × 2 0 D = 0 I K × K Smoothing features: ■ f is smooth and pleasing ■ The amount of smoothness is controlled by C , and does not depend on number/placement of knots EVA 2005 F. Pauli & F. Laurini - p. 8/34
Solution of constrained optimization With Lagrange multiplier argument, for some λ ≥ 0 , choose β Smoothing modeling to minimize Regression stick � y − Xβ � 2 + λ 2 β T Dβ β λ = ( X T X + λ 2 D ) − 1 X T y. with solution ˆ spline knots placement ■ λ 2 β T Dβ is the roughness penalty term penalized splines optimization ■ λ is the smoothing parameter generalization LMM ■ Connections with ridge regression spline & LMM extensions breath λ small λ big EVA 2005 F. Pauli & F. Laurini - p. 9/34
Generalization of spline models Generalization may involve Smoothing modeling Use of different basis Regression stick ■ Use of B -spline (numerical stability) spline ■ Use of natural cubic splines (arise as solution of knots placement penalized splines optimization problem) optimization generalization Use of different penalties LMM ■ Penalize “some difference” in spline coefficients spline & LMM extensions ■ Penalizing degree of spline function. Example: q -th breath derivative of f EVA 2005 F. Pauli & F. Laurini - p. 10/34
Penalized splines as Linear Mixed Model Take the Linear Mixed Model (LMM) Smoothing modeling y = Xβ + Zu + ε Regression stick and assume spline knots placement penalized splines � � � � � � � � G = σ 2 u u 0 G 0 u I optimization E = , COV = and R = σ 2 generalization ε ε 0 0 R ε I LMM spline & LMM ■ Xβ is the fixed component extensions breath ■ u is the random component or random effect EVA 2005 F. Pauli & F. Laurini - p. 11/34
Spline embedded in LMM For LMM Smoothing modeling Regression y = Xβ + Zu + ε stick spline K knots placement � u k ( x i − κ k ) + + ε i y i = β 0 + β 1 x i + penalized splines k =1 optimization generalization splines are the Best Linear Unbiased Predictor where u is a LMM spline & LMM vector of random coefficients. extensions Details: breath ( x 1 − κ 1 ) + · · · ( x 1 − κ K ) + 1 x 1 . . . . ... . . . . X = and Z = . . . . ( x n − κ 1 ) + · · · ( x n − κ K ) + 1 x n ■ In general u k i.i.d. N (0 , σ 2 u ) ■ σ u = ∞ leads to wiggly (over)fit ■ Finite σ u shrinks u k (smooth fit) EVA 2005 F. Pauli & F. Laurini - p. 12/34
Sketch of extensions to LMM (increasing complexity) ■ Semiparametric models: one covariate is nonparametric Smoothing modeling i ) + X [ − 1] y i = f ( X 1 β + ε i , f smooth Regression i stick spline ■ Semiparamtric mixed models: add random effects u knots placement ■ Additive models: covariates as penalized linear splines, e.g. penalized splines optimization generalization y i = c + f 1 ( x i ) + f 2 ( t i ) + ε i ; f 1 and f 2 are smooth LMM spline & LMM extensions ■ Generalized parametric regression with random effects breath (GLMM), e.g. exp { ( Xβ + Zu ) i } � � y i | u ∼ Ber u ∼ MV N ( 0 , G θ ) 1 + exp { ( Xβ + Zu ) i } ■ Combine all above ingredients, eventually with a Bayesian approach. EVA 2005 F. Pauli & F. Laurini - p. 13/34
Recommend
More recommend
