Sparse Robust Regression using Non-concave Penalized Density Power Divergence Subhabrata Majumdar Joint work with Abhik Ghosh University of Florida Informatics Institute IISA-2018 conference, Gainesville, FL May 19, 2018 Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Table of contents Motivation 1 Formulation 2 Influence functions 3 Theory 4 Simulations 5 Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Outline Motivation 1 Formulation 2 Influence functions 3 Theory 4 Simulations 5 Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Penalized linear regression Standard linear regression model (LRM): y = X β + ǫ , where y = ( y 1 , . . . , y n ) T are responses, X = ( x 1 · · · x n ) T is the design matrix, and ǫ = ( ǫ 1 , . . . , ǫ n ) T ∼ N n ( 0 , σ 2 I n ) are the random error components. Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Penalized linear regression Standard linear regression model (LRM): y = X β + ǫ , where y = ( y 1 , . . . , y n ) T are responses, X = ( x 1 · · · x n ) T is the design matrix, and ǫ = ( ǫ 1 , . . . , ǫ n ) T ∼ N n ( 0 , σ 2 I n ) are the random error components. Sparse estimators of β = ( β 1 , . . . , β p ) T , are defined as the minimizer of: n p � � ρ ( y i − x T i β ) + λ n p ( | β j | ) , i = 1 j = 1 where ρ ( . ) is a loss function, p ( . ) is the sparsity inducing penalty function, and λ n ≡ λ is the regularization parameter depending on n . Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Sparse penalized least squares Linear model : y = X β + ǫ , with X ∈ R n × p , β ∈ R p , ǫ ∼ N ( 0 , σ 2 I ) with σ > 0; Lasso (Tibshirani, 1996) n argmin β � y − X β � 2 + λ � β � 1 ; � β = 1 Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Sparse penalized least squares Linear model : y = X β + ǫ , with X ∈ R n × p , β ∈ R p , ǫ ∼ N ( 0 , σ 2 I ) with σ > 0; Lasso (Tibshirani, 1996) n argmin β � y − X β � 2 + λ � β � 1 ; � β = 1 SCAD (Fan and Li, 2001) � β = n � y − X β � 2 + λ � p 1 argmin β j = 1 p ( | β j | ) ; MCP (Zhang, 2010) Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Sparse Robust Regression Sparse versions of robust regression methods- RLARS (Khan et al., 2007), sparse least trimmed squares (Wang et al., 2007), LAD-lasso (Alfons et al., 2013); Robust high-dimensional M-estimation- Neghaban et al. (2012); Bean et al. (2013); Donoho and Montanari (2016); Lozano et al. (2016); Loh and Wainwright (2017) Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Why do we need another? Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Why do we need another? All methods until now focus on ℓ 1 -penalization. But the bias of lasso-type 1 estimators is well-known. Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Why do we need another? All methods until now focus on ℓ 1 -penalization. But the bias of lasso-type 1 estimators is well-known. Many proposed methods lack theoretical rigor and only give algorithms. 2 Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Why do we need another? All methods until now focus on ℓ 1 -penalization. But the bias of lasso-type 1 estimators is well-known. Many proposed methods lack theoretical rigor and only give algorithms. 2 Robustness is either shown empirically or theoretically- not both. 3 Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Why do we need another? All methods until now focus on ℓ 1 -penalization. But the bias of lasso-type 1 estimators is well-known. Many proposed methods lack theoretical rigor and only give algorithms. 2 Robustness is either shown empirically or theoretically- not both. 3 Conditions assumed on the design matrix are largely similar to 4 non-robust cases. Example X T X / n → C (Alfons et al., 2013) Restricted eigenvalue condition (Lozano et al., 2016) Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Outline Motivation 1 Formulation 2 Influence functions 3 Theory 4 Simulations 5 Ghosh and Majumdar Robust Sparse Regression May 19, 2018
The DPD loss function D ensity P ower D ivergence is a generalization of the KL-divergence. DPD-based regression (Durio and Isaia, 2011) maximizes the loss function � � n � 1 − ( 1 + α ) 3 / 2 ( yi − xT i β ) 2 1 1 e − α L α n ( β , σ ) = ( 2 π ) α/ 2 σ α √ 2 σ 2 α n 1 + α i = 1 Ghosh and Majumdar Robust Sparse Regression May 19, 2018
The DPD loss function D ensity P ower D ivergence is a generalization of the KL-divergence. DPD-based regression (Durio and Isaia, 2011) maximizes the loss function � � n � 1 − ( 1 + α ) 3 / 2 ( yi − xT i β ) 2 1 1 e − α L α n ( β , σ ) = ( 2 π ) α/ 2 σ α √ 2 σ 2 α n 1 + α i = 1 Why use DPD? Ghosh and Majumdar Robust Sparse Regression May 19, 2018
The DPD loss function D ensity P ower D ivergence is a generalization of the KL-divergence. DPD-based regression (Durio and Isaia, 2011) maximizes the loss function � � n � 1 − ( 1 + α ) 3 / 2 ( yi − xT i β ) 2 1 1 e − α L α n ( β , σ ) = ( 2 π ) α/ 2 σ α √ 2 σ 2 α n 1 + α i = 1 Why use DPD? Adaptive: Large α = more robust, less efficient. Small α = more robust, less efficient. Ghosh and Majumdar Robust Sparse Regression May 19, 2018
The DPD loss function D ensity P ower D ivergence is a generalization of the KL-divergence. DPD-based regression (Durio and Isaia, 2011) maximizes the loss function � � n � 1 − ( 1 + α ) 3 / 2 ( yi − xT i β ) 2 1 1 e − α L α n ( β , σ ) = ( 2 π ) α/ 2 σ α √ 2 σ 2 α n 1 + α i = 1 Why use DPD? Adaptive: Large α = more robust, less efficient. Small α = more robust, less efficient. Generalized: As α ↓ 0, L α n ( β , σ ) coincides (in a limiting sense) with the negative log-likelihood. (why? think L-Hospital’s rule.) Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Penalized DPD p � L α n ( β , σ ) + p λ ( | β j | ) j = 1 where p λ ( · ) is a penalty function (lasso, SCAD, MCP , ...). Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Penalized DPD p � L α n ( β , σ ) + p λ ( | β j | ) j = 1 where p λ ( · ) is a penalty function (lasso, SCAD, MCP , ...). As α ↓ 0, this becomes the (non-robust) non-concave penalized negative log-likelihood. Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Computational algorithm Starting from ˆ β , ˆ σ , Iteratively minimize the following: p � R α λ ( β ) = L α n ( β , ˆ σ ) + p λ ( | β j | ) , j = 1 S α ( σ ) = L α n (ˆ β , σ ) . Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Computational algorithm Starting from ˆ β , ˆ σ , Iteratively minimize the following: p � R α λ ( β ) = L α n ( β , ˆ σ ) + p λ ( | β j | ) , j = 1 S α ( σ ) = L α n (ˆ β , σ ) . Update β using a Concave-Convex Procedure (CCCP): p λ ( | β j | ) = ˜ J λ ( | β j | ) + λ | β j | ≃ ∇ ˜ J λ ( | β c j | ) β j + λ | β j | J ( · ) is differentiable and concave, β c is a current solution. where ˜ Update σ using gradient descent. Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Updating ˆ β and ˆ σ Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Updating ˆ β and ˆ σ � σ ( k ) � p � � � ( k + 1 ) = argmin ∇ ˜ β ( k ) ˆ L α J λ ( | ˆ β β , ˆ + | ) β j + λ | β j | ; n j β j = 1 Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Updating ˆ β and ˆ σ � σ ( k ) � p � � � ( k + 1 ) = argmin ∇ ˜ β ( k ) ˆ L α J λ ( | ˆ β β , ˆ + | ) β j + λ | β j | ; n j β j = 1 � n � � n � − 1 � i β ( k + 1 ) � 2 � � α σ 2 ( k + 1 ) = w ( k ) w ( k ) y i − x T ˆ − , i i ( 1 + α ) 3 / 2 i = 1 i = 1 � � i β ( k ) ) 2 − α ( y i − x T w ( k ) := exp . i σ 2 ( k ) Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Tuning parameter selection To choose λ , we use a robust High-dimensional BIC: σ 2 ) + log log( n ) log p � ˆ HBIC ( λ ) = log(ˆ β � 0 , (1) n and select the optimal λ ∗ that minimizes the HBIC over a pre-determined set of values Λ n : λ ∗ = argmin λ ∈ Λ n HBIC ( λ ) . Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Outline Motivation 1 Formulation 2 Influence functions 3 Theory 4 Simulations 5 Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Definition The Influence Function (IF) is a classical tool of measuring the asymptotic local robustness of any estimator (Hampel, 1968, 1974). Ghosh and Majumdar Robust Sparse Regression May 19, 2018
Recommend
More recommend
