Minimax risk of truncated series estimators over symmetric convex - PowerPoint PPT Presentation

Minimax risk of truncated series estimators over symmetric convex polytopes Adel Javanmard (Stanford University) with Li Zhang (Microsoft Research) July 4, 2012 Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 1 / 27

Motivation: function estimation Consider a continuous function f ✿ ❬ 0 ❀ 1 ❪ ✦ R . We have measurements y i ❂ f ✭ t i ✮ ✰ w i ❀ for 1 ✔ i ✔ n ✿ Estimate ❢ f ✭ t i ✮ ❣ n i ❂ 1 under linear inequality constraints: x ❂ ✭ f ✭ t 1 ✮ ❀ ✁ ✁ ✁ ❀ f ✭ t n ✮✮ Ax ✔ b ◮ Lipschitz constraint: ❥ x i ✰ 1 � x i ❥ ✔ L ❥ t i ✰ 1 � t i ❥ for 1 ✔ i ✔ n � 1 ✿ ◮ General convex constraints What is a good estimator? Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 2 / 27

Motivation: function estimation Consider a continuous function f ✿ ❬ 0 ❀ 1 ❪ ✦ R . We have measurements y i ❂ f ✭ t i ✮ ✰ w i ❀ for 1 ✔ i ✔ n ✿ Estimate ❢ f ✭ t i ✮ ❣ n i ❂ 1 under linear inequality constraints: x ❂ ✭ f ✭ t 1 ✮ ❀ ✁ ✁ ✁ ❀ f ✭ t n ✮✮ Ax ✔ b ◮ Lipschitz constraint: ❥ x i ✰ 1 � x i ❥ ✔ L ❥ t i ✰ 1 � t i ❥ for 1 ✔ i ✔ n � 1 ✿ ◮ General convex constraints What is a good estimator? Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 2 / 27

Motivation: function estimation Consider a continuous function f ✿ ❬ 0 ❀ 1 ❪ ✦ R . We have measurements y i ❂ f ✭ t i ✮ ✰ w i ❀ for 1 ✔ i ✔ n ✿ Estimate ❢ f ✭ t i ✮ ❣ n i ❂ 1 under linear inequality constraints: x ❂ ✭ f ✭ t 1 ✮ ❀ ✁ ✁ ✁ ❀ f ✭ t n ✮✮ Ax ✔ b ◮ Lipschitz constraint: ❥ x i ✰ 1 � x i ❥ ✔ L ❥ t i ✰ 1 � t i ❥ for 1 ✔ i ✔ n � 1 ✿ ◮ General convex constraints What is a good estimator? Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 2 / 27

Motivation: function estimation Consider a continuous function f ✿ ❬ 0 ❀ 1 ❪ ✦ R . We have measurements y i ❂ f ✭ t i ✮ ✰ w i ❀ for 1 ✔ i ✔ n ✿ Estimate ❢ f ✭ t i ✮ ❣ n i ❂ 1 under linear inequality constraints: x ❂ ✭ f ✭ t 1 ✮ ❀ ✁ ✁ ✁ ❀ f ✭ t n ✮✮ Ax ✔ b ◮ Lipschitz constraint: ❥ x i ✰ 1 � x i ❥ ✔ L ❥ t i ✰ 1 � t i ❥ for 1 ✔ i ✔ n � 1 ✿ ◮ General convex constraints What is a good estimator? Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 2 / 27

General problem y ❂ x ✰ w ❀ w ✘ N ✭ 0 ❀ ✛ 2 I n ✂ n ✮ ✿ x ✷ X ✒ R n ❀ For any estimator M ✿ R n ✦ R n , define x ✷ X E y ✘ x ✰ w ❦ x � M ✭ y ✮ ❦ 2 ✿ R ✭ M ❀ X ❀ ✛ ✮ ❂ max Minimax risk of a set R ✭ X ❀ ✛ ✮ ❂ min M R ✭ M ❀ X ❀ ✛ ✮ . Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 3 / 27

General problem y ❂ x ✰ w ❀ w ✘ N ✭ 0 ❀ ✛ 2 I n ✂ n ✮ ✿ x ✷ X ✒ R n ❀ For any estimator M ✿ R n ✦ R n , define x ✷ X E y ✘ x ✰ w ❦ x � M ✭ y ✮ ❦ 2 ✿ R ✭ M ❀ X ❀ ✛ ✮ ❂ max Minimax risk of a set R ✭ X ❀ ✛ ✮ ❂ min M R ✭ M ❀ X ❀ ✛ ✮ . Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 3 / 27

Classes of estimators ◮ Nonlinear estimators: The estimator M can be generally nonlinear. R ✭ X ❀ ✛ ✮ Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 4 / 27

Classes of estimators ◮ Nonlinear estimators: The estimator M can be generally nonlinear. R ✭ X ❀ ✛ ✮ ◮ Linear estimators: When M is linear, we denote the minimax risk by R L ✭ X ❀ ✛ ✮ Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 4 / 27

Classes of estimators ◮ Nonlinear estimators: The estimator M can be generally nonlinear. R ✭ X ❀ ✛ ✮ ◮ Linear estimators: When M is linear, we denote the minimax risk by R L ✭ X ❀ ✛ ✮ ◮ Truncated series estimators: Especial class of linear estimators given by orthogonal projections M ✭ y ✮ ❂ Py ✿ The minimax risk is denoted by R T ✭ X ❀ ✛ ✮ . Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 4 / 27

Classes of estimators ◮ Nonlinear estimators: The estimator M can be generally nonlinear. R ✭ X ❀ ✛ ✮ ◮ Linear estimators: When M is linear, we denote the minimax risk by R L ✭ X ❀ ✛ ✮ ◮ Truncated series estimators: Especial class of linear estimators given by orthogonal projections M ✭ y ✮ ❂ Py ✿ The minimax risk is denoted by R T ✭ X ❀ ✛ ✮ . R ✭ X ❀ ✛ ✮ ✔ R L ✭ X ❀ ✛ ✮ ✔ R T ✭ X ❀ ✛ ✮ . X ✒ Y ✮ R ✭ X ❀ ✛ ✮ ✔ R ✭ Y ❀ ✛ ✮ . Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 4 / 27

Challenges ◮ How to compute the minimax risk for arbitrary convex bodies? ◮ How to design the minimax optimal estimator? Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 5 / 27

Related work ◮ Minimax bounds have been developed for various families: ✎ Orthosymmetric and quadratically convex objects: Hypercubes, ellipsoids, ❵ p balls for p ✕ 2. [Donoho, Liu, MacGibbon’90] ✎ Class of Hölder balls, Sobolev balls, and Besov balls (continuity and energy conditions) [Tsybakov’09] ◮ Techniques for bounding the minimax risk [Donoho’90, Nemirovski’99, Yang, Barron’99] Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 6 / 27

Our main contributions ◮ A lower bound for minimax risk ✎ based on a geometric quantity of the set (approximation radius) ✎ depends on the volume of the set (intuitive and strong bounds) ◮ Optimality of truncated estimators over symmetric polytopes information theory tools ✦ geometrical understanding of minimax risks Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 7 / 27

Our main contributions ◮ A lower bound for minimax risk ✎ based on a geometric quantity of the set (approximation radius) ✎ depends on the volume of the set (intuitive and strong bounds) ◮ Optimality of truncated estimators over symmetric polytopes information theory tools ✦ geometrical understanding of minimax risks Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 7 / 27

Our main contributions ◮ A lower bound for minimax risk ✎ based on a geometric quantity of the set (approximation radius) ✎ depends on the volume of the set (intuitive and strong bounds) ◮ Optimality of truncated estimators over symmetric polytopes information theory tools ✦ geometrical understanding of minimax risks Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 7 / 27

Outline Main result 1 An application 2 Proof techniques 3 Further comments 4 Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 8 / 27

Main result Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 9 / 27

Notation For p ❃ 0 and m ❀ n ✕ 1, let ❋ m ❀ n ❂ ❢ X ✿ X ❂ ❢ x ✿ ❦ Ax ❦ p ✔ 1 ❣ ❀ for A ✷ R m ✂ n ❣ p ❋ m ❀ n : Family of symmetric polytopes defined by m hyperplanes ✶ Definition R T ✭ X ❀ ✛ ✮ ☞ m ❀ n ☞ ✭ X ✮ ❂ max R ✭ X ❀ ✛ ✮ ❀ ❂ max ☞ ✭ X ✮ ✿ p X ✷❋ m ❀ n ✛❃ 0 p Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 10 / 27

Notation For p ❃ 0 and m ❀ n ✕ 1, let ❋ m ❀ n ❂ ❢ X ✿ X ❂ ❢ x ✿ ❦ Ax ❦ p ✔ 1 ❣ ❀ for A ✷ R m ✂ n ❣ p ❋ m ❀ n : Family of symmetric polytopes defined by m hyperplanes ✶ Definition R T ✭ X ❀ ✛ ✮ ☞ m ❀ n ☞ ✭ X ✮ ❂ max R ✭ X ❀ ✛ ✮ ❀ ❂ max ☞ ✭ X ✮ ✿ p X ✷❋ m ❀ n ✛❃ 0 p Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 10 / 27

Notation For p ❃ 0 and m ❀ n ✕ 1, let ❋ m ❀ n ❂ ❢ X ✿ X ❂ ❢ x ✿ ❦ Ax ❦ p ✔ 1 ❣ ❀ for A ✷ R m ✂ n ❣ p ❋ m ❀ n : Family of symmetric polytopes defined by m hyperplanes ✶ Definition R T ✭ X ❀ ✛ ✮ ☞ m ❀ n ☞ ✭ X ✮ ❂ max R ✭ X ❀ ✛ ✮ ❀ ❂ max ☞ ✭ X ✮ ✿ p X ✷❋ m ❀ n ✛❃ 0 p Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 10 / 27

Optimality of truncated estimators Theorem (Javanmard, Zhang ’11) If n ❂ ✡✭ log m ✮ , for some universal constant C we have ☞ m ❀ n ✔ C log m ✿ ✶ Furthermore, ☞ m ❀ n ♣ ❂ ✡✭ log m ❂ log log m ✮ . ✶ [Recall: a ❂ ✡✭ b ✮ if a is bounded below by b (up to a constant factor) asymptotically] Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 11 / 27

An application Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 12 / 27

Function estimation Consider a univariate Lipschitz function f ✿ ❬ 0 ❀ 1 ❪ ✦ R . We have measurements y i ❂ f ✭ t i ✮ ✰ w i ❀ for 1 ✔ i ✔ n ✿ Goal: estimate ❢ f ✭ t i ✮ ❣ n i ❂ 1 from measurements ❢ y i ❣ n i ❂ 1 . Lipschitz condition (with constant L ): ❥ f ✭ t i ✰ 1 ✮ � f ✭ t i ✮ ❥ ✔ L ❥ t i ✰ 1 � t i ❥ ❀ for 1 ✔ i ✔ n � 1 ✿ Previous work shows near-optimality of truncated estimators for uniform sampling. [Nemirovski, Tsybakov’09] Javanmard, Zhang Minimax risk of truncated estimators July 4, 2012 13 / 27