Uniform bounds for positive random functionals with application to - PowerPoint PPT Presentation

Uniform bounds for positive random functionals with application to density estimation Oleg Lepski Laboratoire d’Analyse, Topologie et Probabilit´ es Universit´ e de Provence Congr` es SMAI 2011, May 23 - May 27, Guidel Oleg Lepski Upper functions. Density estimation

Outline 1 Upper functions. General case 2 Probabilistic study of statistical objects Upper functions. Special cases Comparison with asymptotical results 3 Sup-norm oracle inequality in density estimation Selection from the family of kernel estimators Adaptation over anisotropic H¨ older classes Oleg Lepski Upper functions. Density estimation

Part I Upper functions. General case Oleg Lepski Upper functions. Density estimation

Introduction Problem formulation Let (Ω , A , P) be a probability space, S be a linear space and Θ be a given set. Let χ n : Θ × Ω → S , n ∈ N ∗ , be a given sequence of A -measurable maps and P (n) be the corresponding sequence of f probability laws, parameterized by f ∈ F . Let Ψ : S → R + be a given sub-additive functional. Goal: find non-random positive function on Θ which would be uniform upper bound for Ψ( χ n ,θ ) in the sense � � � � � � P (n) sup θ ∈ Θ Ψ χ n ,θ − U n (y , θ ) ≥ 0 ≤ P n (y , f); f � �� q � � � E (n) sup θ ∈ Θ Ψ χ n ,θ − U n (y , θ ) + ≤ E n (y , f , q) . f The quantities P n (y , f) and E n (y , f , q) should possess several properties discussed later. In particular for any fixed y > y q E n (y , f , q) → 0 , n → ∞ uniformly w.r.t. f . Oleg Lepski Upper functions. Density estimation

Statistical models. Adaptive estimation. Regression model Y i = f(z i ) + ξ i , i = 1 , n ξ i , i ∈ N ∗ are i.i.d.: E ξ 1 = 0 , or med( ξ 1 ) = 0 ; The design points z i ∈ R d , i = 1 , n are supposed to be either fixed real vectors or i.i.d. random vectors. �� � � �� We observe X (n) = Y 1 , z 1 , . . . , Y n , z n . Density model n � X (n) x = (x 1 , . . . , x d ) ∈ R d ∼ p n (x) = f(x i ) , i=1 � � X (n) = , n ∈ N ∗ , where X i ∈ R d , i ∈ N ∗ , are i.i.d. X 1 , . . . , X n random vectors having the density f . Goal: to estimate the function f from the observation X (n) . Oleg Lepski Upper functions. Density estimation

Statistical models. Adaptive estimation. Regression model Y i = f(z i ) + ξ i , i = 1 , n ξ i , i ∈ N ∗ are i.i.d.: E ξ 1 = 0 , or med( ξ 1 ) = 0 ; The design points z i ∈ R d , i = 1 , n are supposed to be either fixed real vectors or i.i.d. random vectors. �� � � �� We observe X (n) = Y 1 , z 1 , . . . , Y n , z n . Density model n � X (n) x = (x 1 , . . . , x d ) ∈ R d ∼ p n (x) = f(x i ) , i=1 � � X (n) = , n ∈ N ∗ , where X i ∈ R d , i ∈ N ∗ , are i.i.d. X 1 , . . . , X n random vectors having the density f . Goal: to estimate the function f from the observation X (n) . Oleg Lepski Upper functions. Density estimation

General case Problem formulation Let (Ω , A , P) be a probability space, S be a linear space and Θ be a given set. Let χ n : Θ × Ω → S , n ∈ N ∗ , be a given sequence of A -measurable maps and P (n) be the corresponding sequence of f probability laws. Let Ψ : S → R + be a given sub-additive functional. Goal: find non-random positive function on θ which would be uniform upper bound for Ψ( χ n ,θ ) in the sense � � � � � � P (n) sup θ ∈ Θ Ψ χ n ,θ − U n (y , θ ) ≥ 0 ≤ P n (y , f); f � �� q � � � E (n) sup θ ∈ Θ Ψ χ n ,θ − U n (y , θ ) + ≤ E n (y , f , q) . f To realize this program we impose the Bernstein-type assumption � � on the tail probability for Ψ χ n ,θ for any given θ ∈ Θ and � � Ψ χ n ,θ 1 − χ n ,θ 2 , θ 1 , θ 2 ∈ Θ . Oleg Lepski Upper functions. Density estimation

General case Assumptions. Bound for a given trajectory. Furthermore P = P ( n ) , E = E ( n ) and χ θ = χ n , θ . f f Assumption (1) 1 There exist A , B : Θ → R + such that ∀ θ ∈ Θ and ∀ z > 0 � � z 2 P { Ψ( χ θ ) ≥ z } ≤ G A 2 ( θ ) + B( θ )z 2 There exist a , b : Θ × Θ → R + s.t. ∀ θ 1 , θ 2 ∈ Θ , ∀ z > 0 � � z 2 � � Ψ( χ θ 1 − χ θ 2 ) ≥ z ≤ G P a 2 ( θ 1 , θ 2 ) + b( θ 1 , θ 2 )z G(x) = c exp {− x } , c > 0 . Oleg Lepski Upper functions. Density estimation

General case Assumptions. Bound for a given trajectory. Furthermore P = P ( n ) , E = E ( n ) and χ θ = χ n , θ . f f Assumption (1) 1 There exist A , B : Θ → R + such that ∀ θ ∈ Θ and ∀ z > 0 � � z 2 P { Ψ( χ θ ) ≥ z } ≤ G A 2 ( θ ) + B( θ )z 2 There exist a , b : Θ × Θ → R + s.t. ∀ θ 1 , θ 2 ∈ Θ , ∀ z > 0 � � z 2 � � Ψ( χ θ 1 − χ θ 2 ) ≥ z ≤ G P a 2 ( θ 1 , θ 2 ) + b( θ 1 , θ 2 )z G(x) = c exp {− x } , c > 0 . Oleg Lepski Upper functions. Density estimation

General case Assumptions. Bound for a given trajectory. G(x) = c exp {− x } , c > 0 . Assumption (1) 1 There exist A , B : Θ → R + such that ∀ θ ∈ Θ and ∀ z > 0 � � z 2 P { Ψ( χ θ ) ≥ z } ≤ G A 2 ( θ ) + B( θ )z 2 There exist a , b : Θ × Θ → R + s.t. ∀ θ 1 , θ 2 ∈ Θ , ∀ z > 0 � � z 2 � � Ψ( χ θ 1 − χ θ 2 ) ≥ z ≤ G P a 2 ( θ 1 , θ 2 ) + b( θ 1 , θ 2 )z Assumption (2) 1 The mappings a and b are semi-metrics on Θ and χ • is stochastically continuous in the topology generated by a ∨ b . 2 Θ is totally bounded with respect to the semi-metric a ∨ b and A θ := sup θ ∈ Θ A( θ ) < ∞ , B θ := sup θ ∈ Θ B( θ ) < ∞ . Oleg Lepski Upper functions. Density estimation

General case Assumptions. Bound for a given trajectory. Examples. Let X be X -valued random vector defined on ( Ω , A , P) and let X i , i = 1 , n be independent copies of X . Let W be a given set of functions w : X → R . � � � D w ( · ) � , Example: Ψ( χ θ ) = θ = w , Θ = W n � � � � � D w = w X i − E w(X) . i=1 If W is a subset of the set of bounded functions then Assumption 1 follows from Bernstein inequality. Here � � � 2 , � � � w(x) � ; A(w) = E w(X) B(w) = sup x ∈X � � � � � � � � a w 1 , w 2 = A w 1 − w 2 , b w 1 , w 2 = B w 1 − w 2 . We remark that Assumption 2 (1) is also fulfilled. Oleg Lepski Upper functions. Density estimation

General case Assumptions. Bound for a given trajectory. Examples. � � � χ θ � , χ θ is zero mean gaussian function Example: Ψ( χ θ ) = The Assumptions 1 and 2 (1) are obviously fulfilled with B = b ≡ 0 � � � � 2 , a( θ 1 , θ 2 ) = � � 2 A( θ ) = χ θ χ θ 1 − χ θ 2 E E Oleg Lepski Upper functions. Density estimation

General case: bounds under Assumptions 1 and 2. Assumption (1) 1 There exist A , B : Θ → R + such that ∀ θ ∈ Θ and ∀ z > 0 � � z 2 P { Ψ( χ θ ) ≥ z } ≤ G A 2 ( θ ) + B( θ )z 2 There exist a , b : Θ × Θ → R + s.t. ∀ θ 1 , θ 2 ∈ Θ , ∀ z > 0 � � z 2 � � Ψ( χ θ 1 − χ θ 2 ) ≥ z ≤ G P a 2 ( θ 1 , θ 2 ) + b( θ 1 , θ 2 )z Assumption (2) 1 The mappings a and b are semi-metrics on Θ and χ • is stochastically continuous in the topology generated by a ∨ b . 2 Θ is totally bounded with respect to the semi-metric a ∨ b and A θ := sup θ ∈ Θ A( θ ) < ∞ , B θ := sup θ ∈ Θ B( θ ) < ∞ . Oleg Lepski Upper functions. Density estimation

General case: bounds under Assumptions 1 and 2. The most important elements of our construction are: � � Θ A (t) = θ ∈ Θ : A( θ ) ≤ t , t > 0; � � Θ B (t) = θ ∈ Θ : B( θ ) ≤ t , t > 0; For any x > 0 , any � Θ ⊆ Θ and any s ∈ S � � � � e (a) x , � = sup δ> 0 δ − 2 E � x(48 δ ) − 1 s( δ ) Θ s Θ , a � � � � e (b) x , � = sup δ> 0 δ − 1 E � x(48 δ ) − 1 s( δ ) Θ s Θ , b Θ , d ( ν ) , ν > 0 , - entropy of � Θ measured in semi-metric d ; E � � s : R → R + \ { 0 } : � ∞ � 2 k / 2 � � S = k=0 s ≤ 1 . Oleg Lepski Upper functions. Density estimation

General case: bounds under Assumptions 1 and 2. Introduced quantities � � Θ A (t) = θ ∈ Θ : A( θ ) ≤ t , t > 0; � � Θ B (t) = θ ∈ Θ : B( θ ) ≤ t , t > 0; � � � � e (a) x , � = sup δ> 0 δ − 2 E � x(48 δ ) − 1 s( δ ) Θ s Θ , a � � � � e (b) x , � = sup δ> 0 δ − 1 E � x(48 δ ) − 1 s( δ ) Θ s Θ , b allow us to define for any u , v ≥ 1 and any � s = (s 1 , s 2 ) � �� � �� � � s (u , v) = e (a) + e (b) � E Au , Θ A Au Bv , Θ B Bv � s 1 s 2 A = inf θ ∈ Θ A( θ ) > 0; B = inf θ ∈ Θ B( θ ) > 0 . � 1 + [ln x] 2 � − 1 , x ≥ 0 Example : s 1 = s 2 = (6 /π 2 ) Oleg Lepski Upper functions. Density estimation