Adaptive Signal Recovery by Convex Optimization Dmitrii Ostrovskii - PowerPoint PPT Presentation

Adaptive Signal Recovery by Convex Optimization Dmitrii Ostrovskii CWI, Amsterdam 19 April 2018

Signal denoising problem Recover complex signal x = ( x τ ) , τ = − n , ..., n , from noisy observations y τ = x τ + σξ τ , τ = − n , ..., n , where ξ τ are i.i.d. standard complex Gaussian random variables. 2 5 1.5 1 0.5 0 0 -0.5 -1 -1.5 -2 -5 0 20 40 60 80 100 0 20 40 60 80 100 Signal Observations • Assumption: signal has unknown shift-invariant structure . Dmitrii Ostrovskii Adaptive Signal Recovery by Convex Optimization 1 / 31

Preliminaries • Finite-dimensional spaces and norms: C n ( Z ) = { x = ( x τ ) τ ∈ Z : x τ = 0 whenever | τ | > n } ; ℓ p -norms restricted to C n ( Z ): �� | τ |≤ n | x τ | p � 1 p ; � x � p = Scaled ℓ p -norms: 1 � x � n , p = (2 n + 1) 1 / p � x � p . • Loss: ℓ ( � x , x ) = | � x 0 − x 0 | – pointwise loss; ℓ ( � x , x ) = � � x − x � n , 2 – ℓ 2 -loss. • Risk: 1 x , x ) 2 ] 2 ; R ( � x , x ) = [ E ℓ ( � R δ ( � x , x ) = min { r ≥ 0 : ℓ ( � x , x ) ≤ r with probability ≥ 1 − δ } . Dmitrii Ostrovskii Adaptive Signal Recovery by Convex Optimization 2 / 31

Adaptive estimation: disclaimer Classical approach Given a set X containing x , look for a near-minimax , over X , estimator x o is linear in y (e.g. for pointwise loss)*. x o . One can often assume that � � x o becomes an unavailable linear oracle . Mimic it! If X is unknown , � Oracle approach x o with small risk R ( � x o , x ), Knowing that there exists a linear oracle � construct an adaptive estimator � x = � x ( y ) satisfying an oracle inequality : x o , x ) + Rem , x o , x ) . R ( � x , x ) ≤ P · R ( � Rem ≪ R ( � x o can change but P and Rem must be uniformly bounded over ( � x o , x ). x , � • P = “price of adaptation”. Inequalities with P = 1 are called sharp *. *[Ibragimov and Khasminskii, 1984; Donoho et al., 1990], *[Tsybakov, 2008] Dmitrii Ostrovskii Adaptive Signal Recovery by Convex Optimization 3 / 31

Classical example: unknown smoothness Let x be a regularly sampled function: x t = f ( t / N ) , t = − N , ..., N , where f : [ − 1 , 1] → R has weak derivative D s f of order s ≥ 1 on [ − 1 , 1], and belongs to a Sobolev ( q = 2) or H¨ older ( q = ∞ ) smoothness class:* F s , L = { f ( · ) : � D s f � L q ≤ L } . • Linear oracle: kernel estimator with properly chosen bandwidth h : � τ � � 1 � f ( t / N ) = K y t − τ , | t | ≤ N − hN . 2 hN + 1 hN | τ |≤ hN • Adaptive bandwidth selection*: Lepski’s method, Stein’s method, ... *[Adams and Fournier, 2003; Brown et al., 1996; Watson, 1964; Nadaraya, 1964; Tsybakov, 2008; Johnstone, 2011], *[Lepski, 1991; Lepski et al., 1997, 2015; Goldenshluger et al., 2011] Dmitrii Ostrovskii Adaptive Signal Recovery by Convex Optimization 4 / 31

Recoverable signals • We consider convolution-type (or time-invariant) estimators � � x t = [ ϕ ∗ y ] t := ϕ τ y t − τ , τ ∈ Z where ∗ is discrete convolution , and ϕ ∈ C n ( Z ) is called a filter . Definition* A signal x is ( n , ρ )- recoverable if there exists φ o ∈ C n ( Z ) which satisfies � E | x t − [ φ o ∗ y ] t | 2 � 1 / 2 σρ √ 2 n + 1 , ≤ | t | ≤ 3 n . 3 n , 2 ] 1 / 2 ≤ • Consequence: small ℓ 2 -risk: [ E � x − φ o ∗ y � 2 σρ √ 2 n +1 . -4n -3n -2n -n 0 n 2n 3n 4n *[Juditsky and Nemirovski, 2009; Nemirovski, 1991; Goldenshluger and Nemirovski, 1997] Dmitrii Ostrovskii Adaptive Signal Recovery by Convex Optimization 5 / 31

Adaptive signal recovery: main questions Goal Assuming that x is ( n , ρ )-recoverable, construct an adaptive filter σρ ϕ = � � ϕ ( y ) such that the pointwise or ℓ 2 -risk of � x = � ϕ ∗ y is close to √ 2 n +1 . Main questions: • Can we adapt to the oracle? Yes , but we must pay the price polynomial in ρ ; • Can � ϕ be efficiently computed? Yes , by solving a well-structured convex optimization problem. • Do recoverable signals with small ρ exist? Yes: when the signal belongs to shift-invariant subspace S ⊂ C ( Z ), dim( S ) = s , we have “nice” bounds on ρ = ρ ( s ). Dmitrii Ostrovskii Adaptive Signal Recovery by Convex Optimization 6 / 31

Adaptive estimators and their analysis

Main idea • “Bias-variance decomposition” x t − [ φ o ∗ y ] t = x t − [ φ o ∗ x ] t + σ [ φ o ∗ ξ ] t . � �� � � �� � � �� � total error bias stochastic error • ( n , ρ )-recoverability implies σρ ρ | x t − [ φ o ∗ x ] t | ≤ � φ o � 2 ≤ √ 2 n + 1 , | t | ≤ 3 n , and √ 2 n + 1 . • Unitary Discrete Fourier transform operator F n : C n ( Z ) → C n ( Z ). Look at the Fourier transforms Estimate x via � x = � ϕ ∗ y , where � ϕ = � ϕ ( y ) ∈ C 2 n ( Z ) minimizes the Fourier-domain residual �F 2 n [ y − ϕ ∗ y ] � p while keeping �F 2 n [ ϕ ] � 1 small. Dmitrii Ostrovskii Adaptive Signal Recovery by Convex Optimization 7 / 31

Motivation: new oracle Oracle with small ℓ 1 -norm of DFT* If x is ( n , ρ )-recoverable, then there exists a ϕ o ∈ C 2 n ( Z ) s.t. for R = 2 ρ 2 , C σ R R | x t − [ ϕ o ∗ x ] t | ≤ �F 2 n [ ϕ o ] � 1 ≤ √ 4 n + 1 , | t | ≤ 2 n , √ 4 n + 1 . Proof. 1 o . Consider ϕ o = φ o ∗ φ o ∈ C 2 n ( Z ). On one hand, for | t | ≤ 2 n , | x t − [ ϕ o ∗ x ] t | = | x t − [ φ o ∗ x ] t | + | [ φ o ∗ ( x − φ o ∗ x )] t | | τ |≤ 3 n | x τ − [ φ o ∗ x ] τ | ≤ σρ (1 + ρ ) ≤ (1 + � φ o � 1 ) max √ 2 n + 1 . 2 o . On the other hand, we get √ 2 ρ 2 4 n + 1 �F 2 n [ ϕ o ] � 1 = √ 4 n + 1 �F 2 n [ φ o ] � 2 4 n + 1 �F n [ φ o ] � 2 2 = 2 ≤ √ 4 n + 1 . � *[Juditsky and Nemirovski, 2009] Dmitrii Ostrovskii Adaptive Signal Recovery by Convex Optimization 8 / 31

Uniform-fit estimators • Constrained uniform-fit estimator *: � � R ϕ ∈ Argmin � �F n [ y − ϕ ∗ y ] � ∞ : �F n [ ϕ ] � 1 ≤ √ 2 n + 1 . (CUF) ϕ ∈ C n ( Z ) • Penalized estimator : for some λ ≥ 0, � � √ ϕ ∈ Argmin �F n [ y − ϕ ∗ y ] � ∞ + σλ 2 n + 1 �F n [ ϕ ] � 1 . (PUF) � ϕ ∈ C n ( Z ) Pointwise upper bound for uniform-fit estimators 2 ⌉ , ρ )-recoverable. Let R = 2 ρ 2 for the constrained estimator, Let x be ( ⌈ n � and λ = 2 log[(2 n + 1) /δ ] for the penalized one, then w.p. ≥ 1 − δ , ϕ ∗ y ] 0 | ≤ C σρ 4 � log[(2 n + 1) /δ ] √ 2 n + 1 | x 0 − [ � . High price of adaptation: O ( ρ 3 √ log n ) . *[Juditsky and Nemirovski, 2009] Dmitrii Ostrovskii Adaptive Signal Recovery by Convex Optimization 9 / 31

Analysis of uniform-fit estimators Let � ϕ be an optimal solution to (CUF) with R = R , and let � Θ n ( ζ ) = �F n [ ζ ] � ∞ = O ( log n ) w.h.p. 1 o . Already in the first step, we see why the new oracle is useful: | [ x − � ϕ ∗ y ] 0 | ≤ σ | [ � ϕ ∗ ζ ] 0 | + | [ x − � ϕ ∗ x ] 0 | ≤ σ �F n [ � ϕ ] � 1 �F n [ ζ ] � ∞ + | [ x − � ϕ ∗ x ] 0 | [Young’s ineq.] ≤ σ Θ n ( ζ ) R √ 2 n + 1 + | [ x − � ϕ ∗ x ] 0 | . [Feasibility of � ϕ ] 2 o . To control | [ x − � ϕ ∗ x ] 0 | , we can add & subtract convolution with ϕ o : ϕ ∗ x ] 0 | ≤ | [ ϕ o ∗ ( x − � ϕ ) ∗ ( x − ϕ o ∗ x )] 0 | | x 0 − [ � ϕ ∗ x )] 0 | + | [(1 − � ϕ � 1 ) � [ x − ϕ o ∗ x ] � ∞ ≤ �F n [ ϕ o ] � 1 �F n [ x − � ϕ ∗ x ] � ∞ + (1 + � � R ϕ ∗ x ] � ∞ + CR (1 + R ) √ 2 n + 1 �F n [ x − � √ 2 n + 1 . ≤ Dmitrii Ostrovskii Adaptive Signal Recovery by Convex Optimization 10 / 31

Analysis of uniform-fit estimators, cont. 3 o . It remains to control �F n [ x − � ϕ ∗ x ] � ∞ which can be done as follows: �F n [ x − � ϕ ∗ x ] � ∞ ≤ �F n [ y − � ϕ ∗ y ] � ∞ + σ �F n [ ζ − � ϕ ∗ ζ ] � ∞ ≤ �F n [ y − � ϕ ∗ y ] � ∞ + σ (1 + � � ϕ � 1 )Θ n ( ζ ) ≤ �F n [ y − ϕ o ∗ y ] � ∞ + σ (1 + � � ϕ � 1 )Θ n ( ζ ) [Feas. of ϕ o ] ≤ �F n [ x − ϕ o ∗ x ] � ∞ + 2 σ (1 + R )Θ n ( ζ ) . 4 o . Finally, note that �F n [ x − ϕ o ∗ x ] � ∞ ≤ �F n [ x − ϕ o ∗ x ] � 2 = � [ x − ϕ o ∗ x ] � 2 [Parseval’s identity] √ 2 n + 1 � x − ϕ o ∗ x � ∞ ≤ σ CR . ≤ Collecting the above, we obtain a bound dominated by σ CR (1+ R )Θ n ( ζ ) . � √ 2 n +1 Dmitrii Ostrovskii Adaptive Signal Recovery by Convex Optimization 11 / 31

Limit of performance Proposition: pointwise lower bound For any integer n ≥ 2, α < 1 / 4, and ρ satisfying 1 ≤ ρ ≤ n α , one can point out a family of signals X n ,ρ ∈ C 2 n ( Z ) such that • any signal in X n ,ρ is ( n , ρ )-recoverable; • for any estimate � x 0 of x 0 from observations y ∈ C 2 n ( Z ), one can find x ∈ X n ,ρ satisfying � � x 0 | ≥ c σρ 2 � (1 − 4 α ) log n P | x 0 − � √ 2 n + 1 ≥ 1 / 8 . Conclusion: there is a gap ρ 2 between upper and lower bounds. • To bridge it (and encompass ℓ 2 -loss), we introduce new estimators. Dmitrii Ostrovskii Adaptive Signal Recovery by Convex Optimization 12 / 31