Optimal and Adaptive Filtering Murat ney M.Uney@ed.ac.uk Institute - PowerPoint PPT Presentation

Optimal and Adaptive Filtering Murat Üney M.Uney@ed.ac.uk Institute for Digital Communications (IDCOM) 20/07/2015 Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 1 / 62

Table of Contents Optimal Filtering 1 Optimal filter design Application examples Optimal solution: Wiener-Hopf equations Example: Wiener equaliser Adaptive filtering 2 Introduction Recursive Least Squares Adaptation Least Mean Square Algorithm Applications Optimal signal detection 3 Summary 4 Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 2 / 62

Optimal Filtering Optimal filter design Optimal filter design Observation Estimation sequence Linear time invariant system Figure 1: Optimal filtering scenario. y ( n ) : Observation related to a signal of interest x ( n ) . h ( n ) : The impulse response of an LTI estimator. Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 3 / 62

Optimal Filtering Optimal filter design Optimal filter design Observation Estimation sequence Linear time invariant system Figure 1: Optimal filtering scenario. y ( n ) : Observation related to a signal of interest x ( n ) . h ( n ) : The impulse response of an LTI estimator. Find h ( n ) with the best error performance: e ( n ) = x ( n ) − ˆ x ( n ) = x ( n ) − h ( n ) ∗ y ( n ) The error performance is measured by the mean squared error (MSE) � ( e ( n )) 2 � ξ = E . Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 3 / 62

Optimal Filtering Optimal filter design Optimal filter design Observation Estimation sequence Linear time invariant system Figure 2: Optimal filtering scenario. The MSE is a function of h ( n ) , i.e., h = [ · · · , h ( − 2 ) , h ( − 1 ) , h ( 0 ) , h ( 1 ) , h ( 2 ) , · · · ] � ( e ( n )) 2 � � ( x ( n ) − h ( n ) ∗ y ( n )) 2 � ξ ( h ) = E = E . Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 4 / 62

Optimal Filtering Optimal filter design Optimal filter design Observation Estimation sequence Linear time invariant system Figure 2: Optimal filtering scenario. The MSE is a function of h ( n ) , i.e., h = [ · · · , h ( − 2 ) , h ( − 1 ) , h ( 0 ) , h ( 1 ) , h ( 2 ) , · · · ] � ( e ( n )) 2 � � ( x ( n ) − h ( n ) ∗ y ( n )) 2 � ξ ( h ) = E = E . Thus, optimal filtering problem is h opt = arg min h ξ ( h ) Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 4 / 62

Optimal Filtering Application examples Application examples 1) Prediction, interpolation and smoothing of signals Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 5 / 62

Optimal Filtering Application examples Application examples 1) Prediction, interpolation and smoothing of signals (a) d = 1 (b) d = − 1 (c) d = − 1 / 2 ◮ Linear predictive coding (LPC) in speech processing. Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 6 / 62

Optimal Filtering Application examples Application examples 2) System identification Figure 3: System identification using a training sequence t ( n ) from an ergodic and stationary ensemble. ◮ Echo cancellation in full duplex data transmission. Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 7 / 62

Optimal Filtering Application examples Application examples 3) Inverse System identification Figure 4: Inverse system identification using x ( n ) as a training sequence. ◮ Channel equalisation in digital communication systems. Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 8 / 62

Optimal Filtering Optimal solution: Wiener-Hopf equations Optimal solution: Normal equations � ( e ( n )) 2 � Consider the MSE ξ ( h ) = E The optimal filter satisfies ∇ ξ ( h ) | h opt = 0 . Equivalently, for all j = . . . , − 2 , − 1 , 0 , 1 , 2 , . . . � � ∂ξ 2 e ( n ) ∂ e ( n ) ∂ h ( j ) = E ∂ h ( j ) � � � x ( n ) − � ∞ � 2 e ( n ) ∂ i = −∞ h ( i ) y ( n − i ) = E ∂ h ( j ) � � 2 e ( n ) ∂ ( − h ( j ) y ( n − j )) = E ∂ h ( j ) = − 2 E [ e ( n ) y ( n − j )] Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 9 / 62

Optimal Filtering Optimal solution: Wiener-Hopf equations Optimal solution: Normal equations � ( e ( n )) 2 � Consider the MSE ξ ( h ) = E The optimal filter satisfies ∇ ξ ( h ) | h opt = 0 . Equivalently, for all j = . . . , − 2 , − 1 , 0 , 1 , 2 , . . . � � ∂ξ 2 e ( n ) ∂ e ( n ) ∂ h ( j ) = E ∂ h ( j ) � � � x ( n ) − � ∞ � 2 e ( n ) ∂ i = −∞ h ( i ) y ( n − i ) = E ∂ h ( j ) � � 2 e ( n ) ∂ ( − h ( j ) y ( n − j )) = E ∂ h ( j ) = − 2 E [ e ( n ) y ( n − j )] Hence, the optimal filter solves the “normal equations” E [ e ( n ) y ( n − j )] = 0 , j = . . . , − 2 , − 1 , 0 , 1 , 2 , . . . Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 9 / 62

Optimal Filtering Optimal solution: Wiener-Hopf equations Optimal solution: Wiener-Hopf equations The error of h opt is orthogonal to its observations, i.e., for all j ∈ Z E [ e opt ( n ) y ( n − j )] = 0 which is known as “the principle of orthogonality”. Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 10 / 62

Optimal Filtering Optimal solution: Wiener-Hopf equations Optimal solution: Wiener-Hopf equations The error of h opt is orthogonal to its observations, i.e., for all j ∈ Z E [ e opt ( n ) y ( n − j )] = 0 which is known as “the principle of orthogonality”. Furthermore, �� � � ∞ � E [ e opt ( n ) y ( n − j )] = E x ( n ) − h opt ( i ) y ( n − i ) y ( n − j ) i = −∞ ∞ � h opt ( i ) E [ y ( n − i ) y ( n − j )] = 0 = E [ x ( n ) y ( n − j )] − i = −∞ Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 10 / 62

Optimal Filtering Optimal solution: Wiener-Hopf equations Optimal solution: Wiener-Hopf equations The error of h opt is orthogonal to its observations, i.e., for all j ∈ Z E [ e opt ( n ) y ( n − j )] = 0 which is known as “the principle of orthogonality”. Furthermore, �� � � ∞ � E [ e opt ( n ) y ( n − j )] = E x ( n ) − h opt ( i ) y ( n − i ) y ( n − j ) i = −∞ ∞ � h opt ( i ) E [ y ( n − i ) y ( n − j )] = 0 = E [ x ( n ) y ( n − j )] − i = −∞ Result (Wiener-Hopf equations) ∞ � h opt ( i ) r yy ( i − j ) = r xy ( j ) i = −∞ Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 10 / 62

Optimal Filtering Optimal solution: Wiener-Hopf equations The Wiener filter Wiener-Hopf equations can be solved indirectly, in the complex spectral domain: h opt ( n ) ∗ r yy ( n ) = r xy ( n ) ↔ H opt ( z ) P yy ( z ) = P xy ( z ) Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 11 / 62

Optimal Filtering Optimal solution: Wiener-Hopf equations The Wiener filter Wiener-Hopf equations can be solved indirectly, in the complex spectral domain: h opt ( n ) ∗ r yy ( n ) = r xy ( n ) ↔ H opt ( z ) P yy ( z ) = P xy ( z ) Result (The Wiener filter) H opt ( z ) = P xy ( z ) P yy ( z ) Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 11 / 62

Optimal Filtering Optimal solution: Wiener-Hopf equations The Wiener filter Wiener-Hopf equations can be solved indirectly, in the complex spectral domain: h opt ( n ) ∗ r yy ( n ) = r xy ( n ) ↔ H opt ( z ) P yy ( z ) = P xy ( z ) Result (The Wiener filter) H opt ( z ) = P xy ( z ) P yy ( z ) The optimal filter has an infinite impulse response (IIR), and, is non-causal, in general. Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 11 / 62

Optimal Filtering Optimal solution: Wiener-Hopf equations Causal Wiener filter We project the unconstrained solution H opt ( z ) onto the set of causal and stable IIR filters by a two step procedure: First, factorise P yy ( z ) into causal (right sided) Q yy ( z ) , and anti-causal (left sided) parts Q ∗ yy ( 1 / z ∗ ) , i.e., P yy ( z ) = σ 2 yy ( 1 / z ∗ ) . y Q yy ( z ) Q ∗ Select the causal (right sided) part of P xy ( z ) / Q ∗ yy ( 1 / z ∗ ) . Result (Causal Wiener filter) � � 1 P xy ( z ) H + opt ( z ) = σ 2 yy ( 1 / z ∗ ) Q ∗ y Q yy ( z ) + Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 12 / 62

Optimal Filtering Optimal solution: Wiener-Hopf equations FIR Wiener-Hopf equations received sequence { y ( n ) } z -1 z -1 z -1 . . . h h h 0 1 N -1 Σ output { } x ( n ) Figure 5: A finite impulse response (FIR) estimator. Wiener-Hopf equations for the FIR optimal filter of N taps: Result (FIR Wiener-Hopf equations) � N − 1 i = 0 h opt ( i ) r yy ( i − j ) = r xy ( j ) , for j = 0 , 1 , ..., N − 1 . Murat Üney (IDCOM) Optimal and Adaptive Filtering 20/07/2015 13 / 62