OPTIMUM OPTIMUM ADAPTIVE ALGORITHMS ADAPTIVE ALGORITHMS for for - PowerPoint PPT Presentation

OPTIMUM OPTIMUM ADAPTIVE ALGORITHMS ADAPTIVE ALGORITHMS for for SYSTEM IDENTIFICATION SYSTEM IDENTIFICATION George V. Moustakides Dept. of Computer Engineering & Informatics University of Patras, GREECE

Definition of the problem w n x n r n y n + Physical System Physical System Model Model r' n - h 0 h 1 … h N- 1 h 0 h 1 … h N- 1 e n Common Model r' n =h 0 x n + h 1 x n- 1 +…+ h N- 1 x n-N+ 1 Transversal Filter:

Applications • Echo Cancellation • Filtering • Equalization • Control • Seismology • Array Processing

Echo Cancellation in Audio-Conferencing Room-2 Room-1 x n x n w n w n y n y n

x n We desire: e n ≅ w n We desire: e n ≅ w n Parametric Multipath Model w n y n e n -

Features of Adaptive Algorithms • Simplicity & Low Computational Complexity • Fast Convergence • Fast Tracking • Robustness in Finite Precision

Mathematical Setting We are given sequentially two sets of data x n : Input sequence y n : Measured sequence We would like to express y n as y n =h 0 x n + h 1 x n- 1 + …+ h N- 1 x n-N+ 1 +e n y n =h 0 x n + h 1 x n- 1 + …+ h N- 1 x n-N+ 1 +e n and identify the filter coefficients h i adaptively using algorithms of the form H n =H n- 1 + µF ( H n- 1 , Y n , X n ) H n =H n- 1 + µF ( H n- 1 , Y n , X n ) where H n = [ h 0 h 1 h 2 … h N- 1 ] T , X n = [ x n x n -1 … ] T , Y n = [ y n y n -1 … ] T

An Important Algorithmic Class = − T e y H X − n n n 1 n = + µ H H e Z − n n 1 n n X n = [ x n x n -1 … x n - N+ 1 ] T Z n : Regression Vector function of the input data x n x n -1 … Well known algorithms in the class: LMS: Z n = X n RLS: Z n = Q n -1 X n with Q n = (1- µ ) Q n- 1 + µX n X n T (exponentially windowed sample covariance matrix) FNTF: Z n = Q n -1 X n with Q n the covariance matrix of the x n sequence assuming that it is AR(M).

Generalization = − = − T e y H X , i 0,..., p 1 − − − n i , n i n 1 n i − p 1 + ∑ = µ H H e Z − n n 1 n i , n i , = i 0 X n = [ x n x n -1 … x n - N+ 1 ] T Z n,i : vector functions of the input data x n x n -1 … Well known algorithms in the class: T +…+ X n-p+1 X T SWRLS: Z n,i = Q n -1 X n-i with Q n =X n X n n-p+1 UDRLS: …

= − = − T e y H X , i 0,..., p 1 − − − n i , n i n 1 n i − p 1 + ∑ = µ H H e Z − n n 1 n i , n i , = i 0 By selecting different Regression Vectors Z n,i we obtain different adaptive algorithms. We need to compare them in order to select the optimum! For simplicity we assume EXACT MODEL!! For simplicity we assume EXACT MODEL!! For simplicity we assume EXACT MODEL!! = + T y H X w n * n n and w w n white noise and n white noise and w n white noise

Comparing the Prediction Error Classically the transient phase refers to stationary data. Let us assume that the noise w n has power 20db and we have two competing algorithms A1 and A2. We observe their prediction errors e n : 10 Prediction Error Power in db A1 5 A2 A2 0 -5 -10 -15 -20 -25 0 1000 2000 3000 4000 5000 Number of Iterations

Comparing the Excess Error e n = w n - ( H n- 1 -H * ) T X n e n = w n - ( H n- 1 -H * ) T X n ε n = ( H n- 1 -H * ) T X n ε n = ( H n- 1 -H * ) T X n Excess Error Excess Error Excess Error 10 A1 Excess Error Power in db 0 A2 A2 -10 -20 -30 -40 -50 -60 0 1000 2000 3000 4000 5000 Number of Iterations

A Fair Comparison Method To fairly compare adaptive algorithms • We first select the step size µ in each algorithm so that all algorithms under comparison have the same steady state excess error power (Excess Mean Square Error EMSE ). • The algorithm that converges faster is considered as the “best”. Can we select the step size µ analytically in order to achieve a predefined value for the EMSE at steady state? Can we characterize the speed of convergence analytically during the transient phase?

Analysis of the EMSE EMSE = E{[ ( H n- 1 -H * ) T X n ] 2 } = γ n + π n EMSE = E{[ ( H n- 1 -H * ) T X n ] 2 } = γ n + π n γ n : Starts from an O (1) value and tends exponentially fast to zero for stable algorithms (due to initial conditions). π n : Starts from an O ( µ 2 ) value and tends to an O ( µ ) value at steady state (due to the additive noise). µ << 1 µ << 1

Assumptions = − = − T e y H X , i 0,..., p 1 − − − n i , n i n 1 n i − p 1 + ∑ = µ H H e Z − n n 1 n i , n i , = i 0 • The input vector process { X n } is stationary. • The input vector process { X n } is stationary. • The Regression vector processes { Z n,i } are stationary. • The Regression vector processes { Z n,i } are stationary. • The noise process { w n } is zero mean white stationary • The noise process { w n } is zero mean white stationary and independent of the process { X n }. and independent of the process { X n }. RLS: Z n = Q n -1 X n with Q n = (1- µ ) Q n- 1 + µX n X n T

Exponential Convergence ( ) − γ 1 log n = µ λ + µ lim 2 ( A ) o ( ) min n →∞ n − − p 1 p 1 { } { } { } ∑ ∑ = = = T T T A E Z X E Z X E Z X − + n i , n i n i i , n n n = = i 0 i 0 − p 1 ∑ = Z Z + n n i i , = i 0 { } ( ) λ = λ ( A ) min Re min i i λ i : eigenvalues of the matrix A

Steady State EMSE { } π = µσ + µ 2 lim trace QP o ( ) n w → ∞ n { } σ = 2 2 E w w n { } = T Q E X X n n + = T AP PA R − − p 1 p 1 { } { } { } ∑ ∑ = = = T T T A E Z X E Z X E Z X − + n i , n i n i i , n n n = = i 0 i 0 − p 1 ∑ = Z Z + n n i i , = i 0 { } = T R E Z Z n n

Analytic Local Measure: Efficacy { } π ≈ µ σ 2 tra ce Q P w π µ ≈ { } σ 2 tra ce Q P w π λ 4 ( A ) ≈ µ λ ≈ m in R a te 2 ( A ) { } m in σ 2 2 tra ce Q P w λ ( A ) = E F F m in { } 2 tra ce Q P

An algorithm A 1 is better than an algorithm A 2 if An algorithm A 1 is better than an algorithm A 2 if EFF 1 >EFF 2 EFF 1 >EFF EFF 1 >EFF 2 2 Goal: Maximization of the Efficacy with respect to the Maximization of the Efficacy with respect to the Goal: Maximization of the Efficacy with respect to the regression vectors Z Z n,i . regression vectors n,i . regression vectors Z n,i .

Optimum Algorithm Theorem: The maximum value of the Efficacy is 1/ N. Theorem: Theorem: The maximum value of the Efficacy is 1/ N. The maximum is attained if and only if: The maximum is attained if and only if: − = α 1 Z Q X n n where Q= E { X n X nT }, α is any positive scalar. where Q= E { X n X nT }, α is any positive scalar. The algorithm is called: LMS LMS- -Newton Newton (LMS-N) The algorithm is called: LMS-Newton (LMS-N) Corollary: If Q= E { X n X nT }= σ x2 I then the optimum Corollary: algorithm in the class is the LMS.

Practically Optimum Algorithms RLS: RLS: Z n = Q n -1 X n Q n = (1- µ ) Q n- 1 + µX n X n T Under steady state Q n is a good approximation to Q= E { X n X n T } So RLS is expected to match the optimum performance. SWRLS: SWRLS: Z n,i = Q n - 1 X n-i Q n = X n X n T + X n- 1 X T n- 1 +…+ X n-p+ 1 X T n-p+ 1 If the window p is small then Q n does approximate well Q . If however p is large then due to the LLN the approximation can be good.

Simulation X n = [ x n x n- 1 … x n- 19 ], H * = [ h 0 h 1 ... h 19 ] x n = 0.9 x n- 1 + v n , Gaussian AR process w n : Gaussian additive white noise 20db N= 20 We select µ such that the steady state EMSE=35db At time n=5000 abrupt change H * = - [ h 0 h 1 ... h 19 ] LMS-N (Optimum) RLS SWRLS-30 (Window size p=30) SWRLS-100 (Window size p=100)

END END