Fast, Accurate, and Robust Pitch Estimation NordicSMC Winter School - PowerPoint PPT Presentation

Fast, Accurate, and Robust Pitch Estimation NordicSMC Winter School 2019 March 7, 2019 Jesper Kjær Nielsen jkn@create.aau.dk Audio Analysis Lab, CREATE Aalborg University, Denmark Website: http://audio.create.aau.dk YouTube: http://tinyurl.com/yd8mo55z

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Motivation 0.3 0.2 0.1 0 -0.1 -0.2 -0.3 0 0.005 0.01 0.015 0.02 time [s] Periodic signals A periodic signal repeats itself after some period τ or, equivalently, with some frequency ω 0 . ◮ We refer to ω 0 as either the pitch (perceptual) or the fundamental frequency (physical). ◮ How do we estimate this value from possibly noisy and non-stationary data? 1 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Motivation Some examples of periodic signals and applications: ◮ Voiced speech and singing - Are people singing on-key? - Diagnosis of the Parkinson’s disease ◮ Many musical instruments (e.g., guitar, violin, flute, trumpet, piano) - Tuning of instruments - Music transcription ◮ Electrocardiographic (ECG) signals - Measure your heart rate or heart rate variability - Heart defect diagnosis ◮ Rotating machines - Vibration analysis - Rotation speed 2 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Motivation Example: RPM estimation from tachometer signal SNR: 40 dB 1 0.8 0.6 voltage [V] 0.4 0.2 0 -0.2 0 0.1 0.2 0.3 0.4 0.5 0.6 time [s] 3 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Motivation Example: RPM estimation from tachometer signal Figure courtesy of A. Brandt, Noise and vibration analysis: signal analysis and experimental procedures. John Wiley & Sons, 2011. 4 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Motivation Example: RPM estimation from tachometer signal SNR: 0 dB 2 1.5 1 0.5 time [s] 0 -0.5 -1 -1.5 0 0.1 0.2 0.3 0.4 0.5 0.6 time [s] 5 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Outline Correlation-based Methods Nonlinear Least Squares Methods The Nonlinear Least Squares (NLS) Estimator The Harmonic Summation (HS) estimator* Comparison of Methods Robustness to noise Time-frequency resolution Summary Model Improvements Summary 6 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Outline Correlation-based Methods Nonlinear Least Squares Methods Comparison of Methods Model Improvements Summary 7 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Correlation-based Methods 0.3 0.2 0.1 0 -0.1 -0.2 -0.3 0 0.005 0.01 0.015 0.02 time [s] For a periodic signal x ( n ) with a period τ = 2 π/ω 0 , we have that x ( n ) = x ( n − τ ) = x ( n − 2 π/ω 0 ) . (1) ◮ Unfortunately, τ is unknown so we have to try out different τ ’s (or ω 0 ’s) to find one that satisfies the above equation. ◮ Real-world signals are not perfectly periodic so we might never find one. ◮ Instead, the estimate of τ is the value which minimises some objective function. 8 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Correlation-based Methods Consider the objective function N − 1 � | e ( n ) | 2 J ( a , τ ) = (2) n = τ MAX for a segment of data { x ( n ) } N − 1 n = 0 where e ( n ) = x ( n ) − ax ( n − τ ) , a > 0 ∧ τ ∈ [ τ MIN , τ MAX ] (3) Often referred to as comb-filtering. x ( n ) 1 − a e − j ωτ e ( n ) 9 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Correlation-based Methods 4 Periodogram 3.5 3 2.5 amplitude 2 1.5 1 0.5 0 0 200 400 600 800 1000 1200 1400 1600 1800 2000 frequency [Hz] 10 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Correlation-based Methods 4 Periodogram 3.5 Comb filter 3 2.5 amplitude 2 1.5 1 0.5 0 0 200 400 600 800 1000 1200 1400 1600 1800 2000 frequency [Hz] 10 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Correlation-based Methods Conditioned on τ , the optimal value for a is �� N − 1 � n = τ MAX x ( n ) x ( n − τ ) ˆ a ( τ ) = max , 0 (4) � N − 1 n = τ MAX x 2 ( n − τ ) Inserting this into the objective J ( a , τ ) yields the estimator τ = ˆ argmax max ( φ ( τ ) , 0 ) (5) τ ∈ [ τ MIN , τ MAX ] where φ ( τ ) ∈ [ − 1, 1 ] is the normalised cross correlation function given by � N − 1 n = τ MAX x ( n ) x ( n − τ ) φ ( τ ) = (6) �� N − 1 n = τ MAX x 2 ( n ) � N − 1 n = τ MAX x 2 ( n − τ ) 11 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Correlation-based Methods = 45 0.2 x(n) 0 -0.2 N-1 MAX 0 100 200 300 400 500 600 n 0.2 x(n-45) 0 -0.2 0 100 200 300 400 500 n-45 2 max( (45),0) ( ),0) 1 max( 0 60 80 100 120 140 160 180 200 220 12 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Correlation-based Methods = 95 0.2 x(n) 0 -0.2 N-1 MAX 0 100 200 300 400 500 600 n 0.2 x(n-95) 0 -0.2 0 100 200 300 400 500 n-95 2 max( (95),0) ( ),0) 1 max( 0 60 80 100 120 140 160 180 200 220 13 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Correlation-based Methods = 169 0.2 x(n) 0 -0.2 N-1 MAX 0 100 200 300 400 500 600 n 0.2 x(n-169) 0 -0.2 -100 0 100 200 300 400 n-169 2 max( (169),0) ( ),0) 1 max( 0 60 80 100 120 140 160 180 200 220 14 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Correlation-based Methods = 220 0.2 x(n) 0 -0.2 N-1 MAX 0 100 200 300 400 500 600 n 0.2 x(n-220) 0 -0.2 -200 -100 0 100 200 300 n-220 2 max( (220),0) ( ),0) 1 max( 0 60 80 100 120 140 160 180 200 220 15 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Correlation-based Methods 1 0.8 ( ),0) 0.6 max( 0.4 0.2 0 60 80 100 120 140 160 180 200 220 [samples] 1 0.8 (f),0) 0.6 max( 0.4 0.2 0 200 300 400 500 600 700 800 900 1000 f [Hz] 16 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Correlation-based Methods .... but is anyone actually using the comb filtering method? PRAAT: (Boersma, 1993), well over 1000 citations (Google Scholar) Maximises a windowed normalised cross-correlation function RAPT: (Talkin, 1995), nearly 1000 citations (Google Scholar) Maximises a normalised cross-correlation function YIN: (Cheveigné, 2002), nearly 2000 citations (Google Scholar) Minimises the comb filtering error for a = 1 Kaldi: (Ghahremani et al., 2014), nearly 150 citations (Google Scholar) Maximises a normalised cross-correlation function 17 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Correlation-based Methods Was that really everything? No! Four problems with the correlation-based methods: 1. is prone to producing subharmonic errors, 2. has a sub-optimal time-frequency resolution, 3. is not robust to noise, and 4. not statistically efficient. 18 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Correlation-based Methods Subharmonic error 1 0.8 ( (f)), 0) 0.6 max( 0.4 0.2 0 100 200 300 400 500 600 700 800 900 1000 frequency [Hz] 19 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Correlation-based Methods Subharmonic error 4 Periodogram 3.5 Comb Filter 3 2.5 amplitude 2 1.5 1 0.5 0 0 200 400 600 800 1000 1200 1400 1600 1800 2000 frequency [Hz] 20 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Correlation-based Methods What can we do about these problems? ◮ Hundreds of published pitch estimators trying to solve these problems using various heuristics. ◮ A fundamental flaw of the comb-filtering principle? 21 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Correlation-based Methods Five minutes active break Please complete the SMCNordic pitch survey. ◮ Go to http://tinyurl.com/y3ny4n4n ◮ Fill out the form to the best of your ability 22 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Outline Correlation-based Methods Nonlinear Least Squares Methods The Nonlinear Least Squares (NLS) Estimator The Harmonic Summation (HS) estimator* Comparison of Methods Model Improvements Summary 23 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Nonlinear Least Squares Methods Harmonic Model 0.1 x ( n ) 0 − 0.1 930 932 934 936 938 940 942 944 946 948 950 n [ms] 24 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Nonlinear Least Squares Methods Harmonic Model 0.1 x ( n ) 0 − 0.1 930 932 934 936 938 940 942 944 946 948 950 n [ms] h 1 ( n ) h 2 ( n ) = + h 3 ( n ) e ( n ) + + 24 / 76

Jesper Kjær Nielsen | Fast, Accurate, and Robust Pitch Estimation Nonlinear Least Squares Methods Harmonic Model 0.1 x ( n ) 0 − 0.1 930 932 934 936 938 940 942 944 946 948 950 n [ms] h 1 ( n ) h 2 ( n ) = + h 3 ( n ) e ( n ) + + 24 / 76