Introduction Non-negative Matrix Factorization Parametric spectrogram model Score informed source separation Conclusion Score informed audio source separation using a parametric model of non-negative spectrogram Romain Hennequin, Roland Badeau and Bertrand David Telecom ParisTech < forename > . < surname > @telecom-paristech.fr May 24, 2011 Romain Hennequin, Roland Badeau and Bertrand David Score informed source separation - slide 1/27
Introduction Non-negative Matrix Factorization Parametric spectrogram model Score informed source separation Conclusion Introduction Monaural source separation in a musical signal: separation of the signal of each instrument. Romain Hennequin, Roland Badeau and Bertrand David Score informed source separation - slide 2/27
Introduction Non-negative Matrix Factorization Parametric spectrogram model Score informed source separation Conclusion Introduction The information in the score of the piece is used to guide the separation. The score is here a MIDI file aligned on the signal (this paper does not deal with alignment). Only harmonic instruments are modeled. Romain Hennequin, Roland Badeau and Bertrand David Score informed source separation - slide 3/27
Introduction Non-negative Matrix Factorization Parametric spectrogram model Score informed source separation Conclusion Introduction Overview Parametric spectrogram model derived from non-negative matrix factorization (NMF) to decompose the mixture spectrogram. A parametric time/frequency mask is computed for each instrument. Masks are initialized (and constrained) from the score and then finely estimated to fit the mixture spectrogram. Masks are used to separate the instruments (Wiener filtering). Romain Hennequin, Roland Badeau and Bertrand David Score informed source separation - slide 4/27
Introduction Non-negative Matrix Factorization Parametric spectrogram model Score informed source separation Conclusion Introduction Why use the score? MIDI files widely available. Very compact description of the audio. Under determined blind separation remains a very difficult problem. Sometimes, blind separation is hopeless (separation of several voices played by the same instrument). Romain Hennequin, Roland Badeau and Bertrand David Score informed source separation - slide 5/27
Introduction Non-negative Matrix Factorization Parametric spectrogram model Score informed source separation Conclusion Outline Non-negative Matrix Factorization 1 Principle Features Parametric spectrogram model 2 Source parametric spectrogram Example Mixture model Score informed source separation 3 Separation process Results Romain Hennequin, Roland Badeau and Bertrand David Score informed source separation - slide 6/27
Introduction Non-negative Matrix Factorization Principle Parametric spectrogram model Features Score informed source separation Conclusion Outline 1 Non-negative Matrix Factorization Principle Features 2 Parametric spectrogram model 3 Score informed source separation Romain Hennequin, Roland Badeau and Bertrand David Score informed source separation - slide 7/27
Introduction Non-negative Matrix Factorization Principle Parametric spectrogram model Features Score informed source separation Conclusion Principle of NMF Low-rank approximation: R � V ft ≈ ˆ V ft = W fr H rt with W ≥ 0 , H ≥ 0 , R ≪ min( F , T ) r =1 Romain Hennequin, Roland Badeau and Bertrand David Score informed source separation - slide 8/27
Introduction Non-negative Matrix Factorization Principle Parametric spectrogram model Features Score informed source separation Conclusion Principle of NMF Features Extract redundant patterns from the data. Fundamental property: non-negativity constraint. Atoms lie in the same space as the data. Only positive combinations (no black energy). Perceptive description: decomposition of musical spectrograms on a basis of notes. Application in automatic transcription, source separation, audio inpainting. . . Romain Hennequin, Roland Badeau and Bertrand David Score informed source separation - slide 9/27
Introduction Non-negative Matrix Factorization Principle Parametric spectrogram model Features Score informed source separation Conclusion Principle of NMF Limitations Does not permit to deal with time-frequency variations (vibrato) We needed a representation linked with parameters of interest (fundamental frequency) Romain Hennequin, Roland Badeau and Bertrand David Score informed source separation - slide 10/27
Introduction Non-negative Matrix Factorization Source parametric spectrogram Parametric spectrogram model Example Score informed source separation Mixture model Conclusion Outline 1 Non-negative Matrix Factorization 2 Parametric spectrogram model Source parametric spectrogram Example Mixture model 3 Score informed source separation Romain Hennequin, Roland Badeau and Bertrand David Score informed source separation - slide 11/27
Introduction Non-negative Matrix Factorization Source parametric spectrogram Parametric spectrogram model Example Score informed source separation Mixture model Conclusion Parametric spectrogram of a single instrument [Hennequin et al., DAFx 2010] What does an atom look like in a musical spectrogram? In a musical spectrogram most of the (non-percussive) elements are instruments notes which are generally harmonic tones. Parameters of interest are generally the fundamental frequency of these tones, and the shape of the amplitudes of the harmonics. Proposed method: parametric model of spectrogram with harmonic atoms. Romain Hennequin, Roland Badeau and Bertrand David Score informed source separation - slide 12/27
Introduction Non-negative Matrix Factorization Source parametric spectrogram Parametric spectrogram model Example Score informed source separation Mixture model Conclusion Parametric spectrogram of a single instrument Time-varying atoms in NMF: R R � � W f rt ˆ ˆ V ft = V ft = 0 W fr H rt fr H rt → r =1 r =1 f rt is the time-varying fundamental frequency associated to each 0 atom. Romain Hennequin, Roland Badeau and Bertrand David Score informed source separation - slide 13/27
Introduction Non-negative Matrix Factorization Source parametric spectrogram Parametric spectrogram model Example Score informed source separation Mixture model Conclusion Parametric atoms Parametric harmonic atom construction n h ( f rt 0 ) � W f rt a k g ( f − kf rt fr = 0 0 ) k =1 Romain Hennequin, Roland Badeau and Bertrand David Score informed source separation - slide 14/27
Introduction Non-negative Matrix Factorization Source parametric spectrogram Parametric spectrogram model Example Score informed source separation Mixture model Conclusion Algorithm Parametric spectrogram (single instrument) n h R � � ˆ a k g ( f − kf rt V ft = 0 ) h rt r =1 k =1 � �� � f rt 0 W fr Minimization Global optimization w.r.t. f rt is impossible (numerous local 0 minima in cost function). ⇒ one atom is introduced for each MIDI note. Optimization thus becomes local (fine estimate of f rt 0 ). Romain Hennequin, Roland Badeau and Bertrand David Score informed source separation - slide 15/27
Introduction Non-negative Matrix Factorization Source parametric spectrogram Parametric spectrogram model Example Score informed source separation Mixture model Conclusion Decomposition of a synthetic spectrogram Original power spectrogram 40 5 35 30 4 25 Frequency (kHz) 20 3 15 10 2 5 0 1 −5 0 50 100 150 200 250 300 Time (frame) Spectrogram of the first bars of Bach’s first prelude played by a synthesizer. Romain Hennequin, Roland Badeau and Bertrand David Score informed source separation - slide 16/27
Introduction Non-negative Matrix Factorization Source parametric spectrogram Parametric spectrogram model Example Score informed source separation Mixture model Conclusion Obtained decomposition 70 −15 60 50 −20 Semitones 40 −25 30 20 −30 10 −35 50 100 150 200 250 300 Frames Activations h rt for each MIDI note. Romain Hennequin, Roland Badeau and Bertrand David Score informed source separation - slide 17/27
Introduction Non-negative Matrix Factorization Source parametric spectrogram Parametric spectrogram model Example Score informed source separation Mixture model Conclusion Mixture spectrogram model Mixture model The mixture is made up of K sources indexed by k . Source k V k following: is modelized with spectrogram ˆ R n h � � V k ˆ a kp g ( f − pf krt ft = ) h krt 0 r =1 p =1 � �� � f krt 0 W kfr Mixture spectrogram is then: K � V mix = ˆ V k ˆ k =1 Romain Hennequin, Roland Badeau and Bertrand David Score informed source separation - slide 18/27
Introduction Non-negative Matrix Factorization Source parametric spectrogram Parametric spectrogram model Example Score informed source separation Mixture model Conclusion Mixture spectrogram model Mixture model Parameters to be estimated for each source k : Fundamental frequency of each atom r at each time t : f krt , 0 Amplitudes of harmonics: a kp , Activations of each note r at each time t : h krt . Decomposition obtained with a multiplicative algorithm aiming at minimizing a β -divergence between V mix and ˆ V mix . Romain Hennequin, Roland Badeau and Bertrand David Score informed source separation - slide 19/27
Recommend
More recommend
