Sparse Audio Models For Inverse Audio Problems Rmi Gribonval INRIA - PowerPoint PPT Presentation

Sparse Audio Models For Inverse Audio Problems Rémi Gribonval INRIA Rennes - Bretagne Atlantique, France remi.gribonval@inria.fr

Outline • Inverse problems in audio processing ✓ audio inpainting ✓ source localization • Learning low-dimensional audio models ✓ dictionaries ... R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 2

Contributors • Audio inpainting small-project.eu ✓ A. Adler, N. Bertin, V. Emiya, M. Elad, C.Guichaoua, M. Jafari, M. Plumbley • Source localization ✓ S. Nam echange.inria.fr • Dictionary learning ✓ F. Bach, R. Jenatton, K. Schnass R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 3

Audio inpainting with A. Adler, V. Emiya, M. Elad, M. Jafari, M. Plumbley R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012-

Image Inpainting Observed image Inpainted image R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 5

Audio Inpainting ? Holes (Packet Loss) Clipping 1 Amplitude 0 − 1 0 0.01 0.02 0.03 Time (s) Limited bandwidth Clicks 8000 Frequency (Hz) 6000 Amplitude 1 4000 0 − 1 2000 0 0.01 0.02 0.03 0 Time (s) 0.1 0.2 0.3 0.4 Time (s) R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 6

Audio Inpainting ? Clipping 1 Amplitude 0 − 1 0 0.01 0.02 0.03 Time (s) R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 6

Declipping as a linear inverse problem • Original (unknown) samples x • Clipped (observed) samples y • Subset of reliable samples y reliable • Linear inverse problem x M y reliable = R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 7

Inverse problems & signal models Observation Domain Need for a model = prior knowledge R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 8

Sparse audio models • Time domain • Time-frequency domain (Black = zero) R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 9

Mathematical expression • Signal / image = high dimensional vector x ∈ R d • Model = linear combination of basis vectors (ex: time-frequency atoms, wavelets ) Dictionary of atoms X x ≈ z k d k = Dz (Mallat & Zhang 93) k • Sparsity = small L0 (quasi)-norm | z k | 0 = card { k, z k � = 0 } X ⇥ z ⇥ 0 = k R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 10

CoSparse models and inverse problems Observation Domain R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 11

Audio Declipping • Model ✓ sparsity in time-frequency dictionary x = Dz • Algorithm: ✓ find sparse coefficients such that y = MD ˆ z ˆ z (Orthonormal) Matching Pursuit ( Mallat & Zhang 93 ) ✦ ✓ + ensure compatibility with clipping constraint Convex optimization ✦ ✓ estimate x = D ˆ z ˆ 0.5 Amplitude 0 − 0.5 0 0.01 0.02 0.03 0.04 0.05 time (s) A. Adler, V. Emiya, M. Jafari, M. Elad, R. Gribonval and M. D. Plumbley, Audio Inpainting, IEEE Trans Audio Speech and Language Proc., 2012 R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 12

Audio Declipping • Model ✓ sparsity in time-frequency dictionary x = Dz • Algorithm: ✓ find sparse coefficients such that y = MD ˆ z ˆ z (Orthonormal) Matching Pursuit ( Mallat & Zhang 93 ) ✦ ✓ + ensure compatibility with clipping constraint Convex optimization ✦ ✓ estimate x = D ˆ z ˆ 0.5 Amplitude 0 Clipped − 0.5 0 0.01 0.02 0.03 0.04 0.05 time (s) A. Adler, V. Emiya, M. Jafari, M. Elad, R. Gribonval and M. D. Plumbley, Audio Inpainting, IEEE Trans Audio Speech and Language Proc., 2012 R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 12

Audio Declipping • Model ✓ sparsity in time-frequency dictionary x = Dz • Algorithm: ✓ find sparse coefficients such that y = MD ˆ z ˆ z (Orthonormal) Matching Pursuit ( Mallat & Zhang 93 ) ✦ ✓ + ensure compatibility with clipping constraint Convex optimization ✦ ✓ estimate x = D ˆ z ˆ 0.5 Amplitude 0 Clipped Declipped − 0.5 0 0.01 0.02 0.03 0.04 0.05 time (s) A. Adler, V. Emiya, M. Jafari, M. Elad, R. Gribonval and M. D. Plumbley, Audio Inpainting, IEEE Trans Audio Speech and Language Proc., 2012 R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 12

Audio Declipping • Model ✓ sparsity in time-frequency dictionary x = Dz • Algorithm: ✓ find sparse coefficients such that y = MD ˆ z ˆ z (Orthonormal) Matching Pursuit ( Mallat & Zhang 93 ) ✦ ✓ + ensure compatibility with clipping constraint Convex optimization ✦ ✓ estimate x = D ˆ z ˆ 0.5 Amplitude 0 Clipped Original Declipped − 0.5 0 0.01 0.02 0.03 0.04 0.05 time (s) A. Adler, V. Emiya, M. Jafari, M. Elad, R. Gribonval and M. D. Plumbley, Audio Inpainting, IEEE Trans Audio Speech and Language Proc., 2012 R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 12

Audio Declipping • Model ✓ sparsity in time-frequency dictionary x = Dz • Algorithm: ✓ find sparse coefficients such that y = MD ˆ z ˆ z (Orthonormal) Matching Pursuit ( Mallat & Zhang 93 ) ✦ ✓ + ensure compatibility with clipping constraint Convex optimization ✦ ✓ estimate x = D ˆ z ˆ 0.5 Amplitude 0 Clipped Original Declipped − 0.5 see also talk by B. Mailhé 0 0.01 0.02 0.03 0.04 0.05 time (s) A. Adler, V. Emiya, M. Jafari, M. Elad, R. Gribonval and M. D. Plumbley, Audio Inpainting, IEEE Trans Audio Speech and Language Proc., 2012 R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 12

Source localization with S. Nam R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012-

Localization with «few» microphones • Possible goals ✓ localize emitting sources ✓ reconstruct emitted signals ✓ extrapolate acoustic field • Linear inverse problem y = Mx (discretized) time-series ∈ R N spatio-temporal recorded ∈ R m acoustic field at sensors • Need a model R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 14

Physics-driven design of model • Pressure field p ( � r, t ) • Wave equation on a domain c 2 ∂ 2 r ∈ ˙ ( ∆ p − 1 ∂ t 2 p )( � r, t ) = s ( � r, t ) , � D • Boundary + initial conditions , e.g. � p � n ( ⇥ r, t ) = 0 , ⇥ r ∈ � D R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 15

Physics-driven design of model Discretization • Pressure field p ( � r, t ) x • Wave equation on a domain c 2 ∂ 2 r ∈ ˙ ( ∆ p − 1 ∂ t 2 p )( � r, t ) = s ( � r, t ) , � } D • Boundary + initial conditions , e.g. Ω x = z � p sources � n ( ⇥ r, t ) = 0 , ⇥ r ∈ � D & boundaries R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 15

Group sparse source model • Few non-moving sources = spatially sparse time t z � r,t space � r R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 16

Group sparse regularization • Inverse problem y = Mx • Sparse regularization with mixed norm 1 2 k y � Mx k 2 2 + λ k Ω x k 1 , 2 x = arg min ˆ x Promotes group sparsity, cf Kowalski & Torresani 2009, Eldar & Mishali ✦ 2009, Baraniuk & al 2010, Jenatton & al 2011 R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 17

Sparse Field Reconstruction • Setting • Results ✓ 2D+t vibrating plate 77x77 ✓ 2 sources, random location ✓ 6 microphones, random location ✓ known complex boundaries ✓ ground truth generated with naive discretization Ground truth Sparse reconstruction S. Nam and R. Gribonval. Physics-driven structured cosparse modeling for source localization, ICASSP 2012 R. GRIBONVAL - Workshop on Sparsity, Compressed Sensing and Applications November 5th 2012 18

Sparse Audio Models For Inverse Audio Problems Rmi Gribonval INRIA - PowerPoint PPT Presentation

Sparse Audio Models For Inverse Audio Problems Rmi Gribonval INRIA Rennes - Bretagne Atlantique, France remi.gribonval@inria.fr Outline Inverse problems in audio processing audio inpainting source localization Learning

Hybrid sparse stochastic processes and the resolution of linear inverse problems Michael

Sparse Signal Processing Parcimonie en Traitement du Signal Rmi Gribonval INRIA Rennes -

CS378 - Mobile Computing Audio Android Audio Use the MediaPlayer class Common Audio

Audio Device Client Better and Faster Audio I/O on Web Hongchan Choi Google Chrome Web Audio

Compressive sensing principles and iterative sparse recovery for inverse and ill-posed problems

8. Audio databases About digital audio: Advent of digital audio CD in 1983. Order of

Outline Successful Multiparty Audio Communication over the Internet Introduction Problems

CobraNet CobraNet Audio Network Audio Network Overview Overview Developed by Peak Audio

Invertible Generative Models for Inverse Problems Mitigating Representation Error and Dataset Bias

Cirrus Audio Solutions Cirrus Audio Solutions Home Audio Portable Audio Personal CD Player

Federal EdTech Legislation and Regulations that You Need to Follow Audio Setup Test Your Audio

Sparse Gaussian Process Approximations Dr. Richard E. Turner ( ret26@cam.ac.uk ) Computational and

Welcome! The webinar will begin at 2:00 Eastern/11:00 Pacific Audio Tips Todays audio is

Web Audio Tutorial 9/4, 2015 Your Goal Learn what digital audio is and how it works

Ceiling Wall Hub Table Control About Stem Audio Here at Stem Audio, we want everyone to have

Sound File Formats Raw data has samples (interleaved w/stereo) Need way to parse raw

Audio Declipping Using Sparse Multiscale Representations Boris Mailh Queen Mary University of

1 Please do not put your phone on hold For telephone audio feed, enter the AUDIO PIN

WebEx Functionality Use audio on computer if available Mute phones or audio to reduce

Housekeeping If you have problems with the webinar audio, you can j oin by phone at

Fundamentals of Audio Programming Bjorn Roche XO Audio, LLC Who Am I? Software Designer

Audio and Speech August 13, 2001 Audio 2 Digital sound anti-aliasing amplifier codec filter

Audio Indexing and Retrieval IT6902; Semester B, 2004/2005; Leung Audio Indexing and Retrieval

Digital Audio 3 4 Sampling 44.1 kHz 48 kHz 96 kHz 192 kHz digitisation =