User-Guided Variable-Rate Time-Stretching Via Stiffness Control Nicholas J. Bryan, Jorge Herrera, and Ge Wang Stanford University | CCRMA DAFx 2012
Introduction • User control over variable-rate time-stretch processing • Stretch some regions more than others (e.g. stretchability, stiffness) • Transient preservation, rhythmic warping, emphasis modification, etc. 1 0.5 0 − 0.5 −
Rhythmic Warping Demo 2 Original s(t) 1.5 k(t) Amplitude � (t) 1 No Stiffness / Stiffness 0.5 0 − 0.5 Stretched (2x) − 1 0 0.2 0.4 0.6 0.8 1 1.2 1.4 Time (s) Swung + Stretched (2x) 2 1.5 Amplitude 1 0.5 0 − 0.5 − 1 0 0.2 0.4 0.6 0.8 1 1.2 1.4 Time (s)
Motivation • Automatic methods have no mechanism for user input • Direct manipulation of the stretch rate is hard! 2x ?
Prior Work 1 0.5 0 − − 0.5 − − − − − − 1 0.5 0 − 0.5 1 − 0.5 0 − 0.5 − − − − − − − ProTools, Logic Pro, FL Studio, etc. Nielson and Brandorff, 2002 − − − −
Proposed Method • User annotates stiffness + timing constraints 1 0.5 0 − 0.5 − • Solve optimization problem to convert stiffness to stretch factor • Use pre-existing time-stretch processor to stretch sound Stiffness Stretch Factor Processor
Step I: Spring Chain • Model audio as chain of springs … • Solve for equilibrium via Hooke’s Law F i = − k i x i • Spring stiffness an intuitive measure (i.e. proportional)
Initial Formulation find x subject to f = 0 x T 1 = L x = spring displacement f i = k i +1 x i +1 − k i x i f = spring forces k = spring sti ff ness L = final length
Problem • Violates intuition: no initial rest length k 1 = 100 k 1 = 100 k 2 = 1 k 2 = 1 Γ = 2
Reformulation find x find x subject to f = 0 subject to f = 0 x T 1 = L ( x + x 0 ) T 1 = L x + x 0 ≥ 0 x 0 = Initial Rest Length
Reformulation minimize || f || 2 + µ || x || 2 x ( x + x 0 ) T 1 = L subject to x + x 0 ≥ 0 x 0 = Initial Rest Length µ = Penalty Weight • Minimize the force disturbance from equilibrium (smoothly)
Step II: Stiffness to Stretch Factor • Given input and output lengths, compute stretch factor as simple ratio 3 2.5 2 γ = x + x 0 1.5 s(t) Amplitude k(t) � (t) 1 x 0 0.5 0 − 0.5 − 1 0 0.2 0.4 0.6 0.8 1 1.2 1.4 Time (s)
Step III: Time-Stretching • Given optimal variable-rate stretch factor, process sound • Phase Vocoder (PV) • Pitch Synchronous overlap add (PSOLA)
Extensions • Rhythmic warping, smoothing of user input, max stretching limits • Example: Straight-to-Swing minimize || f || 2 + µ || x || 2 x ( x + x 0 ) T 1 = L subject to x + x 0 ≥ 0 ( x 1 + x 1 0 ) T 1 = 2 3 L/ 2 ( x 2 + x 2 0 ) T 1 = 1 3 L/ 2 ( x 3 + x 3 0 ) T 1 = 2 3 L/ 2
Results
Varying Stretch Length 4.5 minimize || f || 2 + µ || x || 2 x 4 � = .5 ( x + x 0 ) T 1 = L subject to � = .75 3.5 � = 1 x + x 0 ≥ 0 � = 1.5 3 � = 2.0 Γ = . 5 2.5 � (t) Γ = . 75 2 1.5 Γ = 1 . 0 1 Γ = 1 . 5 0.5 0 Γ = 2 . 0 − 0.5 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 Time (s) • Varying the overall stretch factor gives smooth, intuitive stretch factors
Regularization 3.5 minimize || f || 2 + µ || x || 2 x µ = .001 ( x + x 0 ) T 1 = L subject to µ = .01 3 x + x 0 ≥ 0 µ = .1 µ = 1 2.5 µ = . 001 � (t) 2 µ = . 01 1.5 µ = . 1 1 µ = 1 0.5 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 Time (s) • Regularization penalty smooths the time-varying stretch factor
Rhythmic Emphasis Modification 5 s(t) k(t) � (t) 4 original 3 warped Amplitude 2 warped + stretched 1 (slowed by 1.3x) 0 I’m gonna make him an offer he can’t refuse − 1 0 0.5 1 1.5 2 2.5 Time (s) I’m gonna make him an offer he can’t refuse I’m gonna maaake him an offer he caaaan’t refuse
Conclusions • Method of user control over variable-rate time-stretching • Decouples stiffness control + timing constraints to user • Converts user control into optimal time-dependent stretch rate • Agnostic to time modification algorithms
Acknowledgments & Thanks • National Science Foundation Creative IT grant No. IIS-0855758 • School of Humanities and Sciences, Stanford University
User-Guided Variable-Rate Time-Stretching Via Stiffness Control Nicholas J. Bryan, Jorge Herrera, and Ge Wang Stanford University | CCRMA DAFx 2012
Recommend
More recommend
