Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Graham Percival ∗ , Nicholas Bailey ∗ , George Tzanetakis † ∗ School of Engineering, University of Glasgow, UK † Department of Computer Science, University of Victoria, Canada http://percival-music.ca/vivi.html This work is licensed under the Creative Commons Attribution-ShareAlike 3.0 License .
Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist 1 / 12 Teaching Bow Control to a Virtual Violinist 1 Introduction Music performance with Vivi, the Virtual Violinist 2 Performing on a virtual instrument Generating sound: Physical modeling of a violin Pedagogical inspiration for physical parameters 3 Intelligent control loop Intelligent feedback control of bow force Automatically determining other parameters 4 Making music Musical performance style Conclusion and future work
Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Introduction 2 / 12 Music performance with Vivi, the Virtual Violinist Input Output Music notation Video file Audio file � �� � � = 96 � � � � � � � f Note data Audio signal Control (optional) Violin physical model Virtual violinist (Controller) (System) Physical parameters Automatic processing Select classifier Audio (intelligent feedback control) Sound signal Audio analysis and quality SVM classifiers (Sensor)
Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Introduction 3 / 12 Music example: “black-box testing” Input Output � = 96 � � �� � � � � III � � � � � � � � � � � � � � � � � � � � � � p mp f � � � � � � � � � � � � � � � � � �� � II � II � � � � � � � � � 4 � � � p mf � = 120 �� �� tip �� � �� �� 4 � � � � � � � � � 3 pizz. 6 � mb � � � � � � � � p � � � � � mp f � = 88 lh � �� ��� ��� � � � � � � III �� � � � arco � �� �� � � �� �� � 10 � � � � � � � p mp f II II II II II II (pdf produced with GNU LilyPond, Video: black-box.mpeg MusicXML input also possible)
Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Performing on a virtual instrument 4 / 12 Generating sound: Physical modeling of a violin No recordings of violin performance; we use physics [1] • Wave equation for a stiff string with modal dampening ∂ 2 y ( x , t ) − T ∂ 2 y ( x , t ) + EI ∂ 4 y ( x , t ) + R L ( ω ) ∂ y ( x , t ) ρ L = F ( x , t ) ∂ t 2 ∂ x 2 ∂ x 4 ∂ t [1] M. Demoucron, “On the control of virtual violins: Physical modelling and control of bowed string instruments,” Ph.D. dissertation, IRCAM, Paris, 2008
Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Performing on a virtual instrument 4 / 12 Generating sound: Physical modeling of a violin No recordings of violin performance; we use physics [1] • Wave equation for a stiff string with modal dampening ∂ 2 y ( x , t ) − T ∂ 2 y ( x , t ) + EI ∂ 4 y ( x , t ) + R L ( ω ) ∂ y ( x , t ) ρ L = F ( x , t ) ∂ t 2 ∂ x 2 ∂ x 4 ∂ t Implemented as a C++ library, published under GNU GPLv3+ Input parameters • Violin string number s • Left-hand finger position x 1 • Bow-bridge distance x 0 , velocity v b , force F b [1] M. Demoucron, “On the control of virtual violins: Physical modelling and control of bowed string instruments,” Ph.D. dissertation, IRCAM, Paris, 2008
Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Performing on a virtual instrument 5 / 12 Music performance with Vivi, the Virtual Violinist Input Output Music notation Video file Audio file � �� � � = 96 � � � � � � � f Note data Audio signal Control (optional) Violin physical model Virtual violinist (Controller) (System) Physical parameters Select classifier Audio Sound signal Audio analysis and quality SVM classifiers (Sensor)
Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Performing on a virtual instrument 6 / 12 Pedagogical inspiration for physical parameters Pedagogical inspiration • Suzuki violin book 1 • Treat Vivi like a beginning student
Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Performing on a virtual instrument 6 / 12 Pedagogical inspiration for physical parameters Pedagogical inspiration • Suzuki violin book 1 • Treat Vivi like a beginning student Most parameters can come from sheet music and pedagogy • String s , finger x 1 : printed note • Bow-bridge distance x 0 : dynamic “bow lanes” or “Kreisler Highway” • Bow velocity v b : teacher saying “use half bow” and giving tempo
Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Performing on a virtual instrument 6 / 12 Pedagogical inspiration for physical parameters Pedagogical inspiration • Suzuki violin book 1 • Treat Vivi like a beginning student Most parameters can come from sheet music and pedagogy • String s , finger x 1 : printed note • Bow-bridge distance x 0 : dynamic “bow lanes” or “Kreisler Highway” • Bow velocity v b : teacher saying “use half bow” and giving tempo Bow force F b from SVM classifiers 1 not audible: needs a lot more bow force (example) 2 “whispy”: needs a little more bow force (example) 3 acceptable: no change (example) 4 “harsh”: needs less bow force (example) 5 not recognizable: needs much less bow force (example)
Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Performing on a virtual instrument 7 / 12 Interactive training Basic training: only 32 files (bad example) After interactive training: 203 files (good example) ≈ 4 hours to be fully trained (including calculations)
Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Intelligent control loop 8 / 12 Intelligent feedback control of bow force Note setup Output Note data Dynamic Pitch Select string Initial parameter audio s and finger position signal database x 0 x 1 v b Control s , x 1 , x 0 , v b , F b Violin physical Central controller K model F b Select audio classifier F b signal Adjust force Audio analysis and Sound SVM classifiers quality
Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Intelligent control loop 8 / 12 Intelligent feedback control of bow force Note setup Output Note data Dynamic Pitch Select string Initial parameter audio s and finger position signal database x 0 x 1 v b Control s , x 1 , x 0 , v b , F b Violin physical Central controller K model F b Select 16 classifiers audio classifier (4 strings x F b signal 4 dynamics) Adjust force Audio analysis and Sound SVM classifiers c ∈ { 1 , 2 , 3 , 4 , 5 } quality
Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Intelligent control loop 8 / 12 Intelligent feedback control of bow force Note setup Output Note data Dynamic Pitch Select string Initial parameter audio s and finger position signal database x 0 x 1 v b Control s , x 1 , x 0 , v b , F b Violin physical Central controller K model F b Select 16 classifiers audio classifier (4 strings x F b signal 4 dynamics) Adjust force Audio analysis and Sound SVM classifiers c ∈ { 1 , 2 , 3 , 4 , 5 } quality
Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Intelligent control loop 8 / 12 Intelligent feedback control of bow force Note setup Output Note data Dynamic Pitch Select string Initial parameter audio s and finger position signal database x 0 x 1 v b Control s , x 1 , x 0 , v b , F b Violin physical Central controller K model F b Select 16 classifiers audio classifier (4 strings x F b signal 4 dynamics) Adjust force Audio analysis and Sound F b ← F b K (3 − c ) SVM classifiers c ∈ { 1 , 2 , 3 , 4 , 5 } quality
Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Intelligent control loop 8 / 12 Intelligent feedback control of bow force Note setup Output Note data Dynamic Pitch Select string Initial parameter audio s and finger position signal database x 0 x 1 v b Control pre-calculate s , x 1 , x 0 , v b , F b Violin physical Central controller K model F b Select 16 classifiers audio classifier (4 strings x F b signal 4 dynamics) Adjust force Audio analysis and Sound F b ← F b K (3 − c ) SVM classifiers c ∈ { 1 , 2 , 3 , 4 , 5 } quality
Physical Modeling meets Machine Learning: Teaching Bow Control to a Virtual Violinist Intelligent control loop 9 / 12 Automatically determining K Cost of a candidate K 1 Play a simple musical pattern 2 Get list C of judgements c 3 Split C into sublists A i based on c changing from below to above 3 (and vice versa) 4 Calculate | A | � � (3 − c ) 2 cost = i c ∈ A i 5 Repeat 12 times and find the inter-quartile geometric mean
Recommend
More recommend
