Bowing the violin A case study for auditory-motor pattern modeling - PowerPoint PPT Presentation

Bowing the violin A case study for auditory-motor pattern modeling in the context of music performance Quim Llimona Torras Advisor: Esteban Maestre Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Outline Introduction Experimental design Data acquisition Feature extraction Database construction Preliminary analysis Conclusion 2 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Score vs audio 3 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Score vs audio 3 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Performance encoding Musical � Score Performer Instrument Musical � Sound Intended � Musical � Message ��� Instrumental � Gesture ������������������������������������ Perceived � Musical � Message Note � Event � Sequence ���� Control � Parameters Audio � Perceptual � Features Discrete � Nature Continuous � Nature Continuous � Nature Low � Dimensionality Low � Dimensionality High � Dimensionality Many omit the instrumental gesture step 4 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion MUSMAP Marie Curie IOF action ’s This project is part of the first phase of MUSMAP 5 Quim Llimona, 2014 the ¡musician’s ¡bowing ¡ ’s ¡ will ¡ to ¡ expand ¡ his ¡ career ¡

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Objectives General � Define methodology and setup � Provide software and knowledge � Specific � Design and record experiments Implement bowing acquisition and extraction Process and build a database Upload to repovizz � Perform preliminary analysis 6 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Context: the violin 7 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Parts of the violin scroll tip nut fingerboard hair ribbon plate bridge tailpiece frog 8 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Violin acoustics (i) 9 http://www.phys.unsw.edu.au/jw/Bows.html Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Violin acoustics (i) 9 http://www.phys.unsw.edu.au/jw/Bows.html Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Violin acoustics (i) 10 https://www.youtube.com/watch?v=KPpBvHXYWz4 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Violin acoustics (i) 10 https://www.youtube.com/watch?v=KPpBvHXYWz4 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Violin acoustics (ii) 11 https://www.youtube.com/watch?v=6JeyiM0YNo4 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Violin acoustics (ii) 11 https://www.youtube.com/watch?v=6JeyiM0YNo4 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Violin acoustics (ii) Helmholtz regime 11 https://www.youtube.com/watch?v=6JeyiM0YNo4 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Control parameters (i) Bow velocity: Controls amplitude Bow force (or pressure) : Controls high frequencies � Bow-bridge distance: Controls both Others: Position, tilt, skew, inclination 12 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Control parameters (ii) This is the Schelleng diagram Bow pressure playable region Sounding point (relative bow-bridge distance) 13 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion towards joint modeling of auditory and motor spaces Experimental design 14 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Player-instrument matrix Violin 1 Violin 2 Violin 3 instruments 
 Three Player 1 Player 1 Player 1 players Violin 1 Violin 2 Violin 3 Player 2 Player 2 Player 2 Violin 1 Violin 2 VIolin 3 Player 3 Player 3 Player 3 15 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Score Articulation Duration Tone Dynamics Pitch (string and position) Bow direction Redundancy 16 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Sampling dimensions (i) Articulation Legato 
 Martelé 17 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Sampling dimensions (i) Duration legato: 120 bps martele: 132 bps Half 
 Quarter 18 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Sampling dimensions (ii) Dynamics Piano 
 Mezzoforte 
 Forte 19 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Sampling dimensions (iii) Tone (1) Sul tasto (1) (2) Ordinary (2) (3) Sul ponticello (3) 20 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Sampling dimensions (iv) Pitch 2, 5, 7 semitones (0 to 50% length) Strings sampled independently 21 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Sampling dimensions (v) Bow direction Up Down 22 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Data acquisition 23 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Overview Sync Generator PERFORMANCE CAPTURE SCENARIO Audio I/O hi-speed IR audio cameras (x12) Qualysis Aligment and Track hi-quality formatting video camera Manager MULTIMODAL load cell REPOSITORY Qualysis Acquisition Board 24 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Audio ambience � Schoeps Colette close-up � DPA 4099-V pick-up � Fishman V100 25 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Video reference camera Sony PMW-EX3 (HD) 26 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Motion capture Qualysis It’s an infrared camera based motion capture system with passive markers http://www.labbase.net/Supply/SupplyItems-786112.html 27 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Motion capture | violin (i) plate � top_left top_right bottom_left bottom_right scroll Notice the asymmetry 28 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Motion capture | violin (ii) These are virtual markers string_G_bridge string_D_bridge string_A_bridge string_E_bridge string_G_nut string_E_nut fb_center 29 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Motion capture | bow The antenna breaks colinearity frog � antenna_left antenna_right stick � tip � corner stick tip 30 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Motion capture | body Forehead Nape Wrist Elbow Shoulder 31 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Load cell Virtual string With motion capture as well For calibration purposes 32 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Synchronization SMPTE Word Clock Video frame 33 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Feature extraction 34 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Overview (i) Motion capture Audio Bow force (estimated from other parameters) Pitch Energy Bow velocity Aperiodicity Bow position Bow-bridge distance Bow tilt Bow skew Bow inclination Current string Pseudoforce (left and right) Used in force regression Deformation 35 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion Overview (ii) 36 Quim Llimona, 2014

Introduction | Experiment | Data | Features | Database | Analysis | Conclusion High-level features Bow position Bow velocity Bow force Bow-bridge distance Bow tilt Bow skew Bow inclination Current string 37 Quim Llimona, 2014