REAL-TIME SYSTEMS FOR GESTURE-SOUND INTERACTION Jules Franoise - PowerPoint PPT Presentation

REAL-TIME SYSTEMS FOR GESTURE-SOUND INTERACTION Jules Françoise Real-Time Musical Interactions Team (IMTR) IRCAM jules.francoise@ircam.fr http://imtr.ircam.fr

Interactive musical system ? !"#$"%&'() *$&'() 2

Interactive musical system ? !"#$"%&'() 34%-5*3/-$6"#0# 1())0-2 !"#$%&" '()$%&"*+*,-(./#0# *$&'() Music, Performing arts research: Signal processing, Human-Computer Machine learning gesture & sound Interaction (HCI) Perception, Cognitive sciences 2

Overview 1. Gesture-sound interaction paradigms 2. Formalizing Gesture-sound relationships

GESTURE-SOUND INTERACTION PARADIGMS

New interfaces for musical expression ANR Project Interlude (2008-2011) design: J-L Frechin and U. Petrevski Ircam - Grame - Da Fact - Voxler - Atelier des Feuillantines - nodesign.net 5

Continuous Control sensors Audio analysis Granular synthesis (audio) 6

Triggering sensors kick detection segmented audio recording sample triggering 7

Sound Selection gesture analysis sensors external clock (movement energy) energy comparison sound selection audio database audio samples classified by loudness 8

Navigation: CataRT external clock sensors gesture analysis cursor (x,y) audio database audio units distributed in sound selection a descriptor space audio descriptor segmentation stream extraction Diemo Schwarz, 2005-2011 9

Navigation 10

Intermediate Physical Model Physical Model Parameter extraction sensors (inclination) synchronization parameter comparison sound selection audio database audio samples classified by descriptore 11

Machine learning: Gesture Follower 12

Gesture and action recognition • Common gesture recognition systems: • Gesture = unit → recognition after completion • Simple iconic gestures • Music and performing arts: • Which gesture ? • guess from the gesture beginning ? • How is the gesture performed ? • Motion as a continuous process: • monitor the gesture progression ? → Development of the “gesture follower” 13

Real-time time warping performed gesture (live) gesture parameter recorded example time • Synchronization/following • Recognition • Anticipation (prediction) 14

Aligning dance sequence on prerecorded videos Richard Siegal, Jean-Philippe Lambert, using the gesture follower Ircam 15

Gesture Follower : synchronization Performance { performed gesture Alignment time streching model: encodes temporal structure of the gesture Learning { reference gesture Audio / video recording 16

StreicherKreis - Florence Baschet for augmented string quartet Première in 2008 with Quatuor Danel “gesture” = • acceleration • angular velocity Computer Music Design: Serge Lemouton • pressure • audio energy Commissioned by IRCAM-Centre Pompidou, 2008 18

Gesture Follower - Time Warping acceleration vl 1 vl 2 viola cello 19

FORMALIZING GESTURE-SOUND RELATIONSHIPS

Overview of the paradigms Instantaneous Mapping Temporal Mapping Machine learning 21

Overview of the components ? gesture analysis sensors external clock dimensionality reduction machine learning (gesture modeling) audio audio database machine learning descriptor (sound modeling) extraction audio segmentation time stretching sound synthesis .... sound selection sample triggering 22

Formalization General formalization of these relationships ? Unified framework for gesture-sound interactions Key issues: • Multimodality ( ≠ resolutions) • Combining different paradigms • Multiple temporal scales • Adding hierarchy • Dealing with multiple interacting systems / performers • Synchronization • group interaction 23

Learning gesture-sound couplings Can we learn gesture-sound relationships from example ? • Current system: Gesture Follower to be extended (geometric invariants, hierarchy) • Use learning for other paradigms • Learn in complex situations superposition of paradigms different temporal scales 24

Credits and Acknowledgements Real-Time Musical Interactions team: http://imtr.ircam.fr Frédéric Bevilacqua, Tommaso Bianco, Julien Bloit, Ricardo Borghesi, Baptiste Caramiaux, Arshia Cont, Arnaud Dessein, Sarah Fdili Alaoui, Emmanuel Fléty, Vassilios-Fivos Maniatakos, Norbert Schnell, Diemo Schwarz, Fabrice Guédy, Alain Bonardi, Nicolas Rasamimanana, Bruno Zamborlin, Jean-Philippe Lambert With support of the projects: ANR projects: Interlude, Topophonie (France) EU-ICT project SAME Inside Movement Knowledge Thanks to Richard Siegal/The Bakery 25