A Summary of Formats for Streaming and Storing Music-related - PowerPoint PPT Presentation

A Summary of Formats for Streaming and Storing Music-related Movement and Gesture Data Alexander R. Jensenius Oslo Nicolas Castagné ACROE, Grenoble Antonio Camurri Genova Esteban Maestre Pompeu Fabra Joseph Malloch CIRMMT, McGill Douglas McGilvray Glasgow Enactive Conference 2007

Movement Action Gesture

?

analysis

AOA ASF / AMC BRD BVA / BVH C3D CSM MCML MoCap MIDI Score MPEG 4/7 Control MusicXML Audio OSC Video SDIF Annotations SLML Analyses VHML

!

30 % use raw data (no format) 50 % use a proprietary, home-made format 40 % use the format of the device at hand 80 % don’t use a unique format < 10 % use a known and officially released format Luciani et al. 2006

performance

=>

GMS Gesture Motion Signal GDIF Gesture Description Interchange Format OSC Open Sound Control SDIF Sound Description Interchange Format PML Performance Markup Language

Low-level signals structure encoding

APML AML AOA ASF AMC Proprietary BRD not generic enough Hardware-specific BVA not minimal enough Full-body skeleton BVH not efficient enough 3D only C3D not low level enough ASCII-based CSM not high level enough Low-level data only EBF MCML MPEG 7 MPML MURML SLML VHML

as generic as possible GMS format low-level Gesture and Motion binary minimal Signal format X(t) Tracks Y(t) Channels Scene Units

A Scene made of 3 Units • Unit 1 : “mocap” N 3D Position channel • Unit 2 : “Force Feedback” 1 3D Position channel 1 3D Force channel • Unit 3 : “keyboard” 64 A-Dimensional channels

qualitative Intensity Laban effort parameters head up breathing breathing left arm turn Descriptions 1b 2a 2b 2c 3a 3b 3c Sections 1a Trajectories Segmentation 2 Segmentation 1 Velocity Raw data quantitative time GDIF Gesture Description Interchange Format

Sensor Data Realtime Movement GDIF OSC UDP/IP device processing model GDIF OSC SDIF Analysis Realtime Non-realtime Annotations XML Analysis GDIF Gesture Description Interchange Format

music/tech communication protocol OSC 1990s: CNMAT Open Sound URL-based: Control /my-controller/slider/1/pressure 0.5 /my-reverb/room-size 0.7 www.opensoundcontrol.org

Analysis Namespace Meta /emotion Functional /performance /environment Acquisition Sensor Pre- Descriptive /instrument Movement Raw Data device processing /body /cooked /raw

GDIF-OSC Malloch et al. 2007

GDIF-XML Maestre et al. 2007

1990s: IRCAM, CNMAT, UPF SDIF S ound descriptions Sound Basic data format Description framework Interchange Format Synchronisation Wide support http://sdif.sourceforge.net

PML Extension of MusicXML Analysis of musical performance Performance Markup Score, performance & analysis information in Language separate, overlapping hierarchies

Multimodal data + multimedia Automatic converters between layers Synchronisation EyesWeb XMI Multilayered expressive gesture model: MoCap Trajectories Cues and gestures www.eyesweb.org

=

Both streaming and storage Low-, mid- and high-level Consensus...? Open, Free, eExtensible Compatible (MIDI, OSC, SDIF, MusicXML) Synchronisation (data + media + analyses)

GDIF PML GMS SDIF MusicXML OSC XML