Austrian Research Institute for Artificial Intelligence (OFAI) Thomas Grill, Arthur Flexer Visualization of perceptual qualities in textural sounds International Computer Music Conference (ICMC 2012), Ljubljana/Slovenia
Fundamental questions • How can digital sound material be described? • How can sounds be organized? • How can sounds and collections thereof be visualized? • Focus on sampled sound with textural characteristics Grill and Flexer: Visualization of perceptual qualities in textural sounds 3 ICMC 2012, Ljubljana/Slovenia
ringing cheeping gasping smashing piercing peeping whooping tinkling raucous chattering crooning bellowing sobbing bumping snarling growling pitch crying thumping burping croaking clattering yapping keening splashing yelping rustling volume squealing howling barking sniveling moaning pealing tone rattling grunting clanging coughing quacking whining gagging fizzing wheezing honking hissing bawling trumpeting swishing sneezing rumbling bubbling ripping cooing chirping shouting shuffling tearing popping roaring thunderous scratching snorting crashing crunching cackling tolling clucking silent tapping soothing crowing tranquil melodious cacophonous singing quiet tune loud tinkling noisy rhythmic mumbling twittering din beat blaring cawing racket chattering murmuring whistling clapping booming whispering mewing snapping snoring yelling mooing crackling sighing
Describing sounds • Predominant scheme: Semantic tagging (sound origin, recording context, etc.) • Sonic qualities are equally important/interesting, especially for abstract sounds or use in acousmatic composition • Description ⇨ Organization Grill and Flexer: Visualization of perceptual qualities in textural sounds 5 ICMC 2012, Ljubljana/Slovenia
Identification of perceptual qualities in textural sounds • What are the most significant qualities of textural sounds? ➡ Repertory grid technique used to elicit qualities ( personal constructs ) "ex nihilo", for a specific selection of subjects (interviewees) and objects under examination ( items ) • Interviewees (subjects) are asked to name differences between two randomly chosen sound examples ➡ Bipolar qualities spanning range from one sound to the other Grill, Flexer and Cunningham. Identification of perceptual qualities in textural sounds using the repertory grid method. Proceedings of the 6th Audio Mostly Conference , 2011 Grill and Flexer: Visualization of perceptual qualities in textural sounds 6 ICMC 2012, Ljubljana/Slovenia
Repertory Grid for sounds • Elicitation of ~10 bipolar constructs per subject • Subjects rate all 20 sounds (grades 1 to 5) using own personal constructs 1 … 5 motion textural impulse high excentric evolutionary well-defined regular narrative pitched smooth static coherent continuous low contained repetitive diffused irregular static non-pitched porous A 4 4 4 1 2 4 4 2 4 3 3 B 5 3 5 5 5 1 3 1 5 2 1 C 4 5 2 2 4 -5 5 3 5 5 4 D 4 2 5 4 3 4 4 3 4 2 3 E 2 4 1 1 2 4 1 5 5 3 5 F 1 1 2 2 2 -3 2 5 5 4 5 G 5 5 5 5 5 2 1 2 5 1 1 H 4 3 3 1 2 5 1 1 5 2 4 I 4 2 2 2 2 5 2 2 4 1 4 J 2 1 5 3 1 -2 5 5 3 5 3 K 5 2 4 4 4 4 3 1 5 4 2 L 1 1 1 3 1 -2 1 5 5 5 5 M 4 5 5 1 2 2 3 2 5 3 2 N 3 1 4 4 1 4 4 5 5 4 2 O 4 2 4 3 3 -3 5 4 3 5 3 P 2 2 3 3 3 4 5 3 5 5 4 Q 5 5 5 3 5 -5 1 1 5 1 1 R 3 3 4 2 3 2 2 3 4 2 3 S 2 2 5 2 3 4 4 4 2 3 2 T 1 1 4 4 1 4 3 2 3 5 2 Grill and Flexer: Visualization of perceptual qualities in textural sounds 7 ICMC 2012, Ljubljana/Slovenia
high/low • 16 subjects • expert listeners • 202 constructs • mostly German ordered/chaotic Grill and Flexer: Visualization of perceptual qualities in textural sounds 8 ICMC 2012, Ljubljana/Slovenia
http://grrrr.org/test/classify Grill and Flexer: Visualization of perceptual qualities in textural sounds 9 ICMC 2012, Ljubljana/Slovenia
Inter-rater agreement Agreement α Agreement α Construct (core group)* (all n ≥ 10) high – low 0.588 0.519 ordered – chaotic 0.556 0.447 natural – artificial 0.551 0.492 smooth – coarse 0.527 0.420 tonal – noisy 0.523 0.435 homogeneous – heterogeneous 0.519 0.416 dense – sparse 0.492 0.342 edgy – flowing 0.465 0.376 static – dynamic 0.403 0.383 near – far 0.252 0.249 *nine subjects who took part in the elicitation process Grill and Flexer: Visualization of perceptual qualities in textural sounds 10 ICMC 2012, Ljubljana/Slovenia
Pearson correlation between constructs Grill and Flexer: Visualization of perceptual qualities in textural sounds 11 ICMC 2012, Ljubljana/Slovenia
Pearson correlation between constructs Grill and Flexer: Visualization of perceptual qualities in textural sounds 12 ICMC 2012, Ljubljana/Slovenia
Visualizing sounds in a collection • Representation of properties of individual sounds ➡ Auditory characteristics • Representation of properties of the sound collection ➡ Clusters, similarities, dominating characteristics • Waveforms and sonograms? Grill and Flexer: Visualization of perceptual qualities in textural sounds 13 ICMC 2012, Ljubljana/Slovenia
Perceptual metaphors • Strong synesthesia ➡ very rare, asymmetric, individual • Weak synesthesia – cross-modal similarity Lawrence Marks: On Perceptual Metaphors. Metaphor and Symbolic Activity 11 (1), 39–66, 1996 Grill and Flexer: Visualization of perceptual qualities in textural sounds 14 ICMC 2012, Ljubljana/Slovenia
Cross-modal similarity Wolfgang Köhler, Gestalt psychology ,1929 Grill and Flexer: Visualization of perceptual qualities in textural sounds 15 ICMC 2012, Ljubljana/Slovenia
Visualization of perceptual qualities in textural sounds • Based on most relevant of the previously elicited personal constructs • Layout in two dimensions (screen compatibility) ➡ Tiled map : Iconic representation of individual sounds, map for structural aspects of the collection • Synesthesia-like mappings from auditory to visual domain Grill and Flexer: Visualization of perceptual qualities in textural sounds 16 ICMC 2012, Ljubljana/Slovenia
Transferring auditory qualities to the visual domain • high–low ⇨ brightness and color hue (bright yellow–dark red) • ordered–chaotic ⇨ regularity of elements (on grid–deviating from grid) • tonal–noisy ⇨ saturation (colorful–gray) • smooth–coarse ⇨ jaggedness of element outline (smooth–jagged) • homogeneous–heterogeneous ⇨ variation in color parameters (no variation–much variation) Grill and Flexer: Visualization of perceptual qualities in textural sounds 17 ICMC 2012, Ljubljana/Slovenia
high–low ordered– chaotic tonal– noisy smooth– coarse homogeneous– heterogeneous Grill and Flexer: Visualization of perceptual qualities in textural sounds 18 ICMC 2012, Ljubljana/Slovenia
Online survey: Visualization of textural sounds http://grrrr.org/test/texvis Grill and Flexer: Visualization of perceptual qualities in textural sounds 19 ICMC 2012, Ljubljana/Slovenia
Online survey: Visualization of textural sounds • Survey A: One of the five examples represents the current sound, the generation parameters of the other four examples are taken from a uniform random distribution • Survey B: One of the five examples represents the current sound, the other four represent other existing sounds of the pool Grill and Flexer: Visualization of perceptual qualities in textural sounds 20 ICMC 2012, Ljubljana/Slovenia
Evaluation: Survey B – Dependence on expertise voters / mean RMS error correctness group votes (random: 0,243) (random: 20%) non-musicians, ≥ 20 votes 19 / 876 0,178 33,9% classical musical training, 29 / 1570 0,163 40,0% ≥ 20 votes electronic music practice, 48 / 2811 0,137 45,2% ≥ 20 votes electronic music practice, good listening conditions, 36 / 2019 0,133 46,4% ≥ 20 votes Grill and Flexer: Visualization of perceptual qualities in textural sounds 21 ICMC 2012, Ljubljana/Slovenia
Evaluation: Pearson correlation selection to reference 1.0 homogeneous– -0.25 0.55 0.26 0.18 0.62 0.9 heterogeneous 0.8 tonal–noisy -0.11 0.37 0.55 0.69 0.20 0.7 reference 0.6 smooth–coarse -0.34 0.43 0.68 0.54 0.27 0.5 0.4 ordered–chaotic -0.27 0.65 0.43 0.36 0.54 0.3 0.2 high–low 0.71 -0.24 -0.33 -0.08 -0.22 0.1 0.0 high–low ordered–chaotic smooth–coarse tonal–noisy homogeneous– heterogeneous Survey B: electronic music practitioners, good listening conditions, ≥ 10 votes selected Grill and Flexer: Visualization of perceptual qualities in textural sounds 22 ICMC 2012, Ljubljana/Slovenia
Evaluation: Mean RMS error vs. decision time users=94, x-y correlation=-0.13 @ significance(p=0.05)=0.20 mean duration=13.83 (6.65) 0.35 0.30 avg RMS error 0.25 0.20 0.15 0.10 0.05 0 5 10 15 20 25 30 35 40 45 time per vote(s) Grill and Flexer: Visualization of perceptual qualities in textural sounds 23 ICMC 2012, Ljubljana/Slovenia
Recommend
More recommend
