Austrian Research Institute for Artificial Intelligence (OFAI) Thomas Grill Perceptually informed organization of textural sounds
Research context • Towards automatic annotation of electroacoustic music (2008–2010) • Audiominer – Mathematical signal analysis and modeling for manipulation of sound objects (2010–2013) • Automatic segmentation, labelling, and characterization of audio streams (2013–2016) Thomas Grill: Perceptually informed organization of textural sounds 2
Towards automatic annotation of electroacoustic music (2008–2010) • Annotation of un-notated music • Manual vs. automatic annotation • The system of spectromorphology • Structural overview and detection of representative sound groupings John Chowning: Turenas Smalley D.: Klien V., Grill T., Flexer A.: On automated annotation of acousmatic music, Journal of New Music Research, 2012 Thomas Grill: Perceptually informed organization of textural sounds 3
Audiominer (2010–2013) • Cooperation with NUHAG • Variations on the Gabor transform (aka STFT) – invertible CQ-NSGT • Structured sparsity Holighaus N., Dörfler M., Velasco G. A. and Grill T.: A framework for invertible, real-time constant-Q transforms, IEEE Transactions on Audio, Speech and Language Processing , 2013. Thomas Grill: Perceptually informed organization of textural sounds 4
Automatic segmentation, labelling, and characterization of audio streams (2013–) • Just started • Segmentation and characterization algorithms with a general applicability (perceptually informed) • Improved relation between semantic descriptions and measurable quantities • Interesting data available, partly annotated Thomas Grill: Perceptually informed organization of textural sounds 5
Outline • Why / what are textural sounds? • How can textural sounds be described? • How can (textural) sounds be organized? • How can textural sounds and collections thereof be visualized? Thomas Grill: Perceptually informed organization of textural sounds 6
Textural sounds • Activity sounds: chip, sweep, rustle, typing, scroop, rasp, crumple, clap, rub, walking • Machine sounds: buzz, whir, hammer, grumble, drone, traffic • Natural sounds: fire, water (rain, waterfall, ocean), wind • Animal sounds: sea gulls, crickets, humming • Human utterances: babble, chatter G. Strobl, G. Eckel and D. Rocchesso. “Sound Texture Modeling: A Survey”. Proceedings of the 2006 Sound and Music Computing (SMC) International Conference . Thomas Grill: Perceptually informed organization of textural sounds 7
Examples of textural sounds 12800 6400 frequency [Hz] 3200 1600 800 400 200 100 0.0 0.5 1.0 1.5 2.0 2.5 time [s] Thomas Grill: Perceptually informed organization of textural sounds 8
Examples of textural sounds 12800 6400 frequency [Hz] 3200 1600 800 400 200 100 0.0 0.5 1.0 1.5 2.0 2.5 time [s] Thomas Grill: Perceptually informed organization of textural sounds 9
2.2 Working Definition of Sound Textures First Time Constraint: Constant Long-term Characteristics A definition for a so~und texture could be quite wide, but we chose to restrict our working definition for many perceptual and conceptual reasons. First of all, there is no consensus among people as to what a sound texture might be; and more people will accept sounds that fit a more restrictive definition. The first constraint WC! put on our definition of a sound textures is that it should exhibit similar characteristics over time; that is, a two-second snippet of a texture should not differ significantly from another two-second snippet. To use another metaphor, one could say that any two snippets of a sound texture seem to be cut from the same rug [RIC79]. A sound texture is like wallpaper: it can have local structure and randomness, but the characteristics of the structure and randomness must remain constant on the large scale. This means that the pitch should not change like in a racing car, the rhythm should not increase or decrease, etc. This constraint also means that sounds in which the attack plays a great part (like many timbres) cannot be sound textures. A sound texture is characterized by its sustain. Figure 2.2.1 shows an interesting way of segregating sound tex- tures from other sounds, by showing how the “potential information content” increases with time. “Information” is taken here in the cog- nitive sense rather then the information theory sense. Speech or Definition attempt music can provide new information at any time, and their “potential information content” is shown here as a continuously increasing function of time. Textures, on the other hand, have constant long term characteristics, which translates into a flattening of the potential A sound texture is like wallpaper: it can have local structure information increase. Noise (in the auditory cognitive sense) has and randomness, but the characteristics of the structure and somewhat less information than textures. randomness must remain constant on the large scale. FIGURE 2.2.1 Potential Information Content of A Sound Texture vs. Time speech music content sound texture noise b time Saint-Arnaud, N. (1995). Classification of sound textures. Master’s thesis, MIT Media Lab, Cambridge, MA, USA Sounds that carry a lot of meaning are usually perceived as a message. The semantics take the foremost position in the cognition, Thomas Grill: Perceptually informed organization of textural sounds 10 downplaying the characteristics of the sound proper. We choose to work with sounds which are not primarily perceived as a message. Chapter 2 Human Perception of Sound Textures 24
Sound-based music and textural sound • Sound-based music: “art form in which the sound and not the musical note is the basic unit.” ➡ Acousmatic music and soundscape composition • Textural sound as "sound material" Landy, L. (2007). Understanding the Art of Sound Organization. The MIT Press, Cambridge, MA, USA. Truax, B. (2008). Soundscape composition as global music: Electroacoustic music as soundscape. Organised Sound, 13(2):103–109. Thomas Grill: Perceptually informed organization of textural sounds 11
Low Frequency Orchestra plays Robert Lettner: Das Spiel vom Kommen und Gehen Thomas Grill: Perceptually informed organization of textural sounds 12
Describing sounds • Predominant scheme: Semantic tagging (sound origin, recording context, etc.) • Sonic qualities are equally important/interesting, especially for abstract sounds or use in sound design etc. • Description ⇨ Organization Thomas Grill: Perceptually informed organization of textural sounds 14
Identification of perceptual qualities in textural sounds • What are the most significant qualities of textural sounds? ➡ Repertory grid technique used to elicit qualities ( personal constructs ) "ex nihilo", for a specific selection of subjects (interviewees) and objects under examination ( items ) • Interviewees (subjects) are asked to name differences between two randomly chosen sound examples ➡ Bipolar qualities spanning range from one sound to the other Grill, Flexer and Cunningham. Identification of perceptual qualities in textural sounds using the repertory grid method. Proceedings of the 6th Audio Mostly Conference , 2011 Thomas Grill: Perceptually informed organization of textural sounds 15
Example • Straight differentiation : In which ways do two sounds differ? • Triads : Group three objects to form two groups, then name differences between groups Thomas Grill: Perceptually informed organization of textural sounds 16
Repertory Grid for sounds • Elicitation of ~10 bipolar constructs per subject • Subjects rate all 20 sounds (grades 1 to 5) using own personal constructs 1 … 5 motion textural impulse high excentric evolutionary well-defined regular narrative pitched smooth static coherent continuous low contained repetitive diffused irregular static non-pitched porous A 4 4 4 1 2 4 4 2 4 3 3 B 5 3 5 5 5 1 3 1 5 2 1 C 4 5 2 2 4 -5 5 3 5 5 4 D 4 2 5 4 3 4 4 3 4 2 3 E 2 4 1 1 2 4 1 5 5 3 5 F 1 1 2 2 2 -3 2 5 5 4 5 G 5 5 5 5 5 2 1 2 5 1 1 H 4 3 3 1 2 5 1 1 5 2 4 I 4 2 2 2 2 5 2 2 4 1 4 J 2 1 5 3 1 -2 5 5 3 5 3 K 5 2 4 4 4 4 3 1 5 4 2 L 1 1 1 3 1 -2 1 5 5 5 5 M 4 5 5 1 2 2 3 2 5 3 2 N 3 1 4 4 1 4 4 5 5 4 2 O 4 2 4 3 3 -3 5 4 3 5 3 P 2 2 3 3 3 4 5 3 5 5 4 Q 5 5 5 3 5 -5 1 1 5 1 1 R 3 3 4 2 3 2 2 3 4 2 3 S 2 2 5 2 3 4 4 4 2 3 2 T 1 1 4 4 1 4 3 2 3 5 2 Thomas Grill: Perceptually informed organization of textural sounds 17
high/low • 16 subjects • expert listeners • 202 constructs • mostly German ordered/chaotic Thomas Grill: Perceptually informed organization of textural sounds 18
http://grrrr.org/test/classify Thomas Grill: Perceptually informed organization of textural sounds 19
Recommend
More recommend
