Musical Structure Visualization Program Introduction Background - PowerPoint PPT Presentation

By Winnie Chan August 2007 a report on Musical Structure Visualization

Program • Introduction  Background  Motivations  Challenges • Definitions and Technology • Visualization Techniques • Discussion • Conclusions 2

Introduction • We hear music unconsciously • We listen to music consciously, which requires efforts to understand  Musical structure  Composition techniques • Such understanding is vital for music appreciation and comprehension 3

Pop Music Vs. Classical Music • Pop music is meant to be friendly • Classical music is seemingly sophisticated  It requires training to recognize structure and form of music  Untrained one may only be able to “feel” the music and the music is thus boring 4

The Music Being Discussed • Classical music  Also known as fine music, classics, …  Within the basic music history context, from Baroque period to 20 th century  Focuses on instrumental music (vocal music is put aside)  Simple definition: Strictly organized compositions (still very vague though) 5

Background • Realizing the musical structure is important to appreciate the music • Music expertise: Able to single out the musical elements by  Reading the score  Listening to the performance 6

Why Learning Music is Harder? • Fine art (paintings and sculptures) and literature have a physical, concrete objects that we can “see” • Music is abstract and dynamic:  It is difficult for unskilled ears to recognize complicated musical elements from multi-layered music  Time-varying nature makes comparisons and appreciation harder 7

Music is Unique • The lack of visual equivalence is what makes music different from non- auditory art forms • The only art form that is perceived with our hearing ability 8

Musical Scores / Sheet Music ( 乐谱 ) • The most common way to learn music • Contains objective notations  Reading scores is demanding  Beginners have to learn the basics of music theory to know the notations, and still a long way for in-depth understanding  It is tedious for amateurs to go through the score, those overwhelming technical details are useful for performance but not listening • In practice not convenient to get full score 9

Motivations • Learning curve of classical music is high; visual display should help • Structure of music is rarely visualized, most attempts visualize the sonic features like Windows Media Player • Multidimensional nature of music is not well addressed 10

Research Issues • How to create informative and insightful visualization which helps people with limited background to  See the sound visually  Understand the structure of music semantically  Identify the musical elements when listening to the performance 11

Challenges (1) – Defining the data structure • Seek a general structure that can capture the essential musical elements in various compositions • Decide what particular genre ( 类型 ) we work on and what the basic elements are 12

Challenges (2) – Choosing the analysis method • Automatically analyze MIDI file  Ensure this digital score is accurate  Derive algorithms to retrieve cognitive elements from the score (!)  Help audience to appreciate music qualitatively  Can be quantified artificially, but not sure if computation is possible  Be validated to be held universally • Manually analyze music literature  Extract useful information  Provide user-interaction for human-defined input 13

Challenges (3) – Designing effective encoding scheme • Find the best visual mapping (equivalence) for music elements • Combine these representations without much interference 14

Program • Introduction • Definitions and Technology  Sound and Music  Music Visualization • Visualization Techniques • Discussion • Conclusions 15

What is Sound? • Does not have specific meanings • Qualities:  Frequency (pitch)  Volume  Speed  Spatial position of sound source  (other physical qualities) 16

What is Music? • Is collection of organized sound • Conveys certain messages / emotions • Properties:  Timbre ( 音色 )  Rhythm ( 节奏 )  Chords ( 和弦 )  Texture ( 组织 , 结构 )  Tempo ( 速度 )  Harmony ( 和声 )  Dynamics ( 力度变化 )  … 17

Music Visualization • Visualizes the loudness and frequency spectrum of music, e.g.  Oscilloscope display on radio  Animated imagery rendered by player • Should be distinguished from musical structure visualization 18

Program • Introduction • Definitions and Technology • Visualization Techniques • Discussion • Conclusions 19

Arc Diagrams • Visualizes complex patterns of repetition by connecting a translucent arc between a pair of matching pair 20

Arc Diagrams (cont.) • Outcomes are aesthetically pleasing • Patterns identified may not correspond to musical repeated units  Theme, subject, motif ( 主题 , 主旋律 )  More specific Wagner’s leitmotif and Berlioz’s idée fix 21

Isochords • Visualizes chord structure, progression and voicing of MIDI • Applies Tonnetz (German word for tone-network) to place each chord type in a 2D space • Major contribution:  Proposes an animated display  Show various chord properties in the same representation 22

Constructing Isochords Demo Warum sollt ich mich denn grämen by J. S. Bach • Music note  dot • Dots of chord notes  Isochords geometry • Major / minor  upward / downward pointing (different chord types have different shapes) • Consonant notes ( 协和音 )  connected by edge • Chord progression  adjacent structure • Modulation ( 变调 )  color • Which octave ( 八度音阶 )  size of triangle vertices 23

Isochords Discussion • Chords are important in music analysis but not for listening purpose • The structure of music defined here is low-level about notes and chords  Constructive and structural materials vs. abstract and semantic structure • Most research deal with these low- level details for music theory training 24

ImproViz • It shows jazz improvisations ( 即兴创作 ) • Melodic landscape shows contour of a melodic line • Harmony palette shows use of chords • Whole design is solely based on All Blues by Miles Davis • Results are crafted in Adobe Illustrator • Input is some kind of real sheet music 25

ImproViz Example 26

ImproViz Discussion • It visualizes the improvisations recorded as a score, not the true impromptu • It seems to attack a new problem (jazz improvisations) but the techniques are actually fairly general • Melodic contour is effective 27

Graphic Scores • Are made for two electro-acoustic compositions • Are not generalized • Do not have any solid visual mapping 28

Suggestions from Graphic Scores • Traditional musical score is written for performance, not analysis • A score is a graphic description of sonic events and many structural details are often missed • To make an effective study score for listeners:  Simple but visually identifiable sonic events (qualitative)  Temporal logic should match spacial logic (time-varying)  Full score should be visible at a glance (overview)  Score reading is not the most important  Study scores are for listening (ears), not for analysis 29

Visual Music • Done by the author of graphic scores • Appeared in SIGGRAPH posters and sketches:  Sonic map  Time slice / Heterophonic map • Map color to sound, and vice versa 30

Simplified Scores • Show whole score at once • Target at users with limited background • Map:  Measure ( 小节 )  vertical line  No. of notes  brightness  Dynamics ( 力度变化 )  width • Applied fisheye • Were not formally evaluated 31

Performance Expression Visualization • It deals with the expressions brought by different performance • Input is augmented scores with expressive attributes that do not appear in the original musical scores • It visualizes the depth of performance • Performance is described by cognitive terms: melody, rhythm and phasing 32

More on Performance Visualization • Performance visualization displays qualitative musical characteristics • MIDI parameters do not have music sense connected to human perception • It is not necessary to render the original scores faithfully – should be abstract for comprehending the underlying music and structure 33

Performance Visualization (1) – Vertical Bar Display Grid • Time  x -axis • Relative dynamics  y -axis • Local tempo variation (difference between written and performed)  interval between two grid lines • Note played  bar 34

Performance Visualization (1) – Vertical Bar Display Mapping • Dynamics  height of bar • Articulation ( 弹奏技巧 )  width of bar Connected: legato ( 连奏 ) Discrete: staccato ( 断奏 ) • Expressiveness of note  gray scale of bar • Player’s phasing  repeated patterns 35

Performance Visualization (1) – Vertical Bar Display User Studies • Users were asked to match performances to graphical displays • Results not satisfactory as user- interaction was were limited  No visual clues for users to trace the performance when listening to the music 36