[PPT] - Visualizing the Semantic Structure in Classical Music Works Winnie PowerPoint Presentation

SLIDE 1

Visualizing the Semantic Structure in Classical Music Works

Winnie Wing-Yi Chan

Supervised by Dr. Huamin Qu MPhil Thesis Defense Examination Department of Computer Science and Engineering The Hong Kong University of Science and Technology January 13, 2009

SLIDE 2

Outline

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion 2

SLIDE 3

Outline

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion 3

Background
Motivations
Contributions

SLIDE 4

Background

 Classical music is generally less friendly and approachable

 Realizing structure is difficult for unskilled ears

 Learning with musical scores and essays are demanding 4

The first theme begins presented in a minor key by the violins. Then part of the first theme is repeated by the violins. The transition begins in the violins; it contains a short motive and many scalewise passages. The second theme is presented by the violins and woodwinds. The second theme is repeated by the woodwinds and violins. The music moves on to some transitional material. A motive from the first theme is played by the clarinet and bassoon. The codetta begins with rapidly moving notes played by the violins. The codetta concludes with a solid chord. The development section begins with two short chords and notes played by the woodwinds. A portion of the first theme is played by the violins three times in sequence, each time one note lower than before. A portion

f the first theme is played by the lower strings as the upper strings play a

contrasting line of rapidly moving notes. The upper and lower strings trade what they play. The upper and lower strings again trade what they play. The violins continue playing portions of the first theme. The motive from the first theme alternates quietly several times between the violins and the flute. Fragments of the first theme are exchanged more energetically between the upper and lower strings. A short link leads to the recapitulation. The first theme returns played by the violins in the tonic key. The first theme continues. Part of the first theme is repeated by the violins. The transition begins… (book covers courtesy to amazon)

SLIDE 5

Motivations

 Visualizing the semantic structure of music

 Impress with its sophistication and beauty  Reveal underlying musical structure  Assist music learning and teaching

 Previous approaches

 Rarely on semantic structure for perceptual understanding  Mainly on sonic features or low-level physical properties

5

music visualizers examples

SLIDE 6

Contributions

 We pioneer a visualization solution to convey the semantic

structure in classical music

 General users

 Understanding the semantic structure  Appreciating the sophistication of music

 Music teachers, music students, and musicians

 Teaching and learning different styles  Gaining new insights into features

6

SLIDE 7

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion

Outline

7

Musical structure
Abstracted scores
Performance expression
3D music
Commercial products
Computer music

SLIDE 8

Musical Structure Visualization: Arc Diagrams [InfoVis’02]

 Extract repeated patterns from a MIDI file automatically  May not be musically meaningful  Show repetitions of only one track 8

SLIDE 9

Musical Structure Visualization: Isochords [GI’07]

 Visualize chord structure and progression with animation  Show low-level notes and chords 9

Tonnetz

SLIDE 10

Musical Structure Visualization: ImproViz [CHI’05]

 Shown by a melodic landscape and harmony palette  Designed for one composition manually 10

SLIDE 11

Abstracted Scores

11  Graphic Scores [EMS’05]  Visual Music [SIGGRAPH’05 Poster]  Simplified Scores [CW’02]

SLIDE 12

Performance Expression Visualization

 Understand music performance in cognitive terms  Suggest that MIDI parameters do not link to human perception 12 Vertical Bar Display [AVI’04] Chernoff Faces [Vis’02] comp-i [SIGGRAPH’03 Sketches] Music & Emotion [ICME’04]

SLIDE 13

3D Music Visualization

13 SIGGRAPH’03 Sketches & Applications ICMC’98 VSMM’01 CG&A’07

SLIDE 14

Commercial Products

14 TimeSketch Music Animation Machine Hyperscore

SLIDE 15

Visualization in Computer Music

15 Self-similarity grid for waveform [ICMC’01] Harmonic visualization [ICMC’01] Self-organization map for tonal content [CIE’05] Spiral layout for tonal evolution [CIE’05]

SLIDE 16

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion

Outline

16

Formulation
Definitions
Uniqueness

SLIDE 17

Problem Formulation

17

2) Micro-relationship

Horizontal foundation across time (breadth)



Themes: Recurring main melodies



Repetitions: Balance in time



Variations: Interesting details

1) Macro-relationship

Vertical texture among layers (depth)



Collaborate



Accompany



Blend



Contrast

3) Macro-micro relationship

Interactions between layers and themes



A layer plays some themes



A theme is played by some layers

SLIDE 18

Defining the Data: Macro-Level

18

Layers

 Musically known as parts or voices  Generic view of “instrument tracks”  Determined by musical significance, not physical section

Layer Roles

 Functional role of a layer in the harmony of all at a time  Change over time  Shared by layers in the same functional group  Multiple roles at the same time possible  Not specified for transitional materials

SLIDE 19

Defining the Data: Micro-Level

19

Themes

 Main melodies and musical ideas  Also known as motifs and subjects  Only a few in a composition  Repeated to bring unity and contrasted to introduce variety

Theme Variations

 Melodic, rhythmic, instrumental, and transformative means  May reoccur at different layers

SLIDE 20

Data Uniqueness

Macro-micro

Macro layer roles vs.

micro theme variations

Micro

Repeated and varied

themes across time

Macro

Interrelated layers

with different roles

20  Different levels of viewpoints  Coordinated relationships

SLIDE 21

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion

Outline

21

SLIDE 22

Data Preprocessing

 No existing algorithm to parse high-level structure  Manually retrieve from essays  Front-end processing can be automated, but is beyond the scope

f visualization

22

Processed Data

Beginning and ending beat Instruments involved Layer roles and theme variations

Processing Beethoven’s Symphony No.5, first movement

 2 days without computer

assistance

 At most a few hours with

simple user interface

SLIDE 23

Outline

23  Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion

Design criteria
Color scheme for layers
Theme glyph grammar

SLIDE 24

Design Criteria

Avoid confusion
Assign basic visual channels carefully

Consistent

Be comprehended rapidly without training
Introduce some familiar metaphor

Intuitive

Gain insights into the structure effectively

Effective

Reflect both artistry and complexity

Aesthetic

24

SLIDE 25

Design Overview

 A color scheme for layers

 More flexible than instrument icons

 A theme glyph grammar with typographical concepts

 Conveys semantic meanings of the variations

25

Woodwind: Brass: Percussion: Strings:

augmented C

theme

2 1 A B

form

diminished inverted

transformation

lengthened shortened

duration

echoed

diff. key

transposed

singleton

SLIDE 26

Color Scheme for Layers: Music and Color

26

Quality of a sound
Depending on performance style

Tone Color

Emotions invoked
Human perception of sound

Psychology

Mapping between sound and color spectrum
Rational but lacks perceptual justification

Physics

None is relevant to semantic structure

SLIDE 27

Color Scheme for Layers: Proposed Solution

 Hue: physical color of instrument families  Lightness: pitch range  Percussions in an outlier color  Non-orchestral layers based on users' preferences 27

Woodwind:

flute

boe

clarinet bassoon

Brass:

horn trumpet trombone tuba

Percussion:

triangle snare drum timpani bass drum

Strings:

violin viola cello double bass

SLIDE 28

Theme Glyph Grammar: Glyph Candidates

 No universal design which is the best to represent theme,

depending on visualization purposes

28

#b

SLIDE 29

Theme Glyph Grammar: Typographical Grammar Rules

Duplicate a sub-part of the glyph

Feasible as number of themes is usually limited

Change the core ‘x-height’ shape

Categorical attribute
Overall appearance is preserved with obvious changes

Adjust the ascender or descender part

Changes to rhythmic pattern or partial appearance
One-dimensional variation

Transform the core ‘x-height’ part

A theme is structurally transformed into other shapes
Only modify ‘x-height’ part to remain similar and consistent

Add indicative decorations

Special variations
Indicator can cope with more information if available

29

theme 2 1 C A B form lengthened shortened duration augmented diminished inverted transformation echoed

diff. key

transposed

singleton x-height ascender type size

typography

SLIDE 30

Theme Glyph Grammar: Typography in Our Framework

 How to size and modify type glyphs

 Content clearly defined  Visual impact formally applied  Context to be delivered maximized

 Proposed grammar can be generalized to any glyph designs for

ptimal visual impact and legibility

30

xdpH

x-height ascender descender

capline meanline baseline

part of a lower case letter that equals height of ‘x’

type size

SLIDE 31

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion

Outline

31

Layer braid
Theme fabric

SLIDE 32

Visualization Prototypes

32  Macro-relationships among layers  Micro-relationships of theme occurrences  Macro-micro relationships between layer roles and theme variations

Layers Themes

SLIDE 33

Textile Art Metaphor

 Intuitive and efficient with familiar metaphor  Similarities with musical structure

 Delicate cloth as art piece  Structure along x- and y-axis  Detailed textile pattern

33

(source)

SLIDE 34

Layer Braid: Overview

Threads

 Respective layer colors  Interact and correlate over time 34

ensemble contrast collaborate accompany dominant

flute violin II viola cello double bass clarinet bassoon

boe

violin I horn I horn II

SLIDE 35

Layer Braid: Plaiting Process

 Layer threads follow the musical score order  Braid patterns determined by interweaving positions in 2D  Layers of the same functional group are plaited together  Multiple roles: spread, plait, and rejoin  Threads may cross unwanted roles during interpolation

 May darken segments without specific role

35

layer roles time

A B C

t

(source)

SLIDE 36

Theme Fabric: Arranging Glyph Nodes

 Overlay theme glyphs on a musical score layout

 x-position: when the theme is played  y-position: which layer plays the theme

36 layer 1 layer 2 layer 3 time

SLIDE 37

Theme Fabric: Stitching Styles

 Repetitions of glyphs not obvious without visual clues  Stitch glyphs in various textile art styles 37

plain style bundled style collapsed style decreasing levels of detail of micro-relationships

SLIDE 38

Theme Fabric: Plain Style

 Connect same glyphs in orthogonal direction

 Horizontal curved weft  Vertical straight warp

 Utilize color and lightness of threads 38

warp weft plain weaving

SLIDE 39

Theme Fabric: From Plain Style to Bundled Style

 Replicated horizontal threads

 Bundle horizontal threads  Remove redundant vertical threads

 Maintain betweenness centrality of glyphs 39

butterfly stitches

SLIDE 40

Theme Fabric: Bundled Style

 Reveal more high-level thematic structure

 Same glyphs across layers are visually clustered  Structure is outlined by thread bundles and knots

 Thread blending: address overlapping  Thread patterns: highlight special glyph groups 40

SLIDE 41

Theme Fabric: From Bundled Style to Collapsed Style

 Collapse identical glyphs at different

layers into one single glyph

 Cluster threads into arcs  Become some Arc Diagrams layout 41

1 2 3 4

drop-stitch

SLIDE 42

Theme Fabric: Collapsed Style for Sole Micro View

42

Rainbow arc for layers Split for former and latter glyph Specific arc height for every unique glyph

SLIDE 43

Theme Fabric: Collapsed Style for Macro-Micro View

43

Rainbow arc for layers Split for former and latter glyph Specific arc height for each layer roles

layer role A layer role B

SLIDE 44

Theme Fabric: Collapsed Style for Macro-Micro View

 Combine arc height with layer roles in layer braid  Duplicate arcs for multiple roles 44

SLIDE 45

Theme Fabric: Summary

45

Plain

Most expanded and detailed

Bundled

Connects same glyphs with thread bundles

Collapsed

Most abstract with compressed layers
Macro-micro with arc heights for layer roles

SLIDE 46

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion

Outline

46

SLIDE 47

User Interactions

Customization Configure layer colors and theme glyph type Zooming and Panning Less flavored than focus + context Selection Select any visual entities on screen Filtering Adjust the brightness or opacity value Linking and Brushing Brush to explore different modules Link different views by overlaying them Abstraction Render in different levels of detail Aggregate small changes in data Focus + Context View with bifocal display Details on Demand Show more details when zoomed in Synchronization with Music Animate with music

47

SLIDE 48

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion

Outline

48

Macro-relationship
Theme and variations
Macro-micro confluence

SLIDE 49

Macro-Relationships Mozart's Symphony No. 40, 1st Mov’t

 Development exhibits highly complicated relationships  Ensembles are only heard once at the end of exposition 49

flute violin II viola cello double bass clarinet bassoon

boe

violin I horn I horn II

exposition development recapitulation

SLIDE 50

Macro-Relationships Mozart's Symphony No. 40, 1st Mov’t

 Violins are mostly dominant  Bassoon becomes dominant more frequently 50

flute violin II viola cello double bass clarinet bassoon

boe

violin I horn I horn II flute violin II viola cello double bass clarinet bassoon

boe

violin I horn I horn II

SLIDE 51

Theme and Variations Mozart's Symphony No. 40, 1st Mov’t

 Blue threads (first subject) are usually played by strings  Vivid variations are seen in development and recapitulation 51

flute violin II viola cello double bass clarinet bassoon

boe

violin I horn I horn II Not associated with any theme:

exposition exposition (repeated) development recapitulation

SLIDE 52

Macro-Micro Confluence Beethoven’s Symphony No. 5, 1st Mov’t

 The fate motif never dominates the music  The inverted, shortened motif in the lower strings contrasts the

music at macro-level

52

SLIDE 53

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion

Outline

53

User study
User survey

SLIDE 54

User Study

 Novice users gain insights promptly without prior training  A pictorial guidance for music learning and appreciation 54 Accuracy Time (min) Prototypes Mean SD Mean SD Layer Braid 86% 12% 4.17 1.68 Theme Fabric (Bundled) 95% 9% 5.13 0.73 Theme Fabric (Collapsed) 89% 15% 3.67 1.83 Details



10 college students



12 questions



5-min introduction

SLIDE 55

User Survey

 Braid and fabric are excellent metaphor of music  Layer braid is useful as the layer role concept is abstract  Bundled theme fabric is preferred by advanced users  Collapsed style is concise and aesthetic  Prototypes cooperate with or complement one another 55 10 Experts



7 music students



2 music educators



1 musician

SLIDE 56

Expert Feedback Highlights

Educators

Reading essays diminishes learning interest
Layer information is important

Students

Layer feature helps differentiate instruments
Comparing compositions become easier

Musician

Audience find classical music more approachable
Listeners grow from appreciation to learning

56 Visualizing music without clear form? … design highly generalized.

Improvements

 Include more attributes, e.g. dynamics, tempo, tone color  Add descriptive texts

SLIDE 57

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion

Outline

57

Demo
Summary
Conclusion
Future Work

SLIDE 58

Demo

58

SLIDE 59

Summary

 Innovative visualization solution for semantic structure in music

 Macro-level layer relationships  Micro-level theme variations  Macro-micro interactions between layers and themes

 Design principles

 Color scheme for layers  Glyph grammar founded on typography

 Visualization prototypes derived from textile art

 Layer braid  Theme fabric (plain, bundled and collapsed style)

 Representative case studies  Encouraging user evaluation 59

SLIDE 60

Conclusion

 Apply information visualization to other discipline

 Strong human factors  Artistic sensation

 Introduce art and design analogies to address perceptual issues

 Typography  Textile art metaphor

 Generalize the techniques for applications with semantic context

 News visualization  Story-telling visualization

60

SLIDE 61

Future Work

 Visualize more compositions to fully realize its potential  Implement a simple user-interface for data input  Seek the possibility of using the visualization as teaching tool 61

SLIDE 62

Publications

 Wing-Yi Chan, Huamin Qu, Wai-Ho Mak. “Visualizing the

Semantic Structure in Classical Music Works”. Accepted to IEEE Transactions on Visualization and Computer Graphics (TVCG) with minor revision.

 Huamin Qu, Wing-Yi Chan, Anbang Xu, Kai-Lun Chung, Kai-Hon

Lau, Ping Guo. “Visual Analysis of the Air Pollution Problem in Hong Kong”. IEEE Transactions on Visualization and Computer Graphics (TVCG), vol.13, no. 6, pp.1408-1415, 2007.

62

SLIDE 63

Q & A

63

… Thank you!

After silence, that which comes nearest to expressing the inexpressible is music.

Aldous Huxley, “Music at Night”, 1931

flute violin II viola cello double bass clarinet bassoon

boe

violin I horn I horn II flute violin II viola cello double bass clarinet bassoon

boe

violin I