Visualizing the Semantic Structure in Classical Music Works Winnie - - PowerPoint PPT Presentation

visualizing the semantic structure in classical music
SMART_READER_LITE
LIVE PREVIEW

Visualizing the Semantic Structure in Classical Music Works Winnie - - PowerPoint PPT Presentation

Visualizing the Semantic Structure in Classical Music Works Winnie Wing-Yi Chan Supervised by Dr. Huamin Qu MPhil Thesis Defense Examination Department of Computer Science and Engineering The Hong Kong University of Science and Technology


slide-1
SLIDE 1

Visualizing the Semantic Structure in Classical Music Works

Winnie Wing-Yi Chan

Supervised by Dr. Huamin Qu MPhil Thesis Defense Examination Department of Computer Science and Engineering The Hong Kong University of Science and Technology January 13, 2009

slide-2
SLIDE 2

Outline

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion 2

slide-3
SLIDE 3

Outline

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion 3

  • Background
  • Motivations
  • Contributions
slide-4
SLIDE 4

Background

 Classical music is generally less friendly and approachable

 Realizing structure is difficult for unskilled ears

 Learning with musical scores and essays are demanding 4

The first theme begins presented in a minor key by the violins. Then part of the first theme is repeated by the violins. The transition begins in the violins; it contains a short motive and many scalewise passages. The second theme is presented by the violins and woodwinds. The second theme is repeated by the woodwinds and violins. The music moves on to some transitional material. A motive from the first theme is played by the clarinet and bassoon. The codetta begins with rapidly moving notes played by the violins. The codetta concludes with a solid chord. The development section begins with two short chords and notes played by the woodwinds. A portion of the first theme is played by the violins three times in sequence, each time one note lower than before. A portion
  • f the first theme is played by the lower strings as the upper strings play a
contrasting line of rapidly moving notes. The upper and lower strings trade what they play. The upper and lower strings again trade what they play. The violins continue playing portions of the first theme. The motive from the first theme alternates quietly several times between the violins and the flute. Fragments of the first theme are exchanged more energetically between the upper and lower strings. A short link leads to the recapitulation. The first theme returns played by the violins in the tonic key. The first theme continues. Part of the first theme is repeated by the violins. The transition begins… (book covers courtesy to amazon)
slide-5
SLIDE 5

Motivations

 Visualizing the semantic structure of music

 Impress with its sophistication and beauty  Reveal underlying musical structure  Assist music learning and teaching

 Previous approaches

 Rarely on semantic structure for perceptual understanding  Mainly on sonic features or low-level physical properties

5

music visualizers examples
slide-6
SLIDE 6

Contributions

 We pioneer a visualization solution to convey the semantic

structure in classical music

 General users

 Understanding the semantic structure  Appreciating the sophistication of music

 Music teachers, music students, and musicians

 Teaching and learning different styles  Gaining new insights into features

6

slide-7
SLIDE 7

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion

Outline

7

  • Musical structure
  • Abstracted scores
  • Performance expression
  • 3D music
  • Commercial products
  • Computer music
slide-8
SLIDE 8

Musical Structure Visualization: Arc Diagrams [InfoVis’02]

 Extract repeated patterns from a MIDI file automatically  May not be musically meaningful  Show repetitions of only one track 8

slide-9
SLIDE 9

Musical Structure Visualization: Isochords [GI’07]

 Visualize chord structure and progression with animation  Show low-level notes and chords 9

Tonnetz

slide-10
SLIDE 10

Musical Structure Visualization: ImproViz [CHI’05]

 Shown by a melodic landscape and harmony palette  Designed for one composition manually 10

slide-11
SLIDE 11

Abstracted Scores

11  Graphic Scores [EMS’05]  Visual Music [SIGGRAPH’05 Poster]  Simplified Scores [CW’02]

slide-12
SLIDE 12

Performance Expression Visualization

 Understand music performance in cognitive terms  Suggest that MIDI parameters do not link to human perception 12 Vertical Bar Display [AVI’04] Chernoff Faces [Vis’02] comp-i [SIGGRAPH’03 Sketches] Music & Emotion [ICME’04]

slide-13
SLIDE 13

3D Music Visualization

13 SIGGRAPH’03 Sketches & Applications ICMC’98 VSMM’01 CG&A’07

slide-14
SLIDE 14

Commercial Products

14 TimeSketch Music Animation Machine Hyperscore

slide-15
SLIDE 15

Visualization in Computer Music

15 Self-similarity grid for waveform [ICMC’01] Harmonic visualization [ICMC’01] Self-organization map for tonal content [CIE’05] Spiral layout for tonal evolution [CIE’05]

slide-16
SLIDE 16

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion

Outline

16

  • Formulation
  • Definitions
  • Uniqueness
slide-17
SLIDE 17

Problem Formulation

17

2) Micro-relationship

Horizontal foundation across time (breadth)

Themes: Recurring main melodies

Repetitions: Balance in time

Variations: Interesting details

1) Macro-relationship

Vertical texture among layers (depth)

Collaborate

Accompany

Blend

Contrast

3) Macro-micro relationship

Interactions between layers and themes

A layer plays some themes

A theme is played by some layers

slide-18
SLIDE 18

Defining the Data: Macro-Level

18

Layers

 Musically known as parts or voices  Generic view of “instrument tracks”  Determined by musical significance, not physical section

Layer Roles

 Functional role of a layer in the harmony of all at a time  Change over time  Shared by layers in the same functional group  Multiple roles at the same time possible  Not specified for transitional materials

slide-19
SLIDE 19

Defining the Data: Micro-Level

19

Themes

 Main melodies and musical ideas  Also known as motifs and subjects  Only a few in a composition  Repeated to bring unity and contrasted to introduce variety

Theme Variations

 Melodic, rhythmic, instrumental, and transformative means  May reoccur at different layers

slide-20
SLIDE 20

Data Uniqueness

Macro-micro

  • Macro layer roles vs.

micro theme variations

Micro

  • Repeated and varied

themes across time

Macro

  • Interrelated layers

with different roles

20  Different levels of viewpoints  Coordinated relationships

slide-21
SLIDE 21

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion

Outline

21

slide-22
SLIDE 22

Data Preprocessing

 No existing algorithm to parse high-level structure  Manually retrieve from essays  Front-end processing can be automated, but is beyond the scope

  • f visualization

22

Processed Data

Beginning and ending beat Instruments involved Layer roles and theme variations

Processing Beethoven’s Symphony No.5, first movement

 2 days without computer

assistance

 At most a few hours with

simple user interface

slide-23
SLIDE 23

Outline

23  Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion

  • Design criteria
  • Color scheme for layers
  • Theme glyph grammar
slide-24
SLIDE 24

Design Criteria

  • Avoid confusion
  • Assign basic visual channels carefully

Consistent

  • Be comprehended rapidly without training
  • Introduce some familiar metaphor

Intuitive

  • Gain insights into the structure effectively

Effective

  • Reflect both artistry and complexity

Aesthetic

24

slide-25
SLIDE 25

Design Overview

 A color scheme for layers

 More flexible than instrument icons

 A theme glyph grammar with typographical concepts

 Conveys semantic meanings of the variations

25

Woodwind: Brass: Percussion: Strings:

augmented C

theme

2 1 A B

form

diminished inverted

transformation

lengthened shortened

duration

echoed

  • diff. key

transposed

singleton
slide-26
SLIDE 26

Color Scheme for Layers: Music and Color

26

  • Quality of a sound
  • Depending on performance style

Tone Color

  • Emotions invoked
  • Human perception of sound

Psychology

  • Mapping between sound and color spectrum
  • Rational but lacks perceptual justification

Physics

None is relevant to semantic structure

slide-27
SLIDE 27

Color Scheme for Layers: Proposed Solution

 Hue: physical color of instrument families  Lightness: pitch range  Percussions in an outlier color  Non-orchestral layers based on users' preferences 27

Woodwind:

flute

  • boe

clarinet bassoon

Brass:

horn trumpet trombone tuba

Percussion:

triangle snare drum timpani bass drum

Strings:

violin viola cello double bass

slide-28
SLIDE 28

Theme Glyph Grammar: Glyph Candidates

 No universal design which is the best to represent theme,

depending on visualization purposes

28

#b

slide-29
SLIDE 29

Theme Glyph Grammar: Typographical Grammar Rules

Duplicate a sub-part of the glyph

  • Feasible as number of themes is usually limited

Change the core ‘x-height’ shape

  • Categorical attribute
  • Overall appearance is preserved with obvious changes

Adjust the ascender or descender part

  • Changes to rhythmic pattern or partial appearance
  • One-dimensional variation

Transform the core ‘x-height’ part

  • A theme is structurally transformed into other shapes
  • Only modify ‘x-height’ part to remain similar and consistent

Add indicative decorations

  • Special variations
  • Indicator can cope with more information if available

29

theme 2 1 C A B form lengthened shortened duration augmented diminished inverted transformation echoed

  • diff. key

transposed

singleton x-height ascender type size

typography

slide-30
SLIDE 30

Theme Glyph Grammar: Typography in Our Framework

 How to size and modify type glyphs

 Content clearly defined  Visual impact formally applied  Context to be delivered maximized

 Proposed grammar can be generalized to any glyph designs for

  • ptimal visual impact and legibility

30

xdpH

x-height ascender descender

capline meanline baseline

part of a lower case letter that equals height of ‘x’

type size

slide-31
SLIDE 31

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion

Outline

31

  • Layer braid
  • Theme fabric
slide-32
SLIDE 32

Visualization Prototypes

32  Macro-relationships among layers  Micro-relationships of theme occurrences  Macro-micro relationships between layer roles and theme variations

Layers Themes

slide-33
SLIDE 33

Textile Art Metaphor

 Intuitive and efficient with familiar metaphor  Similarities with musical structure

 Delicate cloth as art piece  Structure along x- and y-axis  Detailed textile pattern

33

(source)
slide-34
SLIDE 34

Layer Braid: Overview

Threads

 Respective layer colors  Interact and correlate over time 34

ensemble contrast collaborate accompany dominant

flute violin II viola cello double bass clarinet bassoon
  • boe
violin I horn I horn II
slide-35
SLIDE 35

Layer Braid: Plaiting Process

 Layer threads follow the musical score order  Braid patterns determined by interweaving positions in 2D  Layers of the same functional group are plaited together  Multiple roles: spread, plait, and rejoin  Threads may cross unwanted roles during interpolation

 May darken segments without specific role

35

layer roles time

A B C

t

(source)
slide-36
SLIDE 36

Theme Fabric: Arranging Glyph Nodes

 Overlay theme glyphs on a musical score layout

 x-position: when the theme is played  y-position: which layer plays the theme

36 layer 1 layer 2 layer 3 time

slide-37
SLIDE 37

Theme Fabric: Stitching Styles

 Repetitions of glyphs not obvious without visual clues  Stitch glyphs in various textile art styles 37

plain style bundled style collapsed style decreasing levels of detail of micro-relationships

slide-38
SLIDE 38

Theme Fabric: Plain Style

 Connect same glyphs in orthogonal direction

 Horizontal curved weft  Vertical straight warp

 Utilize color and lightness of threads 38

warp weft plain weaving

slide-39
SLIDE 39

Theme Fabric: From Plain Style to Bundled Style

 Replicated horizontal threads

 Bundle horizontal threads  Remove redundant vertical threads

 Maintain betweenness centrality of glyphs 39

butterfly stitches

slide-40
SLIDE 40

Theme Fabric: Bundled Style

 Reveal more high-level thematic structure

 Same glyphs across layers are visually clustered  Structure is outlined by thread bundles and knots

 Thread blending: address overlapping  Thread patterns: highlight special glyph groups 40

slide-41
SLIDE 41

Theme Fabric: From Bundled Style to Collapsed Style

 Collapse identical glyphs at different

layers into one single glyph

 Cluster threads into arcs  Become some Arc Diagrams layout 41

1 2 3 4

drop-stitch

slide-42
SLIDE 42

Theme Fabric: Collapsed Style for Sole Micro View

42

Rainbow arc for layers Split for former and latter glyph Specific arc height for every unique glyph

slide-43
SLIDE 43

Theme Fabric: Collapsed Style for Macro-Micro View

43

Rainbow arc for layers Split for former and latter glyph Specific arc height for each layer roles

layer role A layer role B

slide-44
SLIDE 44

Theme Fabric: Collapsed Style for Macro-Micro View

 Combine arc height with layer roles in layer braid  Duplicate arcs for multiple roles 44

slide-45
SLIDE 45

Theme Fabric: Summary

45

Plain

  • Most expanded and detailed

Bundled

  • Connects same glyphs with thread bundles

Collapsed

  • Most abstract with compressed layers
  • Macro-micro with arc heights for layer roles
slide-46
SLIDE 46

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion

Outline

46

slide-47
SLIDE 47

User Interactions

Customization Configure layer colors and theme glyph type Zooming and Panning Less flavored than focus + context Selection Select any visual entities on screen Filtering Adjust the brightness or opacity value Linking and Brushing Brush to explore different modules Link different views by overlaying them Abstraction Render in different levels of detail Aggregate small changes in data Focus + Context View with bifocal display Details on Demand Show more details when zoomed in Synchronization with Music Animate with music

47

slide-48
SLIDE 48

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion

Outline

48

  • Macro-relationship
  • Theme and variations
  • Macro-micro confluence
slide-49
SLIDE 49

Macro-Relationships Mozart's Symphony No. 40, 1st Mov’t

 Development exhibits highly complicated relationships  Ensembles are only heard once at the end of exposition 49

flute violin II viola cello double bass clarinet bassoon
  • boe
violin I horn I horn II

exposition development recapitulation

slide-50
SLIDE 50

Macro-Relationships Mozart's Symphony No. 40, 1st Mov’t

 Violins are mostly dominant  Bassoon becomes dominant more frequently 50

flute violin II viola cello double bass clarinet bassoon
  • boe
violin I horn I horn II flute violin II viola cello double bass clarinet bassoon
  • boe
violin I horn I horn II
slide-51
SLIDE 51

Theme and Variations Mozart's Symphony No. 40, 1st Mov’t

 Blue threads (first subject) are usually played by strings  Vivid variations are seen in development and recapitulation 51

flute violin II viola cello double bass clarinet bassoon
  • boe
violin I horn I horn II Not associated with any theme:

exposition exposition (repeated) development recapitulation

slide-52
SLIDE 52

Macro-Micro Confluence Beethoven’s Symphony No. 5, 1st Mov’t

 The fate motif never dominates the music  The inverted, shortened motif in the lower strings contrasts the

music at macro-level

52

slide-53
SLIDE 53

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion

Outline

53

  • User study
  • User survey
slide-54
SLIDE 54

User Study

 Novice users gain insights promptly without prior training  A pictorial guidance for music learning and appreciation 54 Accuracy Time (min) Prototypes Mean SD Mean SD Layer Braid 86% 12% 4.17 1.68 Theme Fabric (Bundled) 95% 9% 5.13 0.73 Theme Fabric (Collapsed) 89% 15% 3.67 1.83 Details

10 college students

12 questions

5-min introduction

slide-55
SLIDE 55

User Survey

 Braid and fabric are excellent metaphor of music  Layer braid is useful as the layer role concept is abstract  Bundled theme fabric is preferred by advanced users  Collapsed style is concise and aesthetic  Prototypes cooperate with or complement one another 55 10 Experts

7 music students

2 music educators

1 musician

slide-56
SLIDE 56

Expert Feedback Highlights

Educators

  • Reading essays diminishes learning interest
  • Layer information is important

Students

  • Layer feature helps differentiate instruments
  • Comparing compositions become easier

Musician

  • Audience find classical music more approachable
  • Listeners grow from appreciation to learning

56 Visualizing music without clear form? … design highly generalized.

Improvements

 Include more attributes, e.g. dynamics, tempo, tone color  Add descriptive texts

slide-57
SLIDE 57

 Introduction  Related Work  The Musical Structural Data  Data Preprocessing  Design Principles  Visualization Prototypes  User Interactions  Case Studies  Evaluation  Conclusion

Outline

57

  • Demo
  • Summary
  • Conclusion
  • Future Work
slide-58
SLIDE 58

Demo

58

slide-59
SLIDE 59

Summary

 Innovative visualization solution for semantic structure in music

 Macro-level layer relationships  Micro-level theme variations  Macro-micro interactions between layers and themes

 Design principles

 Color scheme for layers  Glyph grammar founded on typography

 Visualization prototypes derived from textile art

 Layer braid  Theme fabric (plain, bundled and collapsed style)

 Representative case studies  Encouraging user evaluation 59

slide-60
SLIDE 60

Conclusion

 Apply information visualization to other discipline

 Strong human factors  Artistic sensation

 Introduce art and design analogies to address perceptual issues

 Typography  Textile art metaphor

 Generalize the techniques for applications with semantic context

 News visualization  Story-telling visualization

60

slide-61
SLIDE 61

Future Work

 Visualize more compositions to fully realize its potential  Implement a simple user-interface for data input  Seek the possibility of using the visualization as teaching tool 61

slide-62
SLIDE 62

Publications

 Wing-Yi Chan, Huamin Qu, Wai-Ho Mak. “Visualizing the

Semantic Structure in Classical Music Works”. Accepted to IEEE Transactions on Visualization and Computer Graphics (TVCG) with minor revision.

 Huamin Qu, Wing-Yi Chan, Anbang Xu, Kai-Lun Chung, Kai-Hon

Lau, Ping Guo. “Visual Analysis of the Air Pollution Problem in Hong Kong”. IEEE Transactions on Visualization and Computer Graphics (TVCG), vol.13, no. 6, pp.1408-1415, 2007.

62

slide-63
SLIDE 63

Q & A

63

… Thank you!

After silence, that which comes nearest to expressing the inexpressible is music.

  • Aldous Huxley, “Music at Night”, 1931
flute violin II viola cello double bass clarinet bassoon
  • boe
violin I horn I horn II flute violin II viola cello double bass clarinet bassoon
  • boe
violin I