Modelling music Dorien Herremans ISTD, Singapore University of Technology and Design IHPC, A*STAR Sound and Music Computing, NUS - 22.03.18 1
Outline • Why modelling music? • A brief early history of generation systems • Approaches to modelling music? • Rules — counterpoint • Machine learning: • Markov models • LSTMs • Word2vec • Conclusions 2
A brief early history of generation systems 3
Automatically generating music? • Muzikalisches Wurfelspiel (Mozart) • Actual origin: Der allezeit fertige Menuetten- und Polonaisencomponist (Johann Philipp Kirnberger, 1757) 4
Automatically generating music? • Other chance inspired compositions • John Cage's “Atlas Eclipticalis” • Charles Dodge's “The Earth's Magnetic Field” • Iannis Xenakis’ “Analogique A” (Markov) • Fractal music • … 5
Automatically generating music? “[The Engine's] operating mechanism might act upon other things besides numbers [. . . ] Supposing, for instance, that the fundamental relations of pitched sounds in the signs of harmony and of musical composition were susceptible of such expressions and adaptations, the engine might compose elaborate and scientific pieces of music of any degree of complexity or extent." — Ada Lovelace 1840 6
Automatically generating music? • 1957 Illiac Suite — Lejaren Hiller & Leonard Isaacson • Rule based + statistical models 7
Automatically generating music? • David Cope • Experiments in Musical Intelligence • 1981 Composer's block • Rules to model of his own style —> different composers • e.g. Chopin Nocturne 8
Automatically generating music? • The Continuator — Francois Pachet • Interactive system: “extend the technical ability of musicians with stylistically consistent, automatically learnt musical material” • Markov models (real time learning) • Flow machines http://www.francoispachet.fr/continuator/ 9
Automatically generating music? • Impro-Visor — Bob Keller (2005) • Probabilistic grammars, deep believe networks,. . . 10
What’s going on in the field? Problems • Generating melody, rhythm, timbre, harmony (chords),. . . • Constraining to a narrative (games, video, emotions,. . . ) • Interacting with live performer Approaches • Rule-based methods • Optimization approaches (genetic algorithms, constraint programming, local search, etc) • Statistical models (Markov models, factor oracles, deep learning, etc) Herremans, Chuan, Chew. 2017. A functional taxonomy of music generation systems. ACM Computing Surveys. 11
The rapid rise of digital music • In 2015: revenue from digital music: 6.4 billion USD = 45% of the global music industry • Asia: 14% of these revenues • Number of digital music users expected to grow 15% a year until 2020 12
New application opportunities 13
Approaches to modelling music 14
Rules and optimisation • Music theory • Counterpoint - 5th species: —> One of the most formally defined musical styles Rules written by Johan Fux in 1725 15
Rules and optimisation Composing music = combinatorial optimization problem • Find the right combination of notes • So that the piece sounds ‘good’, i.e. fits a style as well as possible 16
Optimization objective • 19 melodic and 19 harmonic rules e.g. Each large leap should be followed by stepwise motion in the opposite direction Penalty points for each rule violation —> Minimized 17
Finding best combination of notes • Exact algorithms • Metaheuristics • Population Based (genetic algorithms,. . . ) • Constructive (ant colony, GRASP , . . . ) • Search Based (tabu search, variable neighbourhood search,…) A metaheuristic is a high-level problem-independent algorithmic framework that provides a set of guidelines or strategies to develop heuristic optimization algorithms (Sörensen and Glover, 2014). 18
Finding best combination of notes • Local search with 3 types of "moves" 19
Finding best combination of notes 20
Result D. Herremans, K. Sorensen. 2013. Composing Fifth Species Counterpoint Music With A Variable Neighborhood Search 21 Algorithm. Expert Systems with Applications. 40(16):6427{6437.
From rules to machine learning • Limitation: rules need to be defined! Rule-based objective function Machine learned objective function Why still optimization? Because it allows us to: - Easily integrate constraints - Enforce structure - Correctly evaluate circles 22
Markov models • Trained on dataset • Transition probabilities • Issue with machine learning: plagiarism (Papadopoulo et al., 2014) • Different viewpoints (Conklin, 1995) How integrated in optimisation objective function? 23
High probability sequences? • No. • High probability can be very repetitive • Find the right mix of surprising notes • One approach: Objective function: Match an expectancy profile 24 D. Herremans, S. Weisser, K. Sorensen, and D. Conklin, 2015. Generating structured music using quality metrics based on Markov models. Expert Systems with Applications.
The future of music generation •Research started in the 1950s •Why don’t we listen to generated music on our phones? 2 Challenges: - How to control emotion in music? - Long-term structure D. Herremans, K. Sorensen. 2013. FuX, an Android app that generates counterpoint. Proceedings of IEEE Symposium on Computational 25 Intelligence for Creativity and Affective Computing (CICAC) . Singapore. 48-55.
MorpheuS music generation 2 Challenges: - How to control emotion in music? - Long-term structure 26
1. Modeling emotion: tension • Important for music in: • Video games • Film background • YouTube • Tension: many aspects: tonal , rhythmic, loudness etc. • First comprehensive model by Mary Farbood (2002) 27
1. Modeling emotion: tension • The Spiral Array: 3D geometrical mathematical model of tonality (Chew, 2012) • 3D representation of higher level musical entities in tonal space • Helix for pitch classes, chords and keys • Used in Key detection (Chuan & Chew, 2005), Chord detection, finding key boundaries (Chew, 2000), etc. —> 3 aspects of tonal tension • Tool available online Herremans D., Chew E.. 2016. Tension ribbons: Quantifying and visualising tonal tension. Second International Conference on 28 Technologies for Music Notation and Representation (TENOR). 2:8-18. Cambridge, UK.
Cloud diameter C major triad and the C diminished triad 29
Cloud momentum C major and C# major chord with their center of effect ( ce , in green). 30
Tensile strain (distance from the key) C major — C# major — A minor chord together with their distance from the ce of the key (in orange). 31
Tonal tension - Tristan chord • Wagner's opera ‘Tristan und Isolde’ • Bass note and augmented 4th, 6th and 9th 32
2. Long-term structure • We can generate good sounding short fragments of music • Guitar solos generated by Integer Programming • Short solo fragments Cunha N., Subramanian A., Herremans D. 2017. Generating guitar solos by integer programming. Journal of the Operational Research Society. 33
Pattern detection • Compression algorithm: COSIATEC (Meredith, 2013) • Point set representation • Computes a compressed encoding of the piece —> maximal translatable patterns 34
Example: Bach prelude 20, bk 2 35
MorpheuS: optimisation algorithm • Problem: find new pitches • Objective: match tension profile to template • Hard constraint: detected patterns 36
MorpheuS: generating affective music Random (with patterns) Herremans D., Chew E. 2017. MorpheuS: generating structured music with constrained patterns and tension. IEEE Transactions on Affective Computing. 37 PP (99).
MorpheuS: generating affective music Herremans D., Chew E. 2017. MorpheuS: generating structured music with constrained patterns and tension. IEEE Transactions on Affective Computing. 38 PP (99).
Example: Kabalevsky’s ‘Clowns’ 39
MorpheuS: generating affective music How can we fully model music so we don’t need a template? 40
Long-short term networks With CNN 41
Modeling music with LSTM • Convolutional nets have transformed the field of vision. Can we leverage this to music? • Challenge: time… • RNNs • LSTMs • … • Examples: • Boulanger-Lewandowski (2012): piano roll representation with RNN-RBM • Sigtia et al. (2014): Piano roll representation with CNN and RNN 42
Music as tonnetz images • Tonnetz (Euler, 1739) 43
Extended Tonnetz matrix 44
Modeling music with LSTM Code: http://dorienherremans/software Chuan C.-H., Herremans D.. 2018. Modeling temporal tonal relations in polyphonic music through deep networks with a novel image-based 45 representation. The Thirty-Second AAAI Conference on Artificial Intelligence . New Orleans, US.
Modeling music with LSTM • Filters reflect fifth relationships • Generated music is more tonally stable, with less tension • Higher compression rates —> more structure 46
Word2vec & music 47
Recommend
More recommend
