Applications of AI in music Smart music through machine learning Dorien Herremans ISTD, Singapore University of Technology and Design IHPC, A*STAR Music Research Symposium, 2/2/18, IHPC, A*STAR 1
The rapid rise of digital music • In 2015: revenue from digital music: 6.4 billion USD = 45% of the global music industry • Asia: 14% of these revenues • Number of digital music users expected to grow 15% a year until 2020 2
New application opportunities 3
Overview 1. Hit song prediction 2. Modelling music with machine learning 4
Hit song prediction 5
Can we predict hits? • In 2011 record companies invested 4.5 billion USD in new talent worldwide (IFPI, 2012) “Hit song science is not yet a science” (Pachet and Roy, 2008) 6
Dance hit song prediction • Dataset: • Official Charts Company • Billboard • 1985 until 2013 • What are hits? —> Top 10 hits versus Top 30-40 • 139 Audio features such as loudness, tempo, time between beats, timbre, etc. 7
Dance hit song prediction • Classification results: Top 10 vs Top 30-40 D. Herremans, D. Martens, K. Sörensen. 2014. Dance Hit Song Prediction. Journal of New Music Research – special issue on music and machine learning. 8 43(3):291–302.
Hit song prediction through social media • “There are people… with some sort of sixth sense for hit songs, such that they listen to hit songs even before they actually climb to the top of the record charts.” (Smit, 2013) • Early adopters (Rogers, 2010) • last.fm listening behaviour: • ~8000 members • 6 months Herremans D., Bergmans T.. 2017. Hit Song Prediction Based on Early Adopter Data and Audio Features. The 18th International Society for Music 9 Information Retrieval Conference (ISMIR) - Shuzou, China.
Future challenges • Social networking data • Deeper models • More advanced audio features • Other genres • Sentiment analysis of lyrics Related to other classification problems, e.g. composer identification, emotions detection. 10
Modeling music 11
For music generation •Research started in the 1950s •Why don’t we listen to generated music on our phones? 2 Challenges: - How to control emotion in music? - Long-term structure 12
1. Modeling emotion: tension • Important for music in: • Video games • Film background • YouTube • Tension: many aspects: tonal , rhythmic, loudness etc. 13
1. Modeling emotion: tension • 3D geometrical mathematical model of tonality (Chew, 2012) • Tool available online • First step towards more complex emotional models Herremans D., Chew E.. 2016. Tension ribbons: Quantifying and visualising tonal tension. Second International Conference on 14 Technologies for Music Notation and Representation (TENOR). 2:8-18. Cambridge, UK.
2. Long-term structure • We can generate good sounding short fragments of music • Guitar solos generated by Integer Programming • Short solo fragments Cunha N., Subramanian A., Herremans D. 2017. Generating guitar solos by integer programming. Journal of the Operational Research Society. 15
2. Long-term structure • MorpheuS app constrains tension AND long-term structure: Pattern detection on template —> constrained during generation Herremans D., Chew E. 2017. MorpheuS: generating structured music with constrained patterns and tension. IEEE Transactions on Affective Computing. 16 PP (99).
MorpheuS: generating affective music Herremans D., Chew E. 2017. MorpheuS: generating structured music with constrained patterns and tension. IEEE Transactions on Affective Computing. 17 PP (99).
MorpheuS: generating affective music How can we fully model music so we don’t need a template? 18
Modeling music with LSTM • Leveraging success from image based systems in deep neural networks • Novel image based representation: tonnetz Chuan C.-H., Herremans D.. 2018. Modeling temporal tonal relations in polyphonic music through deep networks with a novel image-based 19 representation. The Thirty-Second AAAI Conference on Artificial Intelligence . New Orleans, US.
Modeling music with LSTM • Filters reflect fifth relationships • Generated music is more tonally stable, with less tension • Higher compression rates —> more structure 20
Modeling music with word2vec NLP: Words that are used and occur in the same contexts tend to purport similar meanings A slice of music = word -> Captures tonality We can calculate a ‘semantic’ similarity between slices of music based only on their context Herremans D., Chuan C.-H.. 2017. Modeling Musical Context with Word2vec. First International Workshop On Deep Learning and Music joint with IJCNN. 1:11-18. 21 Anchorage, US
Future of music modeling • Generating music with long-term structure and affect: • Film, YouTube background, automatic accompaniment apps and video games • Why else model music? • Audio transcription • Classification problems • Emotion recognition • Recommender systems • … Herremans D., Chuan C.-H., Chew E.. In Press. A Functional Taxonomy of Music Generation Systems. ACM Computing Surveys. 22
Conclusions Machine learning brings a range of new possibilities to digital music/audio , including: • Hit prediction • Classification problems, e.g. sentiment analysis of lyrics, emotion classification • Accompaniment app for beginner guitarists • Chord detection • Automatic music production • Music generation for game and film • Music and health applications • Automatic piano fingering • … 23
Applications of AI in music Smart music through machine learning Dorien Herremans ISTD, Singapore University of Technology and Design IHPC, A*STAR dorienherremans.com Music Research Symposium, 2/2/18, IHPC, A*STAR 24
Recommend
More recommend
