Multimodal Music Tagging Task Nicola Orio University of Padova - PowerPoint PPT Presentation

Multimodal Music Tagging Task Nicola Orio – University of Padova Cynthia C.S. Liem – Delft University of Technology Geoffroy Peeters – IRCAM Markus Schedl – Johannes Kepler University 1 CLEF2012, Rome 18/09/2012 MusiClef: Multimodal Msic Tagging Task

Evaluation Initiatives – 1 • MIREX (since 2004) • Community-based selection of tasks • Many tasks address audio feature extraction algorithms • Participants submit algorithms that are run by organizers • Music files are not shared with participants • Million Song Dataset (since 2011) • One task on music proposed by organizers • Participants can run their • Audio features are computed using proprietary algorithms • Only features are shared with participants 2 CLEF2012, Rome 18/09/2012 MusiClef: Multimodal Msic Tagging Task

Evaluation Initiatives – 2 • Quaero-Eval (since 2012) • Tasks agreed with participants • Strategies to grant public access to evaluation results • Participants run training experiments on a shared repository • Runs on test set made by the organizers • MediaEval (since 2008) • Started as VideoCLEF • Scope extended to other media • Emphasis on multimodality • Regular tasks, plus Brave New Tasks (e.g. MusiClef) 3 CLEF2012, Rome 18/09/2012 MusiClef: Multimodal Msic Tagging Task

Goals of MusiClef • To focus evaluation on professional application scenarios • Textual description of music items • To grant replication of experiments and results • Feature extraction phase is crucial • To promote the exploitation of multimodal sources of information • Content (audio) + Context (tags & webpages) • To disseminate music related initiatives • Outside the music information retrieval community 4 CLEF2012, Rome 18/09/2012 MusiClef: Multimodal Msic Tagging Task

The workflow public web site not public MusicCLEF MusicCLEF if(conn SELEC 1.read WHERE if(conn 2. produce print SELEC WHERE print MP3 features Library extractor last.fm data 7. publish Low level features Campaign Results Participant 1 evaluation 3 . Results . h t d t p o w r e n participant pc . Participant q webservice l o u last.fm a e 6 d s . t s u b m i t 4. read Low level features <script var a= 5. produce var xl if(xls tags if(conn 4. read SELEC results WHERE Participant print algorithms last.fm data 5 CLEF2012, Rome 18/09/2012 MusiClef: Multimodal Msic Tagging Task

Multimodal Music Tagging • Definition • Songs of a commercial music library need to be categorized according to their usage in TV and radio broadcasts (e.g. soundtracks, jingles) • Practical motivation • The search for suitable music for video productions is a major activity for music professionals • Collaborative filtering systems are taking their role • Notwithstanding their known limitations: long-tail, cold start… • Annotating professional music libraries is another important professional activity 6 CLEF2012, Rome 18/09/2012 MusiClef: Multimodal Msic Tagging Task

Human Assessment Different sources of information are routinely exploited by professionals to overcome limitations of individual media 7 CLEF2012, Rome 18/09/2012 MusiClef: Multimodal Msic Tagging Task

Test Collection – 1 • Individual songs of pop and rock music • 1355 songs (form 218 artists) • Social tags • Gathered from Last.fm API • Multilingual sets of Web pages related to artists+albums • Mined querying Google • Acoustic features: MFCC (using MIRToolbox) with a window length of 200ms and 50% overlap 8 CLEF2012, Rome 18/09/2012 MusiClef: Multimodal Msic Tagging Task

Test Collection – 2 • Test collection created starting from the “500 Greatest Songs of All Time” (Rolling Stone) • Expected high number of social tags and web pages • Ground truth created by experts in the domain • 355 tags selected (167 genre, 288 usage) • Tags associated to less than 20 songs were discarded • Reference implementation in Matlab • Participants has an example to run a complete experiment • Code for the evaluation made already available 9 CLEF2012, Rome 18/09/2012 MusiClef: Multimodal Msic Tagging Task

Conclusions • Participants are still submitting their runs • So far, 8-10 perspective participants have been involved • We are collecting the runs • Additional evaluation will be run on subsets of tags • Grouping tags in classes related to: • Affective, situational, sociocultural aspects • Correlate different modalities with different aspects • Results will be presented at MediaEval Workshop • Pisa, 4-5 October 2012 10 CLEF2012, Rome 18/09/2012 MusiClef: Multimodal Msic Tagging Task