20112022: SIMSSA Single Interface for Music Score Searching and - PowerPoint PPT Presentation

Update on the SIMSSA Project Single Interface for Music Score Searching and Analysis Ichiro Fujinaga Music Technology Area, Schulich School of Music M c G i l l U n i v e r s i t y

2011–2022: SIMSSA 
 Single Interface for Music Score Searching and Analysis ❖ Create a prototype for single website to search all digitized music scores world wide 
 (“Google scores” minus Google) ❖ Use OMR to make them content searchable: concentrate on early music ❖ Includes basic analytical tools on the website ❖ Funded by SSHRC and FRQSC: $3.4M CAD ( € 2.3M) Utrecht DM@DH2019 Fujinaga 2 /32

Optical Music Recognition (OMR) A process of converting images of music scores into a symbolic computer representation, such as MIDI, MusicXML, or MEI (Music Encoding Initiative). OMR Utrecht DM@DH2019 Fujinaga 3 /32

Steps Involved in OMR Image 
 Music Symbol Music Notation Preprocessing Recognition Reconstruction Final Symbol 
 Staves 
 Binarization Output Combination Digitized 
 Processing Noise Removal Score Semantic 
 Structural 
 Symbol 
 Assignment 
 Analysis Segmentation (pitch, value) Musical 
 Image 
 Symbol 
 Structure 
 Segmentation Classification Reconstruction Utrecht DM@DH2019 Fujinaga 4 /32

SIMSSA Workflow for Neume Notation Digitized Manuscript Automatic Pitch Detection Layout Analysis 
 Pitch and Other Calvo’s Method Corrections: Neon.js Layout Correction Pixel.js Cantus Database Symbol Classification Gamera Cantus Ultimus Interface Classification Correction InteractiveClassifier.js Utrecht DM@DH2019 Fujinaga 5 /32

punctum Greyscale d d c dc cb Binarization c dedcd dfd edc C clef Border Removal Lyric Removal Sta fg Removal Shape Classification Music Reconstruction Shape/Image Alignment


 Layout Analysis Calvo’s Method

Three Di fg erent Outputs in One Step! 
 Using Convolutional Neural Networks Utrecht DM@DH2019 Fujinaga 8 /32

Separation of Staves, Notes & Texts Jorge Calvo Zaragoza ❖ S Utrecht DM@DH2019 Fujinaga 9 /32

Selectional Auto Encoders Jorge Calvo Zaragoza Utrecht DM@DH2019 Fujinaga 10 /32

Utrecht DM@DH2019 Fujinaga 11 /32

Accuracy & Training Time Comparison Selective Auto Encoders vs Convolutional Neural Nets Two Medieval Manuscripts: Salzinnes & Einsiedeln Utrecht DM@DH2019 Fujinaga 12 /32

Pixel.js Zeyad Saleh, Ké Zhang & Eric Liu Utrecht DM@DH2019 Fujinaga 13 /32

Original Image & Ground Truth Original Image Ground Truth Utrecht DM@DH2019 Fujinaga 14 /32

Classification of an Unseen Page Utrecht DM@DH2019 Fujinaga 15 /32

Classification of an Unseen Page Utrecht DM@DH2019 Fujinaga 16 /32

Classification of an Unseen Page Utrecht DM@DH2019 Fujinaga 17 /32

InteractiveClassifier.js Minh Anh Nguyen Utrecht DM@DH2019 Fujinaga 18 /32

Neume Mapping Table to MEI Utrecht DM@DH2019 Fujinaga 19 /32

Neume Mapping Tool Imane Chafi Utrecht DM@DH2019 Fujinaga 20 /32

OCR & Text Alignment Timothy de Reuse Utrecht DM@DH2019 Fujinaga 21 /32

Neon.js: Version 3 Juliette Regimbal Utrecht DM@DH2019 Fujinaga 22 /32

Neon.js: Version 3 Juliette Regimbal What’s new ❖ Background images displayed with diva.js ❖ IIIF compliant! ❖ Editing via Verovio ❖ The first of version of Verovio that is editable! ❖ Also text is editable Utrecht DM@DH2019 Fujinaga 23 /32

Neon.js: Text Editing Jacob Hutnyk Utrecht DM@DH2019 Fujinaga 24 /32

Neon.js: Text Editing Jacob Hutnyk Utrecht DM@DH2019 Fujinaga 25 /32

Neon.js: Text Editing Jacob Hutnyk Utrecht DM@DH2019 Fujinaga 26 /32

Scoring-up Tool: Song Book Martha Thomae ALTUS SUPERIUS TENOR BASSUS Utrecht DM@DH2019 Fujinaga 27 /32

Scoring-up Tool: Song Book Martha Thomae ALTUS SUPERIUS TENOR BASSUS Utrecht DM@DH2019 Fujinaga 28 /32

Summary ❖ SIMSSA ❖ Document an analysis with Selective Auto Encoders ❖ Pixel.js ❖ Interactive Classifier ❖ Neume Mapping Tool ❖ OCR and text editing ❖ Neon.js ❖ Text editing ❖ Scoring-up Tool Utrecht DM@DH2019 Fujinaga 29 /32

SIMSSA Team @McGill: Summer 2019 Rian Adamian Jacob Hutnyk Cory McKay Peter Schubert Imane Chafi Alessandra Ignesti Zoé McLennan Martha Thomae Julie Cumming Yaolong Ju Néstor Nápoles Andrew Tran Alex Daigle Sam Howes Minh Anh Nguyen Vi-An Tran Tim de Reuse Andrew Kam Gustavo Pedro Gabriel Vigliensoni Glen Ethier Ian Lorenz Juliette Regimbal Emily Hopkins Sylvain Margot Evan Savage Utrecht DM@DH2019 Fujinaga 30 /32

Acknowledgements Christopher Antila Mahtab Ghamsari Minh Anh Nguyen Arielle Goldman Chris Niven Claire Arthur Ryan Groves Rory O’Connor William Bain Laura Osterlund Jamie Klassen Ryan Bannon Phyllis Ouyang Peter Henderson Laurier Baribeau Jérôme Parent-Lévesque Jason Hockman Noah Baxter Alexandre Parmentier Emily Hopkinson Laura Beauchamp Gustavo Pedro Andrew Horwitz Justin Bell Sacha Perry-Fagant Yaolong Ju Ruth Berkow Alastair Porter Andrew Kam Marina Borsodi-Benson Juliette Regimbal Anton Khelou John Ashley Burgoyne Deepanjan Roy Jamie Klassen Greg Burlet Zeyad Saleh Reiner Krämer Jorge Calvo-Zaragoza Harry Simmonds Véronique Lagacé Remi Chiu Brian Stern Saining Li Morgane Ciot Tristano Tenaglia Eric Liu Nat Condit-Schultz Martha Thomae Wendy Liu Julie Cumming Andrew Tran Evan Magoni Alex Daigle Vi-An Tran Zoé McLennan Marie DeYoung Gabriel Vigliensoni Nicky Mirfallah Tim de Reuse Tim Wilfong Lillio Mok Natasha Dillabough Mike Winters Alexander Morgan Daniel Donnelly Ling-Xiao Yang Catherine Motuz Neda Eshraghi Ké Zhang Maria Murphy Meredith Evans Néstor Nápoles Wei Gao Clare Neil David Garfinkel Utrecht DM@DH2019 Fujinaga 31 /32

Webpage: http://simssa.ca Github sources: https://github.com/DDMAL Funded by Utrecht DM@DH2019 Fujinaga /32 32