course introduction
play

Course Introduction Juhan Nam Who We Are Instructor: Juhan Nam - PowerPoint PPT Presentation

Dec 22, 2022 •25 likes •355 views

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Course Introduction Juhan Nam Who We Are Instructor: Juhan Nam Associate Professor, Graduate School of Culture Technology (GSCT) Affiliated Professor, Graduate

  1. GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Course Introduction Juhan Nam

  2. Who We Are ● Instructor: Juhan Nam ○ Associate Professor, Graduate School of Culture Technology (GSCT) ○ Affiliated Professor, Graduate School of Artificial Intelligence (GSAI) ○ Music and Audio Computing Lab: https://mac.kaist.ac.kr/ ● TAs ○ Taegyun Kwon, PhD student, GSCT ○ Taejun Kim, PhD student, GSCT ○ Wonil Kim, PhD student, GSCT ○ Seunghyun Lee, MS Student, GSAI

  3. Music ● The most widely enjoyed cultural contents and activities Music in KAIST Source: http://times.kaist.ac.kr/news/articleView.html?idxno=3835, http://times.kaist.ac.kr/news/articleView.html?idxno=3185

  4. Music and Computer ● Computer is now essential in musical activities ○ Music listening: download music tracks as compressed audio files and uncompress them into waveforms

  5. Music and Computer ● Computer is now essential in musical activities ○ Music performance: musical instrument, karaoke machine

  6. Music and Computer ● Computer is now essential in musical activities ○ Music composition and production: recording, MIDI, processing, mixing

  7. Data and Processing ● The role of computer is representing musical data in a digital form and processing them according to the target task Output data Input data ○ Data: audio, MIDI, text (meta data) ○ Processing: audio (un)compression, sound synthesis, recording, digital audio effect, editing, and mixing

  8. Data and Processing ● The role of computer is representing musical data in a digital form and processing them according to the target task Output data Input data In such music systems, each step of processing is hand-designed and programmed by human based on domain knowledge such as digital signal processing, acoustics, and music theory

  9. Machine Learning for Music? ● Machine Learning (ML) is a method of teaching computer how to make accurate predictions using data Output data Input data In the ML-based systems, each step of processing is learned from data through a learning algorithm ● Why we need machine learning for music ?

  10. Music Listening ● Music at scale ○ Spotify: 60M tracks, 40K new tracks per day, and 4B playlists (2020) ○ SoundCloud: 200M tracks (2019) ○ YouTube: 500 min videos upload per min (2020) (music tracks, music videos and performance videos) ● Content organization, search, and recommendation become important ○ Meta data is not enough to explain the “content” of music ○ Need more rich descriptions Naver VIBE Source: https://newsroom.spotify.com/company-info/

  11. Music Listening ● Pandora’s Music Genome Project (1999) ○ Annotate a track with about 450 music attributes Genre, instruments, timbre, vocal quality, … ■ ○ Playlists are generated using the similarity of music attribute vectors ○ Not biased by the popularity Pandora Internet Radio ● Problems ○ Takes 20-30 mins to annotate a single track by a music expert ○ The dictionary size of music attributes is fixed

  12. Music Listening ● Need to teach computer how to describe music with natural language as humans do ○ Associate music with not only musical terms (e.g., genre, instrument, timbre) but also listening contexts (e.g. mood, time of day, location and activity) Input data Output data

  13. Google for Music (KAIST MAC Lab)

  14. Music Performance ● Mobile apps become popular in music education and entertainment ○ Music score ○ Karaoke ○ Instrument learning game ● Emergence of “smart” features MuseScore Smule ○ Performance evaluation ○ Score following and page turning ○ Auto-accompaniment Yousician Source: https://promusicianhub.com/yousician-review/, https://musescore.org, https://www.smule.com/

  15. Music Performance ● Extract music score information from audio ○ Identify and separate sound sources in adverse acoustic conditions: microphone, reverberation and interfering sources ○ Detect multiple pitches from polyphonic musical instruments: e.g., piano, guitar Source: http://jameasy.com/ko/company.html, https://magenta.tensorflow.org/onsets-frames

  16. Music Performance ● Need to teach computer how to separate individual sources and extract musical information from complex auditory scenes ○ Source separation from mixed audio ○ Transcribe polyphonic music into music score or MIDI Input data Output data

  17. Re-Performance by Polyphonic Piano Transcription (KAIST MAC Lab)

  18. Music Composition ● Automatic music composition has been a dream project since the birth of computer ○ Illiac Suite (String Quartet No. 4) (1957) ■ The first music score composed by an electronic computer ■ Composed using a Markov model https://www.youtube.com/watch?v=n0njBFLQSk8 ■ ○ Experiments in Musical Intelligence (EMI) (1980s) The Experimental Music Studio (Lejaren Hiller and Leonard Isaacson) Style imitation using pre-composed patterns (recombinant) ■ ■ https://www.youtube.com/watch?v=t6WeiyvAiYQ&t=52s ○ Numerous approaches in “algorithmic composition” ■ Audio/MIDI programming languages: Music-N, CSound, Max/PD, Common Music, Supercollider, Chuck ■ Rule-based or statistical models EMI: recombinant music (David Cope) Source: http://www.moz.ac.at/sem/lehre/lib/es/ems/hist/battisti.html

  19. Music Composition ● Recent advances in machine learning ○ Learn the sequential order of music data in a highly data-driven way ○ MIDI or audio generation ■ Flow Machine (Sony): https://www.flow-machines.com/ Music Transformer (Google): https://magenta.tensorflow.org/music-transformer ■ ■ Jukebox (OpenAI): https://openai.com/blog/jukebox/ Flow Machine Jukebox Music Transformer (“Hello World”: AI-composed album)

  20. Music Composition ● Need to teach computer how to learn the distribution of the high-dimensional long-term sequential data and generate music from conditions given by human ○ The conditions can be semantic, artist, lyrics, score, audio or even preference ○ Possible to create novel pieces (collaborating with human) ? Input data Output data

  21. Machine Learning for Music ● Powerful means to teach computer how to listen, perform and compose music Audio Learning Score (MIDI) Model Text

  22. Deep Learning ● The key element in recent AI technology and developed mainly in the computer vision, speech processing and natural language processing communities ○ Each of them handles a different modality: image, audio, text (i.e., symbol) ○ Due to the nature of data-driven approach (or less use of domain knowledge), the advance of deep learning has been naturally applied to a wide variety of domains that use image, audio and text as a input or output data form ○ Music is one of the domains that have benefited a lot from them ● Deep learning is representation learning ○ Transform a type of data onto a more meaningful vector space (i.e., feature space) ○ The vector spaces from different modalities of data are associated with each other by their correspondence

  23. Deep Learning for Music ● Modality-agnostic representation learning Audio Score (MIDI) Learning Model Text Image

  24. Objectives of This Course ● Understanding machine learning and deep learning ● Learning how to apply it to various tasks in the music domain ● Hands-on experiences with Python language and machine learning libraries through homework ● Gain experience of the full cycle of research through the final project

  25. Course Format ● This course is served as an 100% online format ● There are two types of online sessions ○ Pre-Recorded videos ■ Cover the lecture part Uploaded weekly in KLMS (YouTube link) ■ ■ Students must watch videos before the weekly Zoom meeting ○ Weekly Zoom meeting ■ Focus on review, interactive Q&A and hands-on practice ■ Thursday from Week #2: 2:30-3:45 PM (Hopefully, less than 60 minutes)

  26. Schedules ● Week 1 ○ Course introduction ● Week 2 ○ Audio data representations ● Week 3 ○ Machine learning review: supervised learning ● Week 4 ○ Machine learning review: unsupervised learning

  27. Schedules ● Week 5 ○ Chusuk (no class) ● Week 6 ○ Convolutional neural network (CNN): music classification and tagging ● Week 7 ○ Recurrent neural network (RNN): automatic music transcription ● Week 8 ○ Break (no class)

  28. Schedules ● Week 9 ○ Auto-encoder, U-net: source separation ● Week 10 ○ Variational auto-encoder (VAE), generative adversarial network (GAN): music generation and sound synthesis ● Week 11 ○ Auto-regressive models: music generation and sound synthesis ● Week 12 ○ Transformer: music transcription and generation

  29. Schedules ● Week 13 ○ Invited talk or advanced topics (TBD) ● Week 14 ○ Invited talk or advanced topics (TBD) ● Week 15 ○ TBD ● Week 16 ○ Final project presentations

  30. Pre-requisite ● Linear Algebra ● Probability and Statistics ● Basic understanding of machine learning and deep learning ● Digital Signal Processing: digital filters, discrete Fourier transform, and spectral analysis ● Programming Language: Python

  31. Software ● Audio processing: Librosa ● Machine learning and deep learning: Scikit-learn, PyTorch ● And more…

  32. Grading ● 4 assignments: 50% ● Final project (paper review, presentation and report): 50%

Recommend

INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION

INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION

TRAINING COURSE ON SOCIAL PROTECTION & FORMALIZATION TRINIDAD AND TOBAGO MARCH 15, 2017 INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION Design of the NIS Assistance from the

547 views • 23 slides
INTRODUCTION Introduction 2/42 INTRODUCTION Alternations I am giving bread to the

INTRODUCTION Introduction 2/42 INTRODUCTION Alternations I am giving bread to the

Patterns of variation in the expression of case and agreement Andrs Brny Leiden University Centre for Linguistics 9 November 2019, BaSIS Workshop a.barany@hum.leidenuniv.nl INTRODUCTION Introduction 2/42 INTRODUCTION

1.37k views • 42 slides
Introduction Introduction Introduction Introduction Outline Motivation Failures

Introduction Introduction Introduction Introduction Outline Motivation Failures

Introduction Introduction Introduction Introduction Outline Motivation Failures Definition and concepts Process and product properties Principles SE Approaches Motivation Software and the economy The economies of

1.03k views • 49 slides
Introduction Introduction Introduction Nationwide Cause for Concern 1

Introduction Introduction Introduction Nationwide Cause for Concern 1

Introduction Introduction Introduction Nationwide Cause for Concern 1 https://www.cdc.gov/std/stats17/infographic.htm Introduction Epidemiology in Western Colorado CPHE Surveillance Data Report, May 2018 Chlamydia Epidemiology in Western

977 views • 39 slides
DAQ introduction Purpose of this talk : (1) Introduction for those who have not been in every

DAQ introduction Purpose of this talk : (1) Introduction for those who have not been in every

DAQ introduction Purpose of this talk : (1) Introduction for those who have not been in every meeting (2) Some ideas of diagrams for section 7.1 (Introduction) of TP. Giles Barr 25 Jan 2018, Pembroke College On DUNE, the reality is there is a

474 views • 10 slides
INTRODUCTION cf. Schneider, Chapter 1 INTRODUCTION THEY ARE OUT TO GET YOU INTRODUCTION WHAT

INTRODUCTION cf. Schneider, Chapter 1 INTRODUCTION THEY ARE OUT TO GET YOU INTRODUCTION WHAT

ADVANCED COMPUTER SECURITY INTRODUCTION cf. Schneider, Chapter 1 INTRODUCTION THEY ARE OUT TO GET YOU INTRODUCTION WHAT IS SECURITY? A systems security policies describe What the system is supposed to do Store and provide access

479 views • 16 slides
Overview Introduction to SMIL Introduction to W3C and XML Introduction to SMIL

Overview Introduction to SMIL Introduction to W3C and XML Introduction to SMIL

Overview Introduction to SMIL Introduction to W3C and XML Introduction to SMIL Writing a SMIL file Adding Media Objects Martin Halvey Clipping media files January 2008 Timing and Synchronising Layout World Wide

387 views • 14 slides
14 Introduction Introduction Bad guys can put malware into Example: hosts

14 Introduction Introduction Bad guys can put malware into Example: hosts

Introduction Introduction source Encapsulation ISO/OSI reference model message M application segment H t H t M transport network datagram H n H n H t M presentation: allow applications to frame link H l H n H t M interpret

583 views • 4 slides
Previous Lecture Introduction to SMIL 2 Introduction to W3C and XML Introduction to SMIL

Previous Lecture Introduction to SMIL 2 Introduction to W3C and XML Introduction to SMIL

Previous Lecture Introduction to SMIL 2 Introduction to W3C and XML Introduction to SMIL Writing a SMIL file Adding Media Objects Martin Halvey Clipping media files January 2008 Timing and Synchronising Layout

322 views • 8 slides
Lecture 1 Andreas Habegger Introduction Zynq Introduction Zynq Introduction Zynq PS vs. PL

Lecture 1 Andreas Habegger Introduction Zynq Introduction Zynq Introduction Zynq PS vs. PL

Introduction Zynq Lecture 1 Andreas Habegger Introduction Zynq Introduction Zynq Introduction Zynq PS vs. PL Processing System Processor Peripherals Data Buses AXI Bus Conclusion BTE5380 - Embedded Systems Octobre 2014 Andreas Habegger

1.81k views • 30 slides
2018.06 01 SMILE5 Introduction S E 5 02 Alpha Cloud M I L 03 Company Introduction 04

2018.06 01 SMILE5 Introduction S E 5 02 Alpha Cloud M I L 03 Company Introduction 04

2018.06 01 SMILE5 Introduction S E 5 02 Alpha Cloud M I L 03 Company Introduction 04 Project References S E 5 PART.1 SMILE5 Introduction M I L Introduction STORION-SMILE5 Residential Energy Storage System Inverter 5kW Output Input

1.06k views • 25 slides
Introduction to CICS Course introduction Course introduction What is CICS? What is an

Introduction to CICS Course introduction Course introduction What is CICS? What is an

Introduction to CICS Course introduction Course introduction What is CICS? What is an application server? Why use an application server? Course introduction Services provided by CICS How CICS applications are defined Scaling CICS to meet

1.84k views • 146 slides
Network Visualization Introduction Presented by Shahed Introduction Introduction Basic

Network Visualization Introduction Presented by Shahed Introduction Introduction Basic

Network Visualization Introduction Presented by Shahed Introduction Introduction Basic building blocks Node Links (relationship between nodes) Spatial information Network data

584 views • 9 slides
Introduction Introduction (1) Main argument of the paper: universities are not only

Introduction Introduction (1) Main argument of the paper: universities are not only

Le jugement des pairs comme outil de management des universits Peer-review as a management tool Christine Musselin (CSO, Sciences Po et CNRS) Avril 2013, Lirrsistible ascension du capitalisme acadmique Introduction Introduction (1)

590 views • 28 slides
Introduction Introduction to storage and to storage and filesystems filesystems Introduction

Introduction Introduction to storage and to storage and filesystems filesystems Introduction

Moreno Baricevic Gilberto Daz Axel Kohlmeyer Stefano Cozzini ULA ICTP CNR-IOM DEMOCRITOS Merida, VENEZUELA Trieste, ITALY Trieste, ITALY Introduction Introduction to storage and to storage and filesystems filesystems Introduction

1.19k views • 51 slides
JABLOTRON JA-100 introduction November 2011 Introduction Todays goals First introduction of

JABLOTRON JA-100 introduction November 2011 Introduction Todays goals First introduction of

JABLOTRON JA-100 introduction November 2011 Introduction Todays goals First introduction of the brand new JA-100 alarm system Visions Basic architecture System components Setting Help us with final validations You are the FIRST ones

1.69k views • 111 slides
1 Introduction Introduction Communication System Baseband an adjective that describes

1 Introduction Introduction Communication System Baseband an adjective that describes

Introduction Introduction Physical Layer Mobile network Global ISP Provides the means to transmit bits Basic Concepts of from sender to receiver, Data Transmission Defines the mechanical, electrical, and Home network timing

463 views • 7 slides
Welcome! Todays Agenda: Topic Introduction Course Introduction Team Practical

Welcome! Todays Agenda: Topic Introduction Course Introduction Team Practical

INFOGR Computer Graphics Jacco Bikker - April-July 2015 - Lecture 1: Introduction Welcome! Todays Agenda: Topic Introduction Course Introduction Team Practical Details Assignments Field Study State of

869 views • 62 slides
Introduction (00) RNDr. Martin Madaras, PhD. madaras@skeletex.xyz Introduction Introduction

Introduction (00) RNDr. Martin Madaras, PhD. madaras@skeletex.xyz Introduction Introduction

Virtual and Augmented Reality Introduction (00) RNDr. Martin Madaras, PhD. madaras@skeletex.xyz Introduction Introduction Who am I? What do I do here? 2 About me Short research bio: - 2014 - finished PhD at FMFI UK - 2014 2016 -

1.3k views • 39 slides
PRESENTATION The Introduction What is the Introduction ? The introduction prepares the audience

PRESENTATION The Introduction What is the Introduction ? The introduction prepares the audience

PRESENTATION The Introduction What is the Introduction ? The introduction prepares the audience for your presentation. It tells them what your presentation is about, why it is important, and finally, what to listen for in your speech. Get the

505 views • 3 slides
How to make a presentation with L A T EX? Introduction to L A T EX Introduction to Beamer

How to make a presentation with L A T EX? Introduction to L A T EX Introduction to Beamer

Introduction to Beamer Hafida Benhidour How to make a presentation with L A T EX? Introduction to L A T EX Introduction to Beamer Introduction to Beamer Hafida Benhidour Department of computer science King Saud University November

732 views • 49 slides
Shenzhen Cuilu jewelry Co., Ltd was founded in 1996 and its a large private enterprise

Shenzhen Cuilu jewelry Co., Ltd was founded in 1996 and its a large private enterprise

Introduction Introduction - - Brief Brief Introduction Introduction of of Cuilu Cuilu Group Group Introduction Introduction - - Brief Brief Introduction Introduction of of Cuilu Cuilu Group Group Shenzhen Cuilu jewelry Co., Ltd

147 views • 14 slides
Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org

Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org

V olunteer C ommunications N etwork (V-C-N.org) Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org Academy Course - 2012-02.24-2208 262: Introduction to ICS-214 : V-C-N.org : Slide: 1 V olunteer C

735 views • 34 slides
Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org

Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org

V olunteer C ommunications N etwork (V-C-N.org) Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org Academy Course - 2012-02.24-2208 262: Introduction to ICS-214 : V-C-N.org : Slide: 1 V olunteer C

558 views • 18 slides

More recommend