musegan multi track sequential generative adversarial
play

MuseGAN: Multi-track Sequential Generative Adversarial Networks for - PowerPoint PPT Presentation

May 09, 2023 •335 likes •602 views

MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment Hao-Wen Dong*, Wen-Yi Hsiao*, Li-Chia Yang, Yi-Hsuan Yang Research Center of IT Innovation, Academia Sinica Demo Page

  1. MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment Hao-Wen Dong*, Wen-Yi Hsiao*, Li-Chia Yang, Yi-Hsuan Yang Research Center of IT Innovation, Academia Sinica Demo Page https://salu133445.github.io/musegan/ *these authors contributed equally to this work

  2. Outline 。 Goals & Challenges 。 Data 。 Proposed Model 。 Results & Evaluation 。 Future Works Source Code https://github.com/salu133445/musegan Demo Page https://salu133445.github.io/musegan/ 2

  3. Generate pop music 。 of multiple tracks Goals 。 in piano-roll format [Source Code] https://github.com/ salu133445/musegan [Demo Page] https://salu133445. 。 using GAN with CNNs github.io/musegan/ 3

  4. Multi-track GAN Challenge I Multitrack Interdependency vocal piano strings bass drums music & clip by phycause 4

  5. Convolutional Challenge II Neural Networks Music Texture melody chord (harmony) 5

  6. Challenge III Temporal Structure song paragraph 1 paragraph 2 paragraph 3 phrase 1 phrase 2 phrase 2 phrase 3 phrase 4 bar 1 bar 2 bar 3 bar 4 4/4 time beat 1 beat 2 beat 3 beat 4 step 1 step 2 ··· step 24 6

  7. Challenge III Temporal Structure Convolutional Neural Networks Fixed Structure phrase 2 bar 1 bar 2 bar 3 bar 4 4/4 time beat 1 beat 2 beat 3 beat 4 step 1 step 2 ··· step 24 7

  8. Piano-roll (with symbolic timing) Data Representation polyphonic  multi-track  time step Bar 1 Bar 2 Bar 3 Bar 4 pitch time 8

  9. Piano-roll (with symbolic timing) Data Representation polyphonic  multi-track  Bar 1 Bar 2 Bar 3 Bar 4 A3 pitch t 0 t 1 time 9

  10. Multi-track Piano-roll (with symbolic timing) Data Representation polyphonic  multi-track  pitch tracks time 10

  11. Bass Data Representation Drums Strings Piano Guitar 4 bars 84 5 tracks pitches a 4 × 96 × 84 × 5 tensor 96 time steps 11

  12. LPD (Lakh Pianoroll Dataset) 。 >170,000 multi-track piano-rolls 。 Derived from Lakh MIDI Dataset 。 Mainly pop songs Data Pypianoroll (Python package) 。 Manipulation & Visualization 。 Efficient Save/Load [Dataset] https://salu133445.gith 。 Parse/Write MIDI files ub.io/musegan/dataset 。 On PYPI (pip installable) [Pypianoroll] https://salu133445. github.io/pypianoroll/ 12

  13. Generative Adversarial Networks Generator random noise fake data critic Discriminator z ~ p ( z ) G G( z ) (wgan-gp) D real/fake X real data 4-bar phrases of 5 tracks 13

  14. MuseGAN – An Overview temporal bar generator generator G temp G bar 1 random noise 4 piano-roll matrices 4 latent variables 14

  15. MuseGAN Bar Generator z z z G G z G G G z z z z z z z z z 15

  16. MuseGAN Bar Generator Coordination z z z z G track-independent G z G G G No Coordination z z z z z z z z z track-dependent 16

  17. MuseGAN Bar Generator z G z z z z G G z z G z G z z G G z z G G z z G z G z z z z z z z z z 17

  18. MuseGAN Bar Generator z G z z z z G G z z G z G z z G G z z G G z z G z G z z z z z z z z z 18

  19. Time MuseGAN Dependent Independent Dependent Melody Groove Track Independent Chords Style Bar Generator Chords z G z Style z z z G G z z G z G z G z G Melody z z G G z z G z G z z z z Groove z z z z z 19

  20. Bass Line Drum pattern Results Sample 1 Sample 2 Chords More Samples on Demo Page https:// salu133445.github.io/musegan / Bass Drums Guitar Strings Piano Step 0 Step 700 Step 2500 Step 6000 Step 7900 20

  21. Monitor the Training Objective Metrics UPC Negative Critic Loss 10 12 10 10 10 8 10 6 step QN 10 4 0 2000 4000 6000 8000 step UPC number of used pitch classes per bar QN ratio of qualified notes step 21

  22. User Study composer H : harmonious R : rhythmic jamming MS : musically structured C : coherent OR: overall rating hybrid 22

  23. Summary 。 MuseGAN ◦ a novel GAN for multi-track sequence generation ◦ multi-track , polyphonic music ◦ human-AI cooperative scenario (see the paper) 。 Lakh Pianoroll Dataset (LPD) ( new dataset!! ) 。 Pypianoroll ( new package!! ) 23

  24. Full Song Generation Future song Works paragraph 1 paragraph 2 paragraph 3 phrase 1 phrase 2 phrase 2 phrase 3 phrase 4 bar 1 bar 2 bar 3 bar 4 beat 1 beat 2 beat 3 beat 4 step 1 step 2 ··· step 24 Hierarchical Temporal Structure 24

  25. Cross-modal Generation 。 Music + Video Future 。 Music + Lyrics Works 。 Video + Text 25

  26. Source Code https://github.com/salu133445/musegan Demo Page https://salu133445.github.io/musegan/ Q&A MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment

Recommend

MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and

MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and

MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment Hao-Wen Dong*, Wen-Yi Hsiao*, Li-Chia Yang, Yi-Hsuan Yang Research Center of IT Innovation, Academia Sinica Demo Page

449 views • 34 slides
Generative Adversarial Nets(GANs) Troy Cary and Chenzhi Zhao A generative adversarial net is

Generative Adversarial Nets(GANs) Troy Cary and Chenzhi Zhao A generative adversarial net is

Generative Adversarial Nets(GANs) Troy Cary and Chenzhi Zhao A generative adversarial net is a type of neural net, used in deep learning/machine learning problems The goal of a GAN is to train two simultaneous models: a generative model

441 views • 18 slides
Multi-objective training of Generative Adversarial Networks with multiple discriminators Isabela

Multi-objective training of Generative Adversarial Networks with multiple discriminators Isabela

Multi-objective training of Generative Adversarial Networks with multiple discriminators Isabela Albuquerque , Jo ao Monteiro , Thang Doan, Breandan Considine, Tiago Falk, and Ioannis Mitliagkas Equal contribution 1 / 11 The

496 views • 15 slides
GAN-based Photo Video Synthesis Summary of Generative Adversarial Nets Lei Zhang What is

GAN-based Photo Video Synthesis Summary of Generative Adversarial Nets Lei Zhang What is

GAN-based Photo Video Synthesis Summary of Generative Adversarial Nets Lei Zhang What is Generative Adversarial Networks (GAN)? Generative - creating new data that depends on the choice of the training set Adversarial - competitive

894 views • 9 slides
Introduction to Generative Adversarial Network (GAN) Hongsheng Li Department of Electronic

Introduction to Generative Adversarial Network (GAN) Hongsheng Li Department of Electronic

Introduction to Generative Adversarial Network (GAN) Hongsheng Li Department of Electronic Engineering Chinese University of Hong Kong Adversarial adj. 1 Generative Models Density Estimation ( | ) p y x

1.35k views • 98 slides
Conditional Generative Adversarial Networks (and a brief look at image-to-image translation)

Conditional Generative Adversarial Networks (and a brief look at image-to-image translation)

Conditional Generative Adversarial Networks (and a brief look at image-to-image translation) Final Presentation Peter Bromley Generative Models What is generative modeling? Data: Samples from high dimensional probability distribution p real

621 views • 25 slides
Generative Adversarial Networks, Wasserstein Distance, and Adversarial Loss Zhiyu Min Alibaba

Generative Adversarial Networks, Wasserstein Distance, and Adversarial Loss Zhiyu Min Alibaba

Generative Adversarial Networks, Wasserstein Distance, and Adversarial Loss Zhiyu Min Alibaba AliMe X-Lab Outline GAN Definition and formulation Saddle point optimization Vanishing gradient Alternative objective for Generator

516 views • 25 slides
Generative Adversarial Networks presented by Ian Goodfellow presentation co-developed with Aaron

Generative Adversarial Networks presented by Ian Goodfellow presentation co-developed with Aaron

Generative Adversarial Networks presented by Ian Goodfellow presentation co-developed with Aaron Courville 1 In todays talk Generative Adversarial Networks Goodfellow et al., NIPS 2014 Conditional Generative

2.84k views • 40 slides
Applications of GANs Photo-Realistic Single Image Super-Resolution Using a Generative

Applications of GANs Photo-Realistic Single Image Super-Resolution Using a Generative

Applications of GANs Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks Generative Adversarial Text to Image Synthesis

614 views • 27 slides
Applications of GANs Photo-Realistic Single Image Super-Resolution Using a Generative

Applications of GANs Photo-Realistic Single Image Super-Resolution Using a Generative

Applications of GANs Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks Generative Adversarial Text to Image Synthesis

666 views • 39 slides
Robust Estimation and Generative Adversarial Networks Weizhi ZHU Hong Kong University of Science

Robust Estimation and Generative Adversarial Networks Weizhi ZHU Hong Kong University of Science

Robust Estimation and Generative Adversarial Networks Weizhi ZHU Hong Kong University of Science and Technology wzhuai@ust.hk April 3, 2019 Robust Estimation and Generative Adversarial Nets [GLYZ18] Generative Adversarial Nets for Robust

252 views • 24 slides
Generative Adversarial Networks Aaron Mishkin UBC MLRG 2018W2 1 Generative Adversial Networks

Generative Adversarial Networks Aaron Mishkin UBC MLRG 2018W2 1 Generative Adversial Networks

Generative Adversarial Networks Aaron Mishkin UBC MLRG 2018W2 1 Generative Adversial Networks Two imaginary celebrities that were dreamed up by a random number generator. https://research.nvidia.com/publication/2017-10

665 views • 39 slides
LAB MEETING: A Connection Between Generative Adversarial Networks, Inverse Reinforcement Learning

LAB MEETING: A Connection Between Generative Adversarial Networks, Inverse Reinforcement Learning

LAB MEETING: A Connection Between Generative Adversarial Networks, Inverse Reinforcement Learning and Energy-Based models Suwon Suh POSTECH MLG Feb, 13, 2017 Goal Understanding Basic Models 1) Generative Adversarial Networks (GAN) 2)

723 views • 12 slides
Generative Adversarial Network and it its Applications to Human La Language Processing

Generative Adversarial Network and it its Applications to Human La Language Processing

Generative Adversarial Network and it its Applications to Human La Language Processing Hung-yi Lee Full version of the tutorial Outline Part I: General Introduction of Generative Adversarial Network (GAN) Part II: Applications

918 views • 78 slides
Adversarial Training Attacks on Deep Networks and Generative Adversarial Networks Erkut Erdem

Adversarial Training Attacks on Deep Networks and Generative Adversarial Networks Erkut Erdem

images from Geris Game (Pixar, 1997) Adversarial Training Attacks on Deep Networks and Generative Adversarial Networks Erkut Erdem Aykut Erdem Levent Karacan Computer Vision Lab, Hacettepe University Outline Part 1: Attacks on

1.11k views • 72 slides
Generative Adversarial Networks (GANs) Prof. Seungchul Lee Industrial AI Lab. Source 1

Generative Adversarial Networks (GANs) Prof. Seungchul Lee Industrial AI Lab. Source 1

Generative Adversarial Networks (GANs) Prof. Seungchul Lee Industrial AI Lab. Source 1 GAN (Generative Adversarial Network) by YouTube: https://www.youtube.com/watch?v=odpjk7_tGY0 Slides:

2.44k views • 49 slides
Generative Adversarial Networks Ian Goodfellow Research Scientist GPU Technology Conference

Generative Adversarial Networks Ian Goodfellow Research Scientist GPU Technology Conference

Generative Adversarial Networks Ian Goodfellow Research Scientist GPU Technology Conference San Jose, California 2016-04-05 Generative Modeling Have training examples: x p train ( x ) Want a model that can draw samples: x p

317 views • 16 slides
Learning Universal Adversarial Perturbations with Generative Models Jamie Hayes & George

Learning Universal Adversarial Perturbations with Generative Models Jamie Hayes & George

Learning Universal Adversarial Perturbations with Generative Models Jamie Hayes & George Danezis UCL Adversarial examples transfer between different models. An adversarial example crafted against one model will generally fool other models.

515 views • 18 slides
Generative Adversarial Networks (GANs) Ian Goodfellow, OpenAI Research Scientist Presentation at

Generative Adversarial Networks (GANs) Ian Goodfellow, OpenAI Research Scientist Presentation at

Generative Adversarial Networks (GANs) Ian Goodfellow, OpenAI Research Scientist Presentation at Berkeley Artificial Intelligence Lab, 2016-08-31 Generative Modeling Density estimation Sample generation Training examples Model samples

899 views • 37 slides
Introduction to Generative Adversarial Networks Ian Goodfellow, OpenAI Research Scientist NIPS

Introduction to Generative Adversarial Networks Ian Goodfellow, OpenAI Research Scientist NIPS

Introduction to Generative Adversarial Networks Ian Goodfellow, OpenAI Research Scientist NIPS 2016 Workshop on Adversarial Training Barcelona, 2016-12-9 Adversarial Training A phrase whose usage is in flux; a new term that applies to both

792 views • 25 slides
CSC321 Lecture 19: Generative Adversarial Networks Roger Grosse Roger Grosse CSC321 Lecture 19:

CSC321 Lecture 19: Generative Adversarial Networks Roger Grosse Roger Grosse CSC321 Lecture 19:

CSC321 Lecture 19: Generative Adversarial Networks Roger Grosse Roger Grosse CSC321 Lecture 19: Generative Adversarial Networks 1 / 25 Overview In generative modeling, wed like to train a network that models a distribution, such as a

920 views • 25 slides
Adversarial BoltzmannMachines Belief Nets Networks Variational NADE

Adversarial BoltzmannMachines Belief Nets Networks Variational NADE

Unsupervised Learning Non-probabilistic Models Probabilistic Sparse Coding (Generative) Models Autoencoders Others (e.g. k-means) Tractable Models Non-Tractable Models Generative Fully observed Adversarial

815 views • 58 slides
Generative Adversarial Networks (GANs) Ian Goodfellow, OpenAI Research Scientist Re-Work Deep

Generative Adversarial Networks (GANs) Ian Goodfellow, OpenAI Research Scientist Re-Work Deep

Generative Adversarial Networks (GANs) Ian Goodfellow, OpenAI Research Scientist Re-Work Deep Learning Summit San Francisco, 2017-01-26 Generative Modeling Density estimation Sample generation Training examples Model samples

384 views • 22 slides
Generative Adversarial Networks Wanyue Li

Generative Adversarial Networks Wanyue Li

Generating Fundus Fluorescence Angiography Images from Structure Fundus Images Using Generative Adversarial Networks Wanyue Li (wanyueli93@126.com) University of Science and Technology of China (USTC)

144 views • 10 slides

More recommend