Deep Learning Techniques for Music Generation Recurrent (5) - PowerPoint PPT Presentation

Deep Learning Techniques for Music Generation Recurrent (5) Jean-Pierre Briot Jean-Pierre.Briot@lip6.fr Laboratoire d’Informatique de Paris 6 (LIP6) Sorbonne Université – CNRS Programa de Pós-Graduação em Informática (PPGI) UNIRIO Deep Learning – Music Generation – 2019 Jean-Pierre Briot

Recurrent Deep Learning – Music Generation – 2019 2 Jean-Pierre Briot

#1 Limitation – Generation and #2 Limitation – Fixed Length • Works OK But: • Fixed input (and output) length Deep Learning – Music Generation – 2019 3 Jean-Pierre Briot

#1 Limitation – Generation and #2 Limitation – Fixed Length Solution: Recurrent Network (RNN) • Works OK But: • Fixed input (and output) length Solution: • Recurrent Network (RNN) • Variable length • Memorizes previous steps • Predicts next step Deep Learning – Music Generation – 2019 4 Jean-Pierre Briot

Recurrent Network (RNN) • Memorizes previous steps • Can learn from previous step • Predicts next step • Can learn sequences Recurrent connexions Deep Learning – Music Generation – 2019 5 Jean-Pierre Briot

Recurrent Connexions Deep Learning – Music Generation – 2019 6 Jean-Pierre Briot

Alternative (More Common) Notation Deep Learning – Music Generation – 2019 7 Jean-Pierre Briot

RNN Prediction Deep Learning – Music Generation – 2019 8 Jean-Pierre Briot

Training a RNN y = Expected next note(x) Training with • One example (x, y) = one note Or • One example (x, y) = one melody with x = y translated 1 step back Deep Learning – Music Generation – 2019 9 Jean-Pierre Briot

RNN Generation • Iterated generation • Note by Note • Reinject Next Note to Produce Next Next Note • Arbitrary Length input layer hidden layer output layer Deep Learning – Music Generation – 2019 10 Jean-Pierre Briot

RNNMelody Input layer Hidden layer Output layer Current Next Note Slice Note Slice … Deep Learning – Music Generation – 2019 11 Jean-Pierre Briot

RNN – Iterative Feedforward – #1 Example LSTM blocks Corpus: Soprano parts of Bach Chorales input layer hidden layer output layer Deep Learning – Music Generation – 2019 12 Jean-Pierre Briot

Celtic Music • Training Examples/Corpus: • In ABC format (see later) -> Music21 -> representation • 29 songs from the Session (https://thesession.org/) • In the same key (D major) and the same rhythm metric (4/4) Deep Learning – Music Generation – 2019 13 Jean-Pierre Briot

Celtic Melodies RNN LSTM Deep Learning – Music Generation – 2019 14 Jean-Pierre Briot

Gradient Vanishing/Explosion Deep Learning – Music Generation – 2019 15 Jean-Pierre Briot

LSTM (Long Short-Term Memory) [Hochreiter and Schmidhuber, 1997] • Protection of Memory by Gates • Gates are controlled by differentiable functions • Thus subject to Training • Training of the Meta-Level (Control) Deep Learning – Music Generation – 2019 16 Jean-Pierre Briot

RNN – Iterative Feedforward – #2 Example • Ex: Celtic melody generation [Sturm et al., 2016] • Celtic Folk Music Corpus (Melodies) • Text Encoding (ABC Notation) X: 1 T: A Cup Of Tea R: reel M: 4/4 L: 1/8 K: Amix |:eA (3AAA g2 fg|eA (3AAA BGGf|eA (3AAA g2 fg|1afge d2 gf:|2afge d2 cd|| |:eaag efgf|eaag edBd|eaag efge|afge dgfg:| Deep Learning – Music Generation – 2019 17 Jean-Pierre Briot

RNN Celtic Melody Generation • Iterated generation – Note by Note – Arbitrary Length • Ex: Celtic melody generation [Sturm et al., 2016] • Celtic Folk Music Corpus (Melodies) • Text Encoding (ABC Notation) • Ex. of Melody Generated Played by a human accordeonist Deep Learning – Music Generation – 2019 18 Jean-Pierre Briot

#3 Limitation – Variability • No Variability in the Generation • Because Neural Networks are Deterministic – Same Input -> Same Output – Same First Note -> Same Whole Melody Generated • Solution: – Sampling Deep Learning – Music Generation – 2019 19 Jean-Pierre Briot

#3 Limitation – Variability – Solution: Sampling • Input Representation: One-Hot Encoding – Corresponds to a Piano Roll Representation • Softmax Ouput Layer • Classification Task (between possible Notes) Deep Learning – Music Generation – 2019 20 Jean-Pierre Briot

Sampling • Deterministic Strategy: – Choose the Class (Note/Pitch) with the Highest Probability • Sampling (Variability) – Sample within Possible Notes (Classes) (following the Probability Distribution) np.random.multinomial(1, note_one_hot_encoding) Deep Learning – Music Generation – 2019 21 Jean-Pierre Briot

RNN Can Have Ouput Different Than Input Bach Chorale Counterpoint Generation – RNN Version Input layer Hidden layer Output layer Alto Voice Soprano Tenor Voice Voice … Bass Voice Deep Learning – Music Generation – 2019 22 Jean-Pierre Briot

RNN Encoder-Decoder Deep Learning – Music Generation – 2019 23 Jean-Pierre Briot

RNN Autoencoder : RNN Encoder-Decoder Deep Learning – Music Generation – 2019 24 Jean-Pierre Briot

From Speech to Text [Chung et al., 2016] Deep Learning – Music Generation – 2019 25 Jean-Pierre Briot

Translation Sequence to Sequence Deep Learning – Music Generation – 2019 26 Jean-Pierre Briot

Translation Deep Learning – Music Generation – 2019 27 Jean-Pierre Briot

Variational RNN Encoder-Decoder VRAE [Fabius and van Amersfoort, 2015] Deep Learning – Music Generation – 2019 28 Jean-Pierre Briot

MusicVAE [Roberts et al., 2018] Deep Learning – Music Generation – 2019 29 Jean-Pierre Briot

MusicVAE [Roberts et al., 2018] • Hierarchical – Conductor RNN – Bottom RNN • Longer term generation • Structure • Translation • Interpolation (morphing) • Averaging of some points • Addition or subtraction of an attribute vector capturing a given characteristic – This attribute vector is computed as the average latent vector for a collection of examples sharing that attribute (characteristic) Deep Learning – Music Generation – 2019 30 Jean-Pierre Briot

MusicVAE [Roberts et al., 2018] • Averaging the latent space Deep Learning – Music Generation – 2019 31 Jean-Pierre Briot

MusicVAE [Roberts et al., 2018] data space latent space • Comparing Interpolation – In the data space (melodies) – In the latent space Deep Learning – Music Generation – 2019 32 Jean-Pierre Briot

MusicVAE [Roberts et al., 2018] • Comparing Interpolation – In the data space (melodies) – In the latent space Deep Learning – Music Generation – 2019 33 Jean-Pierre Briot

MusicVAE [Roberts et al., 2018] https://www.youtube.com/watch?v=G5JT16flZwM Deep Learning – Music Generation – 2019 34 Jean-Pierre Briot

MusicVAE [Roberts et al., 2018] • Adding a high note density attribute vector Deep Learning – Music Generation – 2019 35 Jean-Pierre Briot

BeatBlender in TensorFlow.js MusicVAE [Roberts et al., 2018] https://experiments.withgoogle.com/ai/beat-blender/view/ Deep Learning – Music Generation – 2019 36 Jean-Pierre Briot

LatentLoops in TensorFlow.js MusicVAE [Roberts et al., 2018] https://teampieshop.github.io/latent-loops/ Deep Learning – Music Generation – 2019 37 Jean-Pierre Briot