GANs + Final practice questions Lecture 23 CS 753 Instructor: Preethi Jyothi
Final Exam Syllabus 1. WFST algorithms/WFSTs used in ASR 2. HMM algorithms/EM/Tied state Triphone models 3. DNN-based acoustic models 4. N-gram/Smoothing/RNN language models 5. End-to-end ASR (CTC, LAS, RNN-T) 6. MFCC feature extraction 7. Search & Decoding 8. HMM-based speech synthesis models 9. Multilingual ASR 10. Speaker Adaptation 11. Discriminative training of HMMs Questions can be asked on any of the 11 topics listed above. You will be allowed a single A-4 cheat sheet of handwritten notes ; content on both sides permitted.
Final Project Deliverables 4-5 page final report: • ✓ Task definition, Methodology, Prior work, Implementation Details, Experimental Setup, Experiments and Discussion, Error Analysis (if any), Summary Short talk summarizing the project: • ✓ Each team will get 8-10 minutes for their presentation ≈ and � 5 minutes for Q/A ✓ Clearly demarcate which team member worked on what part
Final Project Grading Break-up of 20 points: • 6 points for the report • 4 points for the presentation • 6 points for Q/A • 4 points for overall evaluation of the project •
Final Project Schedule Presentations will be held on Nov 23rd and Nov 24th • The final report in pdf format should be sent to • pjyothi@cse.iitb.ac.in before Nov 24th The order of presentations will be decided on a lottery basis • and shared via Moodle before Nov 9th
Generative Adversarial Networks (GANs) D ( x ) Training process is formulated as a • game between a generator network Discriminator and a discriminative network Objective of the generator: Create • samples that seem to be from the same distribution as the training x real x = G ( z ) data Objective of the discriminator: • Generator Examine a generated sample and distinguish between fake or real samples • The generator tries to fool the discriminator network Z
<latexit sha1_base64="7VI+8ZF3gmwRBJn+eQLkMiseN2U=">ACHicbVDLSsNAFJ3UV62vqEsXDhahgpSkCrosWqgLFxXsA5oQJtNpO3QyCTMTsYQu3fgrblwo4tZPcOfOGmz0OqBYQ7n3Mu9/gRo1JZ1peRW1hcWl7JrxbW1jc2t8ztnZYMY4FJE4csFB0fScIoJ01FSOdSBAU+Iy0/dFl6rfviJA05LdqHBE3QANO+xQjpSXP3HcCdO/VoRNQ7tX0h9QI5ZcT0r1Y1g78syiVbamgH+JnZEiyNDwzE+nF+I4IFxhqTs2lak3AQJRTEjk4ITSxIhPEID0tWUo4BIN5keMoGHWunBfij04wpO1Z8dCQqkHAe+rkwXlfNeKv7ndWPVP3cTyqNYEY5ng/oxgyqEaSqwRwXBio01QVhQvSvEQyQVjq7g7Bnj/5L2lVyvZJuXJzWqxeZHkwR4ACVgzNQBVegAZoAgwfwBF7Aq/FoPBtvxvusNGdkPbvgF4yPb3ZxmFI=</latexit> <latexit sha1_base64="VTwlHXywjOChz5LfJXs73m4kjK8=">ACU3icbVFda9RAFJ2kVetq7aqPvlxchATtklTBvghFV+qDxXctrAJy2T27u7YySTM3LS7DfmPIvjgH/HFB539QGzrgYHDOecyd85kpZKWouiH529s3rp9Z+tu69797Qc7YePjm1RGYF9UajCnGbcopIa+yRJ4WlpkOeZwpPs7N3CPzlHY2WhP9O8xDTnEy3HUnBy0rD9JSGcUX0xRYPQJzmgqu6o9NcPgCeiG8Wkmr983w3oGidTQa2Cwm6hiAr1gFqbw/Erm8q8bxLDrIofBZRiG6bDdibrREnCTxGvSYWscDdvfklEhqhw1CcWtHcRSWnNDUmhsGklcWSizM+wYGjmudo03rZSQPnDKCcWHc0QRL9d+JmufWzvPMJRe72+veQvyfN6hovJ/WUpcVoRari8aVAipgUTCMpEFBau4IF0a6XUFMueGC3De0XAnx9SfJMd73fhld+/Tq87B23UdW+wJe8oCFrPX7IB9YEeszwT7yn6y3x7zvnu/fN/fXEV9bz3zmF2Bv/0H1zCvmQ=</latexit> Generative Adversarial Networks max G min D L ( G, D ) where L ( G, D ) = E x ∈ D [ − log D ( x )] + E z [ − log(1 − D ( G ( z )))] • Cost function of the generator is the opposite of the discriminator’s • Minimax game: The generator and discriminator are playing a zero-sum game against each other
Training Generative Adversarial Networks for number of training iterations do for k steps do • Sample minibatch of m noise samples { z (1) , . . . , z ( m ) } from noise prior p g ( z ) . • Sample minibatch of m examples { x (1) , . . . , x ( m ) } from data generating distribution p data ( x ) . • Update the discriminator by ascending its stochastic gradient: m 1 h ⇣ x ( i ) ⌘ ⇣ ⇣ ⇣ z ( i ) ⌘⌘⌘i X log D + log 1 � D G . r θ d m i =1 end for • Sample minibatch of m noise samples { z (1) , . . . , z ( m ) } from noise prior p g ( z ) . • Update the generator by descending its stochastic gradient: m 1 ⇣ ⇣ ⇣ z ( i ) ⌘⌘⌘ X log 1 � D G . r θ g m i =1 end for The gradient-based updates can use any standard gradient-based learning rule. We used momen- tum in our experiments. Image from [Goodfellow16]: https://arxiv.org/pdf/1701.00160.pdf
<latexit sha1_base64="axrtjiZYFfZltVUN2PIU2Ef0eo=">ACKnicbVDLSgMxFM3UV62vqks3wSK0oGWmCroRqrbUhUgF+4CZoWTStA3NPEgyQjvM97jxV9x0oRS3fojpQ9DqgcDJOfdy7z1OwKiQuj7WEkvLK6tryfXUxubW9k56d68u/JBjUsM+83nTQYIw6pGapJKRZsAJch1Gk7/ZuI3ngX1Pce5SAgtou6Hu1QjKSWukry0WyhxGL7uJWNP1wN6qU7+M4WzmGpRy8hN9qWVUMY2ieWMzvwlK2kh3mcnYrndHz+hTwLzHmJAPmqLbSI6vt49AlnsQMCWEaeiDtCHFJMSNxygoFCRDuoy4xFfWQS4QdTU+N4ZFS2rDjc/U8Cafqz4IuUIMXEdVTrYWi95E/M8zQ9m5sCPqBaEkHp4N6oQMSh9OcoNtygmWbKAIwpyqXSHuIY6wVOmVAjG4sl/Sb2QN07zhYezTPF6HkcSHIBDkAUGOAdFcAuqoAYweAav4A28ay/aSBtrH7PShDbv2Qe/oH1+ASE1pcM=</latexit> <latexit sha1_base64="LGWrBEXN/oGBn/UGMdYvD/mje+0=">ACL3icbVDLSgMxFM34rPVdekmWIQpaJmpgm6Eoq1IVLBPqAdSibNtKGZB0lGaIf5Izf+Sjcirj1L0wfgrYeCJycy/3mMHjApGK/awuLS8spqYi25vrG5tZ3a2a0KP+SYVLDPfF63kSCMeqQiqWSkHnCXJuRmt27Gvm1R8IF9b0H2Q+I5aKORx2KkVRSK3XdJHsYsSi27gVjT/cjUrFuzjWS0ewkIEX8EctqopBDBtN5negbsJjWNBL+iCTyVitVNrIGmPAeWJOSRpMUW6lhs2j0OXeBIzJETDNAJpRYhLihmJk81QkADhHuqQhqIecomwovG9MTxUShs6PlfPk3Cs/u6IkCtE37V5Wh1MeuNxP+8RidcyuiXhBK4uHJICdkUPpwFB5sU06wZH1FEOZU7QpxF3GEpYo4qUIwZ0+eJ9Vc1jzJ5u5P0/nLaRwJsA8OgA5McAby4AaUQVg8ASG4A28a8/ai/ahfU5KF7Rpzx74A+3rG3KWprc=</latexit> Better objective for the generator Problem of saturation: If the • generated sample is really poor, the generator’s cost is relatively flat Original cost • L GEN ( G, D ) = E z [log(1 − D ( G ( z )))] Modified cost • L GEN ( G, D ) = E z [ − log D ( G ( z ))]
Large (& growing!) list of GANs Image from https://github.com/hindupuravinash/the-gan-zoo
D ( x ) Conditional GANs x real • Generator and discriminator x = G ( z ) receive some additional conditioning information C Z
Image-to-image Translation using C-GANs { } Labels to Street Scene Labels to Facade BW to Color input output Aerial to Map input output input output Day to Night Edges to Photo input output input output input output Image from Isola et al., CVPR 2017, https://arxiv.org/pdf/1611.07004.pdf
Text-to-Image Synthesis this small bird has a pink this magnificent fellow is breast and crown, and black almost all black with a red primaries and secondaries. crest, and white cheek patch. this white and yellow flower the flower has petals that are bright pinkish purple have thin white petals and a with white stigma round yellow stamen Image from Reed et al., ICML 2016, https://arxiv.org/pdf/1605.05396.pdf
Text-to-Image Synthesis This flower has small, round violet This flower has small, round violet petals with a dark purple center petals with a dark purple center Generator Network Discriminator Network Image from Reed et al., ICML 2016, https://arxiv.org/pdf/1605.05396.pdf
Three Speech Applications of GANs
GANs for speech synthesis Generator Discriminator: • produces synthesised speech Binary OR classifier which the Discriminator distinguishes from real speech Linguistic features Natural samples During synthesis, a • Generator: MSE AND random noise + linguistic features Noise Predicted samples generates speech Image from Yang et al., “SPSS using GANs”, 2017
SEGAN: GANs for speech enhancement Enhancement: Given an input noisy • ˜ x signal � , we want to clean it to obtain an x enhanced signal � ˜ x z Generator G will take both � and � as • inputs; G is fully convolutional Image from https://arxiv.org/pdf/1703.09452.pdf
Voice Conversion Using Cycle-GANs Image from https://arxiv.org/abs/1711.11293
Practice Questions
� HMM 101 A water sample collected from Powai lake is either Clean or Polluted. However, this information is hidden from us and all we can observe is whether the water is muddy, clear, odorless or cloudy. We start at time step 1 in the Clean state. The HMM below models this problem. Let qt and Ot denote the state and observation at time step t, respectively. 0.2 O 2 a)What is P( � = clear)? 0.8 ∣ O 2 q 2 b)What is P( � = Clean � = clear)? Clean Polluted Pr(muddy) = 0.5 O 200 Pr(muddy) = 0.1 c)What is P( � = cloudy)? Pr(clear) = 0.1 Pr(clear) = 0.5 Pr(odorless) = 0.2 Pr(odorless) = 0.2 Pr(cloudy) = 0.2 Pr(cloudy) = 0.2 d)What’s the most likely sequence of states for the following observation O 1 O 2 sequence: { � = clear, � = clear, 0.8 O 3 O 4 O 5 = clear, � = clear, � = clear}? 0.2
HMM 101 Say that we are now given a modified HMM for the water samples as shown below. Initial probabilities and transition probabilities are shown next to the arcs. (Note: You do not need to use the Viterbi algorithm to answer the next two questions.) a) What is the most likely sequence of 0.9 0.9 states given a sequence of three observations: {muddy, muddy, muddy}? 0.1 0.01 0.99 Clean Polluted Pr(muddy) = 0.51 Pr(muddy) = 0.49 b) Say we observe a very long Pr(clear) = 0.49 Pr(clear) = 0.51 0.1 sequence of “muddy” (e.g. 10 million “muddy” in a row). What happens to the most likely state sequence then?
Recommend
More recommend
