Self-Attention For Generative Models Ashish Vaswani and Anna Huang - PowerPoint PPT Presentation

Dec 28, 2022 •137 likes •1.3k views

Self-Attention For Generative Models Ashish Vaswani and Anna Huang Joint work with: Noam Shazeer, Niki Parmar, Lukasz Kaiser, Illia Polosukhin, Llion Jones, Justin Gilmer, David Bieber, Jonathan Frankle, Jakob Uszkoreit, and others. Learning

Continuations to given initial motif Given motif RNN-LSTM
Continuations to given initial motif Given motif RNN-LSTM
Continuations to given initial motif Given motif RNN-LSTM Transformer
Continuations to given initial motif Given motif RNN-LSTM Transformer
Continuations to given initial motif Given motif RNN-LSTM Transformer Music Transformer
Continuations to given initial motif Given motif RNN-LSTM Transformer Music Transformer
Self-Similarity in Music
Sample from Music Transformer
Attention: a weighted average TimeShift100 TimeShift100 TimeShift30 NoteOn60 TimeShift20 NoteOn62 TimeShift90 NoteOff62 NoteOff60 TimeShift90 TimeShift100 TimeShift100 TimeShift30 NoteOn60 TimeShift20 NoteOn62 TimeShift90 NoteOff62 NoteOff60 TimeShift90
Attention: a weighted average TimeShift100 TimeShift100 TimeShift30 NoteOn60 TimeShift20 NoteOn62 TimeShift90 NoteOff62 NoteOff60 TimeShift90 TimeShift100 TimeShift100 TimeShift30 NoteOn60 TimeShift20 NoteOn62 TimeShift90 NoteOff62 NoteOff60 TimeShift90
Convolution: Different linear transformations by relative position. TimeShift100 TimeShift100 TimeShift30 NoteOn60 TimeShift20 NoteOn62 TimeShift90 NoteOff62 NoteOff60 TimeShift90 TimeShift100 TimeShift100 TimeShift30 NoteOn60 TimeShift20 NoteOn62 TimeShift90 NoteOff62 NoteOff60 TimeShift90
Relative attention (Shaw et al, 2018) Multihead attention + convolution? TimeShift100 TimeShift100 TimeShift30 NoteOn60 TimeShift20 NoteOn62 TimeShift90 NoteOff62 NoteOff60 TimeShift90 TimeShift100 TimeShift100 TimeShift30 NoteOn60 TimeShift20 NoteOn62 TimeShift90 NoteOff62 NoteOff60 TimeShift90
Closer look at attention QE rT
Closer look at relative attention Modulated by relative positions 0,0 0 0,1 1 0,2 2 1,0 -1 1,1 0 1,2 1 2,0 -2 2,1 -1 2,2 0 QE rT
Machine Translation (Shaw et al, 2018) Model Position BLEU BLEU Representati En-De En-Fr on Transformer Big Absolute 27.9 41.3 Transformer Big Relative 29.2 41.5
Previous work O(L 2 D): 8.5 GB per layer (Shaw et al, 2018) Per layer, L=2048, D=512 Relative embeddings Multiply by Q Relative distances 0 -1 0 -2 -1 0
Our formulation O(LD): 4.2 MB per layer Per layer, L=2048, D=512 Absolute by absolute Absolute by relative Skew Pad Reshape Slice i q
Goal of skewing procedure Indexed by absolute by absolute absolute by relative
Skewing to reduce relative memory from O(L 2 D) to O(LD) Per layer, L=2048, D=512 0 -1 0 Previous work -2 -1 0 O(L 2 D): 8.5 GB Relative Multiply by Q i q embeddings E r Skew Our work O(LD): 4.2 MB QE rT S rel Reshape Slice Pad i q Directly multiply by Q Per layer, L=2048, D=512 O(L 2 D): 8.5 GB Q skew(QE T ) O(LD): 4.2 MB (ours) E T
A Jazz sample from Music Transformer
A Jazz sample from Music Transformer
Convolutions and Translational Equivariance 0 32 0 32 0.5 0.5 32 32
Relative positions Translational Equivariance 0 32 0 32 0.5 0.5 32 32
Relative Attention And Graphs

Recommend

Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models

Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models Language Models Language models are generative models of text s ~ P(x) The Malfoys! said Hermione. Harry was watching him. He looked like

1.18k views • 37 slides

Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com Nov. 2017 Figures

Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com Nov. 2017 Figures adapted from NIPS 2016 Tutorial Generative Adversarial Networks Generative Models: Learning the Distributions Discriminative: learns the likelihood

1.37k views • 77 slides

Attention Models Focus on parts of input Olof Mogren Improves NN performance on different

Attention Models Attention Models Focus on parts of input Olof Mogren Improves NN performance on different tasks Chalmers University of Technology IBM1 attention mechanism (1980s) Feb 2016 Attention Models Arxiv 2016

348 views • 6 slides

Probabilistic Models of Cognition: Generative models Table of Contents Chapter

Probabilistic Models of Cognition: Generative models Table of Contents Chapter Content Generative Model Example: Plinko Machine Working Model can be used captures some for simulation structure of the world in useful way

825 views • 78 slides

Adversarial BoltzmannMachines Belief Nets Networks Variational NADE

Unsupervised Learning Non-probabilistic Models Probabilistic Sparse Coding (Generative) Models Autoencoders Others (e.g. k-means) Tractable Models Non-Tractable Models Generative Fully observed Adversarial

815 views • 58 slides

Classification with generative models 2 DSE 210 Classification with parametrized models

Classification with generative models 2 DSE 210 Classification with parametrized models Classifiers with a fixed number of parameters can represent a limited set of functions. Learning a model is about picking a good approximation. Typically the

427 views • 39 slides

Deep Generative models for Inverse Problems Alex Dimakis joint work with Ashish Bora, Dave Van

Deep Generative models for Inverse Problems Alex Dimakis joint work with Ashish Bora, Dave Van Veen and Ajil Jalal, Sriram Vishwanath and Eric Price, UT Austin Outline Generative Models Using generative models for Inverse

1.17k views • 101 slides

Augmented Statistical Models: Exploiting Generative Models in Discriminative Classifiers Martin

Augmented Statistical Models: Exploiting Generative Models in Discriminative Classifiers Martin Layton & Mark Gales 9 December 2005 Cambridge University Engineering Department NIPS 2005 Augmented Statistical Models: Exploiting Generative

726 views • 25 slides

Invertible Generative Models for Inverse Problems Mitigating Representation Error and Dataset Bias

Invertible Generative Models for Inverse Problems Mitigating Representation Error and Dataset Bias M. Asim, M. Daniels, O. Leong, P . Hand, A. Ahmed Inverse Problems with Generative Models as Image Priors Inverse Problems with Generative Models

369 views • 22 slides

Modals, conditionals, and probabilistic generative models Topic 1: intro to probability &

Modals, conditionals, and probabilistic generative models Topic 1: intro to probability & generative models; a bit on modality Dan Lassiter, Stanford Linguistics Universit de Paris VII, 25/11/19 4 lectures: The plan 1. probability,

694 views • 56 slides

LEARNING GENERATIVE MODELS ACROSS INCOMPARABLE SPACES Cha harlot otte Bunne unne , David

LEARNING GENERATIVE MODELS ACROSS INCOMPARABLE SPACES Cha harlot otte Bunne unne , David Alvarez-Melis, Andreas Krause, Stefanie Jegelka Pos oster #173 173 Poster #173 173 & Charlotte Bunne Generative Modeling generative = network

250 views • 13 slides

Detecting Communities of Commuters: Graph Based Techniques vs Generative Models Ashish Dandekar,

Detecting Communities of Commuters: Graph Based Techniques vs Generative Models Ashish Dandekar, St ephane Bressan, Talel Abdessalem, Huayu Wu, Wee Siong Ng September 7, 2016 1 Introduction Related Work Generative Models Experiments

786 views • 44 slides

Outline Latent Variable Generative Models Cooperative Vector Quantizer Model Model

2/4/2020 CS 3750 Advanced Machine Learning Latent Variable Generative Models II Ahmad Diab AHD23@cs.pitt.edu Feb 4, 2020 Based on slides of Professor Milos Hauskrecht Outline Latent Variable Generative Models Cooperative Vector

435 views • 20 slides

Learning the Structure of Generative Models without Labeled Data Stephen Bach Bryan He

Learning the Structure of Generative Models without Labeled Data Stephen Bach Bryan He Alex Ratner Chris R Stanford University This Talk We study structure learning for generative models in which a latent variable

599 views • 33 slides

Modals, conditionals, and probabilistic generative models Topic 2: Indicative conditionals

Modals, conditionals, and probabilistic generative models Topic 2: Indicative conditionals Separating semantics from reasoning Dan Lassiter, Stanford Linguistics Universit de Paris VII, 2/12/19 Overall plan 1. probability, generative

997 views • 52 slides

Probability Functional Descent: A Unifying Perspective on GANs, VI, and RL Casey Chu <

Probability Functional Descent: A Unifying Perspective on GANs, VI, and RL Casey Chu < caseychu@stanford.edu > Jose Blanchet Peter Glynn Deep generative models Deep generative models Variational inference Deep generative models

275 views • 16 slides

CSC 411: Lecture 08: Generative Models for Classification Class based on Raquel Urtasun &

CSC 411: Lecture 08: Generative Models for Classification Class based on Raquel Urtasun & Rich Zemels lectures Sanja Fidler University of Toronto Feb 5, 2016 Urtasun, Zemel, Fidler (UofT) CSC 411: 08-Generative Models Feb 5, 2016 1 /

475 views • 23 slides

Generative Models Jo ao Paulo Papa and Marcos Cleison Silva Santana December 17, 2019 UNESP -

Generative Models Jo ao Paulo Papa and Marcos Cleison Silva Santana December 17, 2019 UNESP - S ao Paulo State University School of Sciences, Departament of Computing Bauru, SP - Brazil Outline 1. Generative versus Discriminative Models

513 views • 28 slides

Conditional Generative Adversarial Networks (and a brief look at image-to-image translation)

Conditional Generative Adversarial Networks (and a brief look at image-to-image translation) Final Presentation Peter Bromley Generative Models What is generative modeling? Data: Samples from high dimensional probability distribution p real

621 views • 25 slides

Generative Adversarial Nets(GANs) Troy Cary and Chenzhi Zhao A generative adversarial net is

Generative Adversarial Nets(GANs) Troy Cary and Chenzhi Zhao A generative adversarial net is a type of neural net, used in deep learning/machine learning problems The goal of a GAN is to train two simultaneous models: a generative model

441 views • 18 slides

Bias and Generalization in Deep Generative Models Shengjia Zhao, Hongyu Ren, Arianna Yuan,

Bias and Generalization in Deep Generative Models Shengjia Zhao*, Hongyu Ren*, Arianna Yuan, Jiaming Song, Noah Goodman and Stefano Ermon *equal contribution Success in Generative Modeling of Images Brock A, et al. "Large scale gan

350 views • 19 slides

Attend, Infer, Repeat: Fast Scene Understanding with Generative Models S.M. Eslami,N. Heess, T.

Attend, Infer, Repeat: Fast Scene Understanding with Generative Models S.M. Eslami,N. Heess, T. Weber, Y. Tassa, D. Szepesvari, K.Kavukcuoglu, G. E. Hinton Nicolas Brandt nbrandt@cs.toronto.edu Origins Structured generative methods : Deep

1.13k views • 19 slides

Adaptive Density Estimation for Generative Models Thomas Konstantin Karteek Cordelia Jakob

Adaptive Density Estimation for Generative Models Thomas Konstantin Karteek Cordelia Jakob Lucas Shmelkov Alahari Schmid Verbeek Now at Huawei Generative modelling Goal Given samples from target distribution p , train a model

438 views • 30 slides

Learning Deep Generative Models Inference & Representation Lecture 12 Rahul G. Krishnan

Introduction Variational Inference Deep Generative Models Summary Learning Deep Generative Models Inference & Representation Lecture 12 Rahul G. Krishnan Fall 2015 Rahul G. Krishnan Learning Deep Generative Models Introduction

1.22k views • 83 slides