bregman and wasserstein with applications to generative
play

Bregman and Wasserstein, with Applications to Generative Adversarial - PowerPoint PPT Presentation

Dec 24, 2023 •123 likes •1.62k views

Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Bregman and Wasserstein, with Applications to Generative Adversarial Networks (GANs) and

  1. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions What is Bregman Divergence Function? BDF D φ ( x , y ) Let φ : R d �→ R be a strictly convex, differentiable function Then, D φ : R d × R d �→ R is defined as D φ ( x , y ) = φ ( x ) − φ ( y ) − � x − y , ∇ φ ( y ) � . For any x , y ∈ R d , D φ ( x , y ) ≥ 0, the equality holds iff x = y . 6 / 50

  2. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Some examples of BDFs φ ( x ) = x 2 , then D φ ( x , y ) = ( x − y ) 2 7 / 50

  3. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Some examples of BDFs φ ( x ) = x 2 , then D φ ( x , y ) = ( x − y ) 2 Let p . = ( p 1 , . . . , p d ) be a probability distribution 7 / 50

  4. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Some examples of BDFs φ ( x ) = x 2 , then D φ ( x , y ) = ( x − y ) 2 Let p . = ( p 1 , . . . , p d ) be a probability distribution j =1 p j = 1, with φ ( p ) . � d = � d j =1 p j log p j (negative Shannon entropy) is strictly convex on the d -simplex. 7 / 50

  5. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Some examples of BDFs φ ( x ) = x 2 , then D φ ( x , y ) = ( x − y ) 2 Let p . = ( p 1 , . . . , p d ) be a probability distribution j =1 p j = 1, with φ ( p ) . � d = � d j =1 p j log p j (negative Shannon entropy) is strictly convex on the d -simplex. Let q = ( q 1 , . . . , q d ) be another probability distribution 7 / 50

  6. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Some examples of BDFs φ ( x ) = x 2 , then D φ ( x , y ) = ( x − y ) 2 Let p . = ( p 1 , . . . , p d ) be a probability distribution j =1 p j = 1, with φ ( p ) . � d = � d j =1 p j log p j (negative Shannon entropy) is strictly convex on the d -simplex. Let q = ( q 1 , . . . , q d ) be another probability distribution d d � � D φ ( p , q ) = p j log p j − q j log q j j =1 j =1 −� p − q , ∇ φ ( q ) � d � = p j log ( p j / q j ) , j =1 is the KL-divergence between p and q 7 / 50

  7. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Proof of sufficiency Let Y be any G -measurable random variable, and Y ∗ . = E [ X |G ]. Then E [ D φ ( X , Y )] − E [ D φ ( X , Y ∗ )] E [ φ ( Y ∗ ) − φ ( Y ) − � X − Y , ∇ φ ( Y ) � = + � X − Y ∗ , ∇ φ ( Y ∗ ) � ] . 8 / 50

  8. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Proof of sufficiency Let Y be any G -measurable random variable, and Y ∗ . = E [ X |G ]. Then E [ D φ ( X , Y )] − E [ D φ ( X , Y ∗ )] E [ φ ( Y ∗ ) − φ ( Y ) − � X − Y , ∇ φ ( Y ) � = + � X − Y ∗ , ∇ φ ( Y ∗ ) � ] . Notice E [ � X − Y , ∇ φ ( Y ) � ] = E [ E [ � X − Y , ∇ φ ( Y ) �|G ]] = E [ � Y ∗ − Y , ∇ φ ( Y ) � ] 8 / 50

  9. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Proof of sufficiency Let Y be any G -measurable random variable, and Y ∗ . = E [ X |G ]. Then E [ D φ ( X , Y )] − E [ D φ ( X , Y ∗ )] E [ φ ( Y ∗ ) − φ ( Y ) − � X − Y , ∇ φ ( Y ) � = + � X − Y ∗ , ∇ φ ( Y ∗ ) � ] . Notice E [ � X − Y , ∇ φ ( Y ) � ] = E [ E [ � X − Y , ∇ φ ( Y ) �|G ]] = E [ � Y ∗ − Y , ∇ φ ( Y ) � ] Thus E [ � X − Y ∗ , ∇ φ ( Y ∗ ) � ] = 0, 8 / 50

  10. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Proof of sufficiency Let Y be any G -measurable random variable, and Y ∗ . = E [ X |G ]. Then E [ D φ ( X , Y )] − E [ D φ ( X , Y ∗ )] E [ φ ( Y ∗ ) − φ ( Y ) − � X − Y , ∇ φ ( Y ) � = + � X − Y ∗ , ∇ φ ( Y ∗ ) � ] . Notice E [ � X − Y , ∇ φ ( Y ) � ] = E [ E [ � X − Y , ∇ φ ( Y ) �|G ]] = E [ � Y ∗ − Y , ∇ φ ( Y ) � ] Thus E [ � X − Y ∗ , ∇ φ ( Y ∗ ) � ] = 0, And E [ D φ ( X , Y )] − E [ D φ ( X , Y ∗ )] = E [ D φ ( Y ∗ , Y )] ≥ 0 . 8 / 50

  11. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions More facts about BDFs Introduced and studied in the context of projection (Csiszar (1975)) 9 / 50

  12. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions More facts about BDFs Introduced and studied in the context of projection (Csiszar (1975)) Pythagoras theorem holds for BDF (Censor and Lent (1981)) 9 / 50

  13. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions More facts about BDFs Introduced and studied in the context of projection (Csiszar (1975)) Pythagoras theorem holds for BDF (Censor and Lent (1981)) Bijection between family of exponential distributions and BDFs, via Legendre duality (Merugu, Banerjee, Dhillon, Ghosh (2003)) 9 / 50

  14. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions More facts about BDFs Introduced and studied in the context of projection (Csiszar (1975)) Pythagoras theorem holds for BDF (Censor and Lent (1981)) Bijection between family of exponential distributions and BDFs, via Legendre duality (Merugu, Banerjee, Dhillon, Ghosh (2003)) Widely applied to data analysis and machine learning, such as K-means clustering 9 / 50

  15. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions More facts about BDFs Introduced and studied in the context of projection (Csiszar (1975)) Pythagoras theorem holds for BDF (Censor and Lent (1981)) Bijection between family of exponential distributions and BDFs, via Legendre duality (Merugu, Banerjee, Dhillon, Ghosh (2003)) Widely applied to data analysis and machine learning, such as K-means clustering Well adopted in convex optimization 9 / 50

  16. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Generator Network [Goodfellow et al., 2014] Generate the samples according to P θ . 10 / 50

  17. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Generator Network [Goodfellow et al., 2014] Generate the samples according to P θ . The real samples X is inaccessible. 10 / 50

  18. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Generator Network [Goodfellow et al., 2014] Generate the samples according to P θ . The real samples X is inaccessible. Generate more compelling copies of X . 10 / 50

  19. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Generator Network [Goodfellow et al., 2014] Generate the samples according to P θ . The real samples X is inaccessible. Generate more compelling copies of X . inaccessible generate 10 / 50

  20. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions How to Make Generator Network Better? A knowledgeable mentor (discriminator)— 11 / 50

  21. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Discriminator Network [Goodfellow et al., 2014] Determines whether the samples are generated or not. 12 / 50

  22. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Discriminator Network [Goodfellow et al., 2014] Determines whether the samples are generated or not. has access to the real samples X . 12 / 50

  23. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Discriminator Network [Goodfellow et al., 2014] Determines whether the samples are generated or not. has access to the real samples X . optimizes the generator network by identifying faked samples. 12 / 50

  24. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Discriminator Network [Goodfellow et al., 2014] Determines whether the samples are generated or not. has access to the real samples X . optimizes the generator network by identifying faked samples. pass! fail again! 12 / 50

  25. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Graphical Model 13 / 50

  26. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Generative modeling The procedure of generative modeling is to construct a class of suitable parametric probability distributions P θ . 14 / 50

  27. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Generative modeling The procedure of generative modeling is to construct a class of suitable parametric probability distributions P θ . Generates latent variable Z ∈ Z with a fixed probability distribution P Z . 14 / 50

  28. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Generative modeling The procedure of generative modeling is to construct a class of suitable parametric probability distributions P θ . Generates latent variable Z ∈ Z with a fixed probability distribution P Z . P Z is known and simple , e.g., uniform distribution. 14 / 50

  29. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Generative modeling The procedure of generative modeling is to construct a class of suitable parametric probability distributions P θ . Generates latent variable Z ∈ Z with a fixed probability distribution P Z . P Z is known and simple , e.g., uniform distribution. Generates a sequence of parametric functions g θ : Z → X . 14 / 50

  30. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Generative modeling The procedure of generative modeling is to construct a class of suitable parametric probability distributions P θ . Generates latent variable Z ∈ Z with a fixed probability distribution P Z . P Z is known and simple , e.g., uniform distribution. Generates a sequence of parametric functions g θ : Z → X . g θ is complicated but structured . 14 / 50

  31. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Generative modeling The procedure of generative modeling is to construct a class of suitable parametric probability distributions P θ . Generates latent variable Z ∈ Z with a fixed probability distribution P Z . P Z is known and simple , e.g., uniform distribution. Generates a sequence of parametric functions g θ : Z → X . g θ is complicated but structured . g θ is the reason why the generative modeling is powerful . 14 / 50

  32. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Generative modeling The procedure of generative modeling is to construct a class of suitable parametric probability distributions P θ . Generates latent variable Z ∈ Z with a fixed probability distribution P Z . P Z is known and simple , e.g., uniform distribution. Generates a sequence of parametric functions g θ : Z → X . g θ is complicated but structured . g θ is the reason why the generative modeling is powerful . Construct P θ as the probability distribution of g θ ( Z ). More specifically, � � � P θ ( dx ) = 1 { g θ ( z )= dx } P Z ( dz ) = E Z 1 { g θ ( Z )= dx } . Z 14 / 50

  33. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions GANs: different divergence functions GANs: 15 / 50

  34. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions GANs: different divergence functions GANs: LSGANs [Mao et al., 2016]: Least square loss. DRAGANs [Kodali et al., 2017]: Regret minimization. CGANs [Mirza and Osindero, 2014]: Conditional extension. InfoGANs [Chen et al., 2016]: Information-theoretic extension. ACGANs [Odena et al., 2017] Structured latent space. EBGANs [Zhao et al., 2016]: New perspective of the energy. BEGANs [Berthelot et al., 2017]: Auto-encoder extension. 15 / 50

  35. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions GANs: different divergence functions GANs: LSGANs [Mao et al., 2016]: Least square loss. DRAGANs [Kodali et al., 2017]: Regret minimization. CGANs [Mirza and Osindero, 2014]: Conditional extension. InfoGANs [Chen et al., 2016]: Information-theoretic extension. ACGANs [Odena et al., 2017] Structured latent space. EBGANs [Zhao et al., 2016]: New perspective of the energy. BEGANs [Berthelot et al., 2017]: Auto-encoder extension. GANs training: [Arjovsky and Bottou, 2017] 15 / 50

  36. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions GANs: different divergence functions GANs: LSGANs [Mao et al., 2016]: Least square loss. DRAGANs [Kodali et al., 2017]: Regret minimization. CGANs [Mirza and Osindero, 2014]: Conditional extension. InfoGANs [Chen et al., 2016]: Information-theoretic extension. ACGANs [Odena et al., 2017] Structured latent space. EBGANs [Zhao et al., 2016]: New perspective of the energy. BEGANs [Berthelot et al., 2017]: Auto-encoder extension. GANs training: [Arjovsky and Bottou, 2017] Wasserstein GANs: 15 / 50

  37. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions GANs: different divergence functions GANs: LSGANs [Mao et al., 2016]: Least square loss. DRAGANs [Kodali et al., 2017]: Regret minimization. CGANs [Mirza and Osindero, 2014]: Conditional extension. InfoGANs [Chen et al., 2016]: Information-theoretic extension. ACGANs [Odena et al., 2017] Structured latent space. EBGANs [Zhao et al., 2016]: New perspective of the energy. BEGANs [Berthelot et al., 2017]: Auto-encoder extension. GANs training: [Arjovsky and Bottou, 2017] Wasserstein GANs: WGANs [Arjovsky et al., 2017]: Wasserstein L 1 divergence. Improved WGANs [Gulrajani et al., 2017]: Gradient Penalty. 15 / 50

  38. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Several Choices of Divergence The divergences to measure the difference between P and Q inlcude 16 / 50

  39. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Several Choices of Divergence The divergences to measure the difference between P and Q inlcude Kullback-Leibler divergence: � P ( dx ) � � KL ( P , Q ) = P ( dx ) · log . Q ( dx ) X 16 / 50

  40. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Several Choices of Divergence The divergences to measure the difference between P and Q inlcude Kullback-Leibler divergence: � P ( dx ) � � KL ( P , Q ) = P ( dx ) · log . Q ( dx ) X Jensen-Shannon (JS) divergence: JS ( P , Q ) = 1 � KL ( P , P + Q ) + KL ( Q , P + Q � ) . 2 2 2 16 / 50

  41. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Several Choices of Divergence The divergences to measure the difference between P and Q inlcude Kullback-Leibler divergence: � P ( dx ) � � KL ( P , Q ) = P ( dx ) · log . Q ( dx ) X Jensen-Shannon (JS) divergence: JS ( P , Q ) = 1 � KL ( P , P + Q ) + KL ( Q , P + Q � ) . 2 2 2 Wasserstein divergence/distance of order p � 1 � � p m ( x , y ) p π ( dx , dy ) W p ( P , Q ) = inf , π ∈ Π( P , Q ) X×X with m a metric such as m ( x , y ) = || x − y || q for q ≥ 1. 16 / 50

  42. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Discussions on these divergences Example: Given θ ∈ [0 , 1], assume that P and Q satisfy ∀ ( x , y ) ∈ P , x = 0 , y ∼ Uniform(0 , 1) , ∀ ( x , y ) ∈ Q , x = θ, y ∼ Uniform(0 , 1) , 17 / 50

  43. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Discussions on these divergences Example: Given θ ∈ [0 , 1], assume that P and Q satisfy ∀ ( x , y ) ∈ P , x = 0 , y ∼ Uniform(0 , 1) , ∀ ( x , y ) ∈ Q , x = θ, y ∼ Uniform(0 , 1) , As θ � = 0, KL ( P , Q ) = KL ( Q , P ) = + ∞ , JS ( P , Q ) = log(2) , W 1 ( P , Q ) = | θ | . 17 / 50

  44. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Discussions on these divergences Example: Given θ ∈ [0 , 1], assume that P and Q satisfy ∀ ( x , y ) ∈ P , x = 0 , y ∼ Uniform(0 , 1) , ∀ ( x , y ) ∈ Q , x = θ, y ∼ Uniform(0 , 1) , As θ � = 0, KL ( P , Q ) = KL ( Q , P ) = + ∞ , JS ( P , Q ) = log(2) , W 1 ( P , Q ) = | θ | . As θ = 0, KL ( P , Q ) = KL ( Q , P ) = JS ( P , Q ) = W 1 ( P , Q ) = 0 . 17 / 50

  45. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Remark KL is infinite when two distributions are disjoint; 18 / 50

  46. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Remark KL is infinite when two distributions are disjoint; JS has sudden jump, discontinuous at θ = 0; 18 / 50

  47. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Remark KL is infinite when two distributions are disjoint; JS has sudden jump, discontinuous at θ = 0; W 1 is continuous and relatively smooth; 18 / 50

  48. Bregman Divergence Function Generative Adversarial Networks (GANs) Wasserstein Divergence and GANs Relaxed Wasserstein Empirical Results Conclusions Remark KL is infinite when two distributions are disjoint; JS has sudden jump, discontinuous at θ = 0; W 1 is continuous and relatively smooth; Wasserstein L 1 divergence outperforms KL and JS divergences but lacks the flexibility. 18 / 50

  49. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Remedy: Relaxed Wasserstein Definition (G., Hong, Lin, and Yang 2018) The Relaxed Wasserstein divergence between the probability distributions P and Q is defined as � W D φ ( P , Q ) = inf D φ ( x , y ) π ( dx , dy ) , π ∈ Π( P , Q ) X×X where D φ is the Bregman divergence with a strictly convex and differentiable function φ : R d → R , i.e., D φ ( x , y ) = φ ( x ) − φ ( y ) − �∇ φ ( y ) , x − y � 1 W D φ ( P , Q ) ≥ 0 and = 0 iff P = Q almost everywhere. 19 / 50

  50. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Remedy: Relaxed Wasserstein Definition (G., Hong, Lin, and Yang 2018) The Relaxed Wasserstein divergence between the probability distributions P and Q is defined as � W D φ ( P , Q ) = inf D φ ( x , y ) π ( dx , dy ) , π ∈ Π( P , Q ) X×X where D φ is the Bregman divergence with a strictly convex and differentiable function φ : R d → R , i.e., D φ ( x , y ) = φ ( x ) − φ ( y ) − �∇ φ ( y ) , x − y � 1 W D φ ( P , Q ) ≥ 0 and = 0 iff P = Q almost everywhere. 2 W D φ ( P , Q ) is a metric, as it is asymmetric. 19 / 50

  51. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Remedy: Relaxed Wasserstein Definition (G., Hong, Lin, and Yang 2018) The Relaxed Wasserstein divergence between the probability distributions P and Q is defined as � W D φ ( P , Q ) = inf D φ ( x , y ) π ( dx , dy ) , π ∈ Π( P , Q ) X×X where D φ is the Bregman divergence with a strictly convex and differentiable function φ : R d → R , i.e., D φ ( x , y ) = φ ( x ) − φ ( y ) − �∇ φ ( y ) , x − y � 1 W D φ ( P , Q ) ≥ 0 and = 0 iff P = Q almost everywhere. 2 W D φ ( P , Q ) is a metric, as it is asymmetric. 3 W D φ ( P , Q ) includes W KL with φ ( x ) = − x ⊤ log( x ). 19 / 50

  52. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Relaxed Wasserstein as Divergence Question: Is W φ a good divergence? 20 / 50

  53. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Relaxed Wasserstein as Divergence Question: Is W φ a good divergence? Point 1: W φ ( P , Q ) should be small when P and Q are close. 20 / 50

  54. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Relaxed Wasserstein as Divergence Question: Is W φ a good divergence? Point 1: W φ ( P , Q ) should be small when P and Q are close. Requirement: W φ ( P , Q ) should be dominated by standard divergence, | P ( A ) − Q ( A ) | . TV ( P , Q ) := sup A ∈B 20 / 50

  55. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Relaxed Wasserstein as Divergence Question: Is W φ a good divergence? Point 1: W φ ( P , Q ) should be small when P and Q are close. Requirement: W φ ( P , Q ) should be dominated by standard divergence, | P ( A ) − Q ( A ) | . TV ( P , Q ) := sup A ∈B Point 2: W φ ( P n , P r ) → 0 as n → ∞ where P r is a true distribution P r and P n is the empirical distribution based on X = ( X 1 , X 2 , . . . , X n ) i . i . d . ∼ P r . 20 / 50

  56. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Relaxed Wasserstein as Divergence Question: Is W φ a good divergence? Point 1: W φ ( P , Q ) should be small when P and Q are close. Requirement: W φ ( P , Q ) should be dominated by standard divergence, | P ( A ) − Q ( A ) | . TV ( P , Q ) := sup A ∈B Point 2: W φ ( P n , P r ) → 0 as n → ∞ where P r is a true distribution P r and P n is the empirical distribution based on X = ( X 1 , X 2 , . . . , X n ) i . i . d . ∼ P r . Requirement: W φ ( P n , P r ) should have the moment estimate and concentration inequality, i.e., there exist α, β > 0 such that O ( n − α ) � � E W D φ ( P n , P r ) = (Moment Estimate) , O ( n − β ) Prob � W D φ ( P n , P r ) ≥ ǫ � = (Concentration Inequality) . 20 / 50

  57. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Dominated by TV and Standard Wasserstein Theorem (G., Hong, Lin, and Yang 2018) Assume that φ : X → R is a strictly convex and smooth function with an L-Lipschitz continuous factor, ≤ L [ diam ( X )] 2 · TV ( P , Q ) W D φ ( P , Q ) ≤ L 2 W L 2 ( P , Q ) 2 W D φ ( P , Q ) where P and Q are two probability distributions supported on a compact set X ⊂ R d . 21 / 50

  58. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Table of Contents Bregman Divergence Function 1 Generative Adversarial Networks (GANs) 2 Wasserstein Divergence and GANs 3 Relaxed Wasserstein 4 Moment Estimate, Concentration Inequality, and Duality Continuity, Differentiability Gradient Descent Scheme Empirical Results 5 Experiment Setup MNIST and Fashion-MNIST datasets CIFAR-10 and ImageNet datasets Conclusions 6 22 / 50

  59. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Moment Estimate for RW Theorem (G, Hong, Lin, and Yang 2018) Assume that � � x � q 2 P r ( dx ) < + ∞ M q ( P r ) = X for some q > 2 , then there exists a constant C ( q , d ) > 0 such that, for n ≥ 1 , � � E W D φ ( P n , P r )  n − 1 2 + n − q − 2 q , 1 ≤ d ≤ 3 , q � = 4 , 2  q  C ( q , d ) LM q ( P r )  n − 1 2 log(1 + n ) + n − q − 2 ≤ · q , d = 4 , q � = 4 , 2  d + n − q − 2 n − 2  q , d ≥ 5 , q � = d / ( d − 2) .  23 / 50

  60. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Concentration Inequality for RW Theorem (G., Hong, Lin, and Yang 2018) Assume that � exp ( γ � x � α E α,γ ( P r ) = 2 ) P r ( dx ) . X and one of the three following conditions holds, ∃ α > 2 , ∃ γ > 0 , E α,γ ( P r ) < ∞ , or ∃ α ∈ (0 , 2) , ∃ γ > 0 , E α,γ ( P r ) < ∞ , ∃ q > 4 , M q ( P r ) < ∞ . or Then for n ≥ 1 and ǫ > 0 , there exist the scalar a ( n , ǫ ) and b ( n , ǫ ) such that � W D φ ( P n , P r ) ≥ ǫ � ≤ a ( n , ǫ )1 { ǫ ≤ L Prob 2 } + b ( n , ǫ ) . 24 / 50

  61. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Duality Representation for RW Theorem (G., Hong, Lin, and Yang 2018) Assume that two probability distributions P and Q satisfy � � x � 2 2 ( P + Q ) ( dx ) < + ∞ . X Then there exists a Lipschitz continuous function f : X → R such that the RW divergence has a duality representation as � � W D φ ( P , Q ) = φ ( x ) ( P − Q ) ( dx ) + �∇ φ ( x ) , x � Q ( dx ) X X �� � � f ∗ ( ∇ φ ( x )) Q ( dx ) − f ( x ) P ( dx ) + , X X where f ∗ is the conjugate of f , i.e., f ∗ ( y ) = sup x ∈ R d � x , y � − f ( x ) . 25 / 50

  62. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Key element for proof of duality The classical duality representation for the standard Wasserstein distance 26 / 50

  63. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Key element for proof of duality The classical duality representation for the standard Wasserstein distance The RW can be decomposed in terms of a distorted squared Wasserstein- L 2 distance of order 2, plus some residual terms that are independent of the choice of the coupling π . 26 / 50

  64. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Table of Contents Bregman Divergence Function 1 Generative Adversarial Networks (GANs) 2 Wasserstein Divergence and GANs 3 Relaxed Wasserstein 4 Moment Estimate, Concentration Inequality, and Duality Continuity, Differentiability Gradient Descent Scheme Empirical Results 5 Experiment Setup MNIST and Fashion-MNIST datasets CIFAR-10 and ImageNet datasets Conclusions 6 27 / 50

  65. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Relaxed Wasserstein for GANs Question: Is W φ tractable for GANs? 28 / 50

  66. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Relaxed Wasserstein for GANs Question: Is W φ tractable for GANs? Requirement 1: W φ ( P r , P θ ) should be continuous and differentiable w.r.t. θ . 28 / 50

  67. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Relaxed Wasserstein for GANs Question: Is W φ tractable for GANs? Requirement 1: W φ ( P r , P θ ) should be continuous and differentiable w.r.t. θ . Requirement 2: W φ ( P r , P θ ) should have the easily computed or approximated gradient evaluation, i.e., � � ∇ θ W D φ ( P r , P θ ) = F ( g θ , φ, Z , . . . ) . where F is an abstract mapping. 28 / 50

  68. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Continuity and Differentiablity Theorem (G., Hong, Lin, and Yang 2018) 29 / 50

  69. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Continuity and Differentiablity Theorem (G., Hong, Lin, and Yang 2018) 1 W D φ ( P r , P θ ) is continuous in θ if g θ is continuous in θ . 29 / 50

  70. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Continuity and Differentiablity Theorem (G., Hong, Lin, and Yang 2018) 1 W D φ ( P r , P θ ) is continuous in θ if g θ is continuous in θ . 2 W D φ ( P r , P θ ) is differentiable almost everywhere if g θ is locally Lipschitz with a constant ¯ � ¯ L ( θ, Z ) 2 � < ∞ , L ( θ, z ) such that E i.e., for each given ( θ 0 , z 0 ) , there exists a neighborhood N such that � g θ ( z ) − g θ 0 ( z 0 ) � 2 ≤ L ( θ 0 , z 0 ) ( � θ − θ 0 � 2 + � z − z 0 � 2 ) . for any ( θ, z ) ∈ N . 29 / 50

  71. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Table of Contents Bregman Divergence Function 1 Generative Adversarial Networks (GANs) 2 Wasserstein Divergence and GANs 3 Relaxed Wasserstein 4 Moment Estimate, Concentration Inequality, and Duality Continuity, Differentiability Gradient Descent Scheme Empirical Results 5 Experiment Setup MNIST and Fashion-MNIST datasets CIFAR-10 and ImageNet datasets Conclusions 6 30 / 50

  72. Bregman Divergence Function Generative Adversarial Networks (GANs) Moment Estimate, Concentration Inequality, and Duality Wasserstein Divergence and GANs Continuity, Differentiability Relaxed Wasserstein Gradient Descent Scheme Empirical Results Conclusions Gradient Descent Scheme Corollary (G., Hong, Lin, and Yang 2018) Assume that g θ is locally Lipschitz with a constant L ( θ, z ) such X � x � 2 � L ( θ, Z ) 2 � � < ∞ , and 2 ( P r + P θ ) ( dx ) < + ∞ . Then that E there exists a Lipschitz continuous solution f : X → R such that the gradient of the RW divergence has an explicit form, i.e., [ ∇ θ g θ ( Z )] ⊤ ∇ 2 φ ( g θ ( Z )) g θ ( Z ) � � � � ∇ θ W D φ ( P r , P θ ) = E Z + E Z [ ∇ θ f ( ∇ φ ( g θ ( Z )))] . 31 / 50

  73. Bregman Divergence Function Generative Adversarial Networks (GANs) Experiment Setup Wasserstein Divergence and GANs MNIST and Fashion-MNIST datasets Relaxed Wasserstein CIFAR-10 and ImageNet datasets Empirical Results Conclusions Table of Contents Bregman Divergence Function 1 Generative Adversarial Networks (GANs) 2 Wasserstein Divergence and GANs 3 Relaxed Wasserstein 4 Moment Estimate, Concentration Inequality, and Duality Continuity, Differentiability Gradient Descent Scheme Empirical Results 5 Experiment Setup MNIST and Fashion-MNIST datasets CIFAR-10 and ImageNet datasets Conclusions 6 32 / 50

  74. Bregman Divergence Function Generative Adversarial Networks (GANs) Experiment Setup Wasserstein Divergence and GANs MNIST and Fashion-MNIST datasets Relaxed Wasserstein CIFAR-10 and ImageNet datasets Empirical Results Conclusions Experiment Setup RW: KL divergence where φ ( x ) = − x ⊤ log( x ). 33 / 50

  75. Bregman Divergence Function Generative Adversarial Networks (GANs) Experiment Setup Wasserstein Divergence and GANs MNIST and Fashion-MNIST datasets Relaxed Wasserstein CIFAR-10 and ImageNet datasets Empirical Results Conclusions Experiment Setup RW: KL divergence where φ ( x ) = − x ⊤ log( x ). Approach: RMSProp [Tieleman and Hinton, 2012]. 33 / 50

  76. Bregman Divergence Function Generative Adversarial Networks (GANs) Experiment Setup Wasserstein Divergence and GANs MNIST and Fashion-MNIST datasets Relaxed Wasserstein CIFAR-10 and ImageNet datasets Empirical Results Conclusions Experiment Setup RW: KL divergence where φ ( x ) = − x ⊤ log( x ). Approach: RMSProp [Tieleman and Hinton, 2012]. Experiment I: Baselines: WGANs, CGANs, InfoGANs, GANs, LSGANs, DRAGANs, BEGANs, EBGANs and ACGANs. Datasets: MNIST: 60000 (train) and 10000 (test). Fashion-MNIST: 60000 (train) and 10000 (test). 33 / 50

  77. Bregman Divergence Function Generative Adversarial Networks (GANs) Experiment Setup Wasserstein Divergence and GANs MNIST and Fashion-MNIST datasets Relaxed Wasserstein CIFAR-10 and ImageNet datasets Empirical Results Conclusions Experiment Setup RW: KL divergence where φ ( x ) = − x ⊤ log( x ). Approach: RMSProp [Tieleman and Hinton, 2012]. Experiment I: Baselines: WGANs, CGANs, InfoGANs, GANs, LSGANs, DRAGANs, BEGANs, EBGANs and ACGANs. Datasets: MNIST: 60000 (train) and 10000 (test). Fashion-MNIST: 60000 (train) and 10000 (test). Experiment II: Baselines: WGANs and WGANs-GP. Datasets: CIFAR-10 (color): 50000 (train) and 10000 (test). ImageNet (color): 14197122. 33 / 50

Recommend

MELODI M achin E L earning, O ptimization, &amp; D ata I nterpretation @ UW Iyer &amp; Bilmes,

MELODI M achin E L earning, O ptimization, &amp; D ata I nterpretation @ UW Iyer &amp; Bilmes,

Ranking and Machine Learning The Lov asz-Bregman divergences Properties of the Lov asz-Bregman Applications Summary The Lov asz-Bregman Divergence and Connections to Rank Aggregation, Clustering, and Web Ranking Rishabh Iyer Jeff

879 views • 84 slides
A Gradual, Semi-Discrete Approach to Generative Network Training via Explicit Wasserstein

A Gradual, Semi-Discrete Approach to Generative Network Training via Explicit Wasserstein

A Gradual, Semi-Discrete Approach to Generative Network Training via Explicit Wasserstein Minimization Yucheng Chen 1 Matus Telgarsky 1 Chao Zhang 1 Bolton Bailey 1 Daniel Hsu 2 Jian Peng 1 1 Department of Computer Science, UIUC, Urbana, IL 2

99 views • 8 slides
Generative Adversarial Networks, Wasserstein Distance, and Adversarial Loss Zhiyu Min Alibaba

Generative Adversarial Networks, Wasserstein Distance, and Adversarial Loss Zhiyu Min Alibaba

Generative Adversarial Networks, Wasserstein Distance, and Adversarial Loss Zhiyu Min Alibaba AliMe X-Lab Outline GAN Definition and formulation Saddle point optimization Vanishing gradient Alternative objective for Generator

516 views • 25 slides
Advanced Section #8: Generative Adversarial Networks (GANs) CS109B Data Science 2 Vincent Casser

Advanced Section #8: Generative Adversarial Networks (GANs) CS109B Data Science 2 Vincent Casser

Advanced Section #8: Generative Adversarial Networks (GANs) CS109B Data Science 2 Vincent Casser Pavlos Protopapas Outline Concept and Math Applications Common Problems Wasserstein GANs, Conditional GANs and CycleGANs

631 views • 44 slides
Generative Adversarial Network and it its Applications to Human La Language Processing

Generative Adversarial Network and it its Applications to Human La Language Processing

Generative Adversarial Network and it its Applications to Human La Language Processing Hung-yi Lee Full version of the tutorial Outline Part I: General Introduction of Generative Adversarial Network (GAN) Part II: Applications

917 views • 78 slides
Applications of GANs Photo-Realistic Single Image Super-Resolution Using a Generative

Applications of GANs Photo-Realistic Single Image Super-Resolution Using a Generative

Applications of GANs Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks Generative Adversarial Text to Image Synthesis

666 views • 39 slides
Applications of GANs Photo-Realistic Single Image Super-Resolution Using a Generative

Applications of GANs Photo-Realistic Single Image Super-Resolution Using a Generative

Applications of GANs Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks Generative Adversarial Text to Image Synthesis

614 views • 27 slides
Generative neural networks Sigmund Rolfsjord Practical INF5860 - searching for teaching

Generative neural networks Sigmund Rolfsjord Practical INF5860 - searching for teaching

Generative neural networks Sigmund Rolfsjord Practical INF5860 - searching for teaching assistants Overview of wasserstein GAN: (spring 2019) https://medium.com/@jonathan_hui/gan-wasser https://www.uio.no/studier/emner/matnat/ifi/INF5

1.39k views • 78 slides
A Bregman near neighbor lower bound via directed isoperimetry Amirali Abdullah Suresh

A Bregman near neighbor lower bound via directed isoperimetry Amirali Abdullah Suresh

A Bregman near neighbor lower bound via directed isoperimetry Amirali Abdullah Suresh Venkatasubramanian University of Utah Bregman Divergences For convex : R d R D ( p , q ) = ( p ) ( q ) ( q ) , p q

536 views • 36 slides
An overview of Boosting Yoav Freund UCSD Plan of talk Generative vs. non-generative

An overview of Boosting Yoav Freund UCSD Plan of talk Generative vs. non-generative

An overview of Boosting Yoav Freund UCSD Plan of talk Generative vs. non-generative modeling Boosting Alternating decision trees Boosting and over-fitting Applications 2 Toy Example Computer receives telephone call

735 views • 61 slides
On the Complexity of Approximating Wasserstein Barycenters Alexey Kroshnin, Darina Dvinskikh,

On the Complexity of Approximating Wasserstein Barycenters Alexey Kroshnin, Darina Dvinskikh,

On the Complexity of Approximating Wasserstein Barycenters Alexey Kroshnin, Darina Dvinskikh, Pavel Dvurechensky , Alexander Gasnikov, Nazarii Tupitsa, Csar A. Uribe International Conference on Machine Learning 2019 Wasserstein barycenter m

856 views • 9 slides
Decentralize and Randomize: Faster Algorithm for Wasserstein Barycenters Pavel Dvurechensky,

Decentralize and Randomize: Faster Algorithm for Wasserstein Barycenters Pavel Dvurechensky,

Decentralize and Randomize: Faster Algorithm for Wasserstein Barycenters Pavel Dvurechensky, Darina Dvinskikh, Alexander Gasnikov, Csar A. Uribe, Angelia Nedi c Conference on Neural Information Processing Systems 2018 Wasserstein

391 views • 9 slides
Wasserstein GAN Martin Arjovsky, Soumith Chintala, Lon Bottou, ICML 2017 Presented by Yaochen

Wasserstein GAN Martin Arjovsky, Soumith Chintala, Lon Bottou, ICML 2017 Presented by Yaochen

Wasserstein GAN Martin Arjovsky, Soumith Chintala, Lon Bottou, ICML 2017 Presented by Yaochen Xie 12-222017 Contents GAN and its applications [1] GAN vs. Variational Auto-Encoder [2] Whats wrong with GAN [3], [4] JS

829 views • 27 slides
A variational finite volume scheme for Wasserstein gradient flows es 1 , T. O. Gallou et 2 , G.

A variational finite volume scheme for Wasserstein gradient flows es 1 , T. O. Gallou et 2 , G.

A variational finite volume scheme for Wasserstein gradient flows es 1 , T. O. Gallou et 2 , G. Todeschi 2 C. Canc` Inria Lille, Rapsodi 1 Inria Paris, MOKAPLAN 2 ICODE, January 8, 2010 Wasserstein gradient flows domain R d convex,

609 views • 30 slides
All in the Exponential Family: Bregman Duality in Thermodynamic Variational Inference Rob

All in the Exponential Family: Bregman Duality in Thermodynamic Variational Inference Rob

All in the Exponential Family: Bregman Duality in Thermodynamic Variational Inference Rob Brekelmans Vaden Masrani Frank Wood Greg Ver Steeg Aram Galstyan 1 &lt;latexit

638 views • 31 slides
Generative vs. discriminative Generative Discriminative Belief network A is more More

Generative vs. discriminative Generative Discriminative Belief network A is more More

Generative vs. discriminative Generative Discriminative Belief network A is more More robust modular Dont need precise model Class-conditional densities are specification, so long as it is likely to be local, characteristic

1.09k views • 15 slides
Stochastic Optimization for Regularized Wasserstein Estimators ICML 2020 Francis Bach Quentin

Stochastic Optimization for Regularized Wasserstein Estimators ICML 2020 Francis Bach Quentin

Stochastic Optimization for Regularized Wasserstein Estimators ICML 2020 Francis Bach Quentin Berthet Marin Ballu Wasserstein Distance: a natural geometry for distributions How does one compute the distance between two data distributions? 1

305 views • 29 slides
Flat Metric Minimization with Applications in Generative Modeling Thomas M ollenhoff Daniel

Flat Metric Minimization with Applications in Generative Modeling Thomas M ollenhoff Daniel

Flat Metric Minimization with Applications in Generative Modeling Thomas M ollenhoff Daniel Cremers Motivation Latent concepts often induce an orientation of the data. 1/12 Simard et al. 1992, 1998; Rifai et al. 2011 Motivation Latent

855 views • 49 slides
generative design systems Generative Brief Design Definitions Workshop Processes

generative design systems Generative Brief Design Definitions Workshop Processes

Generative Brief Design Definitions Workshop Processes Applications generative design systems Generative Brief Design Definitions Workshop Processes Applications generative design systems Generative Brief Design Definitions

943 views • 75 slides
Mixed Generative and Handcoded Development of Adaptable data-centric Business Applications 27th

Mixed Generative and Handcoded Development of Adaptable data-centric Business Applications 27th

Mixed Generative and Handcoded Development of Adaptable data-centric Business Applications 27th October 2015 Pedram Mir Seyed Nazari, Alexander Roth, and Bernhard Rumpe Software Engineering RWTH Aachen http://www.se-rwth.de/ Alexander Roth

620 views • 16 slides
Hot Gas Around Elliptical Galaxies Mike Anderson (MPA) Joel Bregman (Michigan), Xinyu Dai

Hot Gas Around Elliptical Galaxies Mike Anderson (MPA) Joel Bregman (Michigan), Xinyu Dai

Hot Gas Around Elliptical Galaxies Mike Anderson (MPA) Joel Bregman (Michigan), Xinyu Dai (Oklahoma), Massimo Gaspari (MPA), Simon White (MPA) Mike Anderson Quenching and Quiescence 16 July 2014 Outline Very brief summary of properties of

488 views • 22 slides
Generative networks part 2: GANs 23 / 54 Recap on generative networks Generative networks provide

Generative networks part 2: GANs 23 / 54 Recap on generative networks Generative networks provide

Generative networks part 2: GANs 23 / 54 Recap on generative networks Generative networks provide a way to sample from any distribution. 1. Sample z , where denotes an efficiently sampleable distribution (e.g., uniform or Gaussian). 2.

486 views • 47 slides
Conditional Generative Adversarial Networks (and a brief look at image-to-image translation)

Conditional Generative Adversarial Networks (and a brief look at image-to-image translation)

Conditional Generative Adversarial Networks (and a brief look at image-to-image translation) Final Presentation Peter Bromley Generative Models What is generative modeling? Data: Samples from high dimensional probability distribution p real

621 views • 25 slides
Generative Adversarial Nets(GANs) Troy Cary and Chenzhi Zhao A generative adversarial net is

Generative Adversarial Nets(GANs) Troy Cary and Chenzhi Zhao A generative adversarial net is

Generative Adversarial Nets(GANs) Troy Cary and Chenzhi Zhao A generative adversarial net is a type of neural net, used in deep learning/machine learning problems The goal of a GAN is to train two simultaneous models: a generative model

441 views • 18 slides

More recommend