Weakly Supervised Disentanglement with Guarantees Rui Shu Joint work with Yining Chen, Abhishek Kumar, Stefano Ermon, Ben Poole
Why Decompose data into a set of underlying Explainable models human-interpretable factors of variation What is in the scene? Blue sky Pink wall Controllable generation Generate a red ball instead Green floor Small purple ball 2
How: Fully-Supervised Strategy: Label everything Controllable generation as label-conditional generative modeling {dark blue wall, green floor, green oval} green wall, red floor, blue cylinder {green wall, red floor, green cylinder} {red wall, green floor, pink ball} 3
How: Fully-Supervised Problem: Some things are hard to label Generate this guy with this hair What kind of hairstyle? What kind of glasses? 4
How: Unsupervised? Strategy: Exploit statistical independence assumption + neural net magic Swivel the chair Beta-VAE TC-VAE FactorVAE 5
How: Unsupervised? Problem: Is statistical independence assumption + neural net magic enough? Z 2 : Shading Z 1 : Shape vs Locatello, et al. Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations , ICML 2019. 6
How: Weakly Supervised Strategy: Leverage “weak” supervision when possible 7
How: Weakly Supervised Restricted Labeling: Label what we can Pink wall Size: ¯\_( ツ )_/¯ Green floor Purple ball 8
How: Weakly Supervised Match Pairing: Find pairs with known similarities Same ground color Real world data : direct intervention to 9 share / change certain factors
How: Weakly Supervised Rank Pairing: Compare pairs Which is bigger? 10
The Plan 1. Definitions : Decompose disentanglement into: a. Consistency b. Restrictiveness 2. Guarantees: Prove whether weak supervision guarantees consistency, restrictiveness, or both 11 Departure from existing literature: no end-to-end theoretical framework of disentanglement
Definitions Disentangle: What does it mean when I say Z1 disentangles size? 1. When z 1 is fixed, is size fixed? 2. When we only change z 1 , does only size change? 12
Definitions Disentangle: What does it mean when I say Z1 disentangles size? 1. When z 1 is fixed, is size fixed? ( Consistency ) 2. When we only change z 1 , does only size change? ( Restrictiveness ) 13
Definitions: Consistency Oracle encoder Generative model When Z I is fixed, S I is fixed Perturbation-based generation 14
Definitions: Restrictiveness Oracle encoder Generative model When only Z I is changed, only S I is changed Perturbation-based generation 15 Equivalently: when Z \I is fixed, S \I is fixed
Definitions: Disentanglement Z I is consistent and restricted to S I 16
Consistency versus Restrictiveness When only Z I is changed, only S I is changed Equivalently: when Z \I is fixed, S \I is fixed 17
Consistency versus Restrictiveness 18
Union Rules Consistency Union: If fixing Z I fixes S I and fixing Z J fixes S J then fixing ( Z I , Z J ) fixes ( S I , S J ) Restrictiveness Union: If changing Z I changes only S I and changing Z J changes only S J then changing ( Z I , Z J ) changes only ( S I , S J ) 19
Intersection Rules Consistency Intersection: If fixing Z I fixes S I and fixing Z J fixes S J then fixing Z V fixes S V Restrictiveness Intersection: If changing Z I changes only S I and changing Z J changes only S J then changing Z V changes only S V 20
Disentanglement Rule Disentanglement via Consistency Consistency on all factors implies disentanglement on all factors Disentanglement via Restrictiveness Restrictiveness on all factors implies disentanglement on all factors 21
Summary of Rules 22
Summary of Rules 23
Strategy for Disentanglement Dataset 1 → C(1) Dataset 2 → C(2) … Dataset n → C(n) Using datasets together (+ right algorithm) guarantees full disentanglement 24
Restricted Labeling Guarantees Consistency s \I s I z I z \I Distribution Match x x Z I will be consistent with S I 25
Match Pairing Guarantees Consistency s ’ \I z ’ \I s \I s I z \I z I Distribution Match x ’ x ’ x x Z I will be consistent with S I 26
Rank Pairing Guarantees Consistency s ’ \i s ’ i z ’ \i z ’ i s \i s i z \i z i Distribution Match x y x y x ’ x ’ Z I will be consistent with S I 27
Summary of Guarantees 28
Targeted Consistency / Restrictiveness Generative model trained via restricted labeling at S 5 Evaluated model on consistency of Z 0 vs S 0 29
Targeted Consistency / Restrictiveness Consistency: Consistency: Restrictiveness: Consistency: Restrictiveness: Restricted Labeling Match Pairing Match Pairing Rank pairing Intersection (Share 1 factor) (Change 1 factor) 30
Consistency versus Restrictiveness ● Models trained to guarantee only consistency or restrictiveness of one factor ● Strong correlation of consistency vs restrictiveness 31
Digression: Style-Content Disentanglement Unobserved Observed Content style class label z y Style x Only content-consistency is guaranteed Style-content disentanglement not guaranteed (but due to neural net magic) 32
Full Disentanglement 33
Full Disentanglement: Visualizations ● Visualize multiple rows of single-factor ablation Elevation ● Check for consistency and restrictiveness Azimuth 34
Full Disentanglement: Visualizations ● Visualize multiple rows of single-factor ablation ● Check for consistency and restrictiveness Ground truth factors: floor color, wall color, object color, object size, object type, and azimuth. 35
Full Disentanglement: Visualizations ● Visualize multiple rows of single-factor ablation Ground truth factor: object size ● Check for consistency and restrictiveness Ground truth factor: wall color 36
Conclusions ● Definitions for disentanglement ● A calculus of disentanglement Analyzed weak supervision methods ● ● Demonstrated guarantees empirically 37
Conclusions ● Definitions for disentanglement ● Better definitions? ● A calculus of disentanglement ● Do new definitions preserve calculus? Analyzed weak supervision methods Analyze other weak supervision methods? ● ● ● Demonstrated guarantees empirically ● Cost of weak supervision in real world? 38
Assumption: X → S is deterministic Blue sky Pink wall Green floor Small purple ball 39
Questions? Entangled Disentangled ruishu@stanford.edu @_smileyball 40 @smiley._.ball
Recommend
More recommend
