Emer Emergenc ence o e of O f Obj bjec ect S Seg egmen mentatio ion n in P in Per ertur urbed Gener bed Generativ ive M e Models dels Adam Bielski, Paolo Favaro University of Bern
Ob Object se segm gmentation Segmentation model 2
Ob Object se segm gmentation Annotated data Segmentation model 3
Ob Object se segm gmentation Annotated data Segmentation model • Human annotated • Expensive • Expertise knowledge 4
Un Unsuper ervis vised ed objec ject t seg egmen entatio tion Annotated data Segmentation model • Human annotated • Expensive • Expertise knowledge Can we do it without labels? 5
Un Unsuper ervis vised ed ob object se segm gmentation E G Mask Image 6
Le Learn rning to to ge generate la layer ered ed sc scenes wi with GA GAN Background Foreground E G Mask Image 𝑨~𝒪(0, 𝐽) 7
Le Learn rning to to ge generate la layer ered ed sc scenes wi with GA GAN Background Image composition Foreground + ⊙ ⊙ = E G Background 1 - Mask Foreground Mask Mask Composite image Image 𝑨~𝒪(0, 𝐽) 8
Le Learn rning to to ge generate la layer ered ed sc scenes wi with GA GAN Background Image composition Foreground + ⊙ ⊙ = E G Background 1 - Mask Foreground Mask Mask Composite image Image 𝑨~𝒪(0, 𝐽) Real / D Fake Unlabeled real images 9
De Degenerate s solu lutio ions Background Foreground Mask Composite image 10
De Degenerate s solu lutio ions Background Foreground Mask Composite image 11
De Degenerate s solu lutio ions Background Foreground Mask Composite image 12
De Degenerate s solu lutio ions Background Foreground Mask Composite image 13
Wha What is is a c a correct ect par partit itio ion? n? 14
Wha What is is a c a correct ect par partit itio ion? n? 15
Ra Rando ndom m shi hifts 16
Ra Rando ndom m shi hifts 17
Ra Rando ndom m shi hifts 18
Ra Rando ndom m shi hifts 19
Invalid partition ⟹ invalid scene after a small shift Valid partition ⟹ valid scene after a small shift Ra Rando ndom m shi hifts 20
Av Avoiding degenerate solutions Background Foreground Mask Composite image Composite image w/shift 21
Av Avoiding degenerate solutions Background Foreground Mask Composite image Composite image w/shift 22
Av Avoiding degenerate solutions Background Foreground Mask Composite image Composite image w/shift 23
Av Avoiding degenerate solutions Background Foreground Mask Composite image Composite image w/shift 24
Av Avoiding degenerate solutions Background Foreground Mask Composite image Composite image w/shift Add a loss term to avoid empty masks 25
Av Avoiding degenerate solutions Background Foreground Mask Composite image Composite image w/shift Add a loss term to avoid empty masks 26
Learn Le rning wi with per pertur urba bations ns Background Image composition Foreground + ⊙ ⊙ = E G Background 1 - Mask Foreground Mask Mask Composite image Image 𝑨~𝒪(0, 𝐽) Minimum mask loss 27
Learn Le rning wi with per pertur urba bations ns Background Image composition Foreground + ⊙ ⊙ = E G Background 1 – Mask Foreground Mask Mask Composite image Image shifted shifted shifted (fake) 𝑨~𝒪(0, 𝐽) Random shift 𝑞~𝒱 Real Minimum mask loss / D Fake Real image 28
Ge Generatio tion results lts LSUN Car LSUN Bird LSUN Chair LSUN Horse Backg�o�nd Backg�o�nd Fo�eg�o�nd Fo�eg�o�nd Ma�k Ma�k Fo�eg�o�nd Fo�eg�o�nd �/�ma�k �/�ma�k Com�o�i�e Com�o�i�e image image 29
Le Learn rning to to se segm gment Background Image composition Foreground + ⊙ ⊙ = G Background 1 - Mask Mask Mask Foreground Composite image 30
Learn Le rning to to se segm gment Frozen Background Image composition Foreground + ⊙ ⊙ = G Background 1 - Mask Mask Mask Foreground Composite image 31
Le Learn rning to to se segm gment Frozen Background Image composition Foreground + ⊙ ⊙ = E G Background 1 - Mask Mask Mask Foreground Composite image Image 𝑀1 𝑀𝑝𝑡𝑡 32
Le Learn rning to to se segm gment Frozen Background Image composition Foreground + ⊙ ⊙ = E G Background 1 - Mask Mask Mask Foreground Composite image Image 𝑀1 𝑀𝑝𝑡𝑡 33
Se Segme mentation on r results Real image Real image O�� Our results me�hod Approx. Ma�k R-CNN ground truth Real image Real image Our Our results method Ground Ground truth truth 34
Po Poster #60 E ast Exhibition Hall B + C 35
Recommend
More recommend
