Arxiv, 8 dec 2018 Main question How does a GAN represent our visual - PowerPoint PPT Presentation

Arxiv, 8 dec 2018

Main question How does a GAN represent our visual world internally? How do activations in the generator correlate with concepts in the output image?

Dissection

Results In the dining room dataset, a unit emerges to match dining table regions. More interestingly, the matched tables have different colors, materials, geometry, viewpoints, and levels of clutter: the only obvious commonality among these regions is the concept of a table. This unit’s featuremap correlates to the fully supervised segmentation model (Xiao et al., 2018) with a high IoU of 0.34.

Results • The set of all object classes matched by the units of a GAN provides a map of what a GAN has learned about the data. • Conference room GAN learns chairs, table, etc., • Bedroom GAN learns bed, window, etc. • Many units represent parts of objects: for example, the conference room GAN contains separate units for the body and head of a person. • Depth influences unit semantics

Debugging GANs • By examining unit semantics, we confirm that [contribution of ProGAN] increases not only the realism of results, but also the diversity of concepts represented by units: the number of types of objects, parts, and materials matching units increases by more than 40%.

Causal intervention

Fixing GANS Strikingly, this simple manual change to a network beats state- of-the-art GANs models.

GAN image editing: removal

GAN image editing: insertion

• https://www.youtube.com/watch?v=yVCgUYe4JTM