MemeSequencer : Sparse Matching for Embedding Image Macros Abhimanyu - PowerPoint PPT Presentation

MemeSequencer : Sparse Matching for Embedding Image Macros Abhimanyu (Abhi) Dubey, Esteban Moro, Manuel Cebrian and Iyad Rahwan Massachusetts Institute of Technology {dubeya*,emoro,cebrian,irahwan}@mit.edu

Image Virality - Thousands of images are shared online every day - Predicting virality can benefit: - content creation (designing content that becomes viral) - content-based network traffic routing (routing content based on predicted virality) - content-based network caching (caching the content that is likely to go viral) - There has been incredible prior work in the area of predicting viral content based on network structure of information dissemination (cascades, community detection etc.) - However, there has been limited work on the impact of the content itself on its virality - In this study we focus solely on the content and its impacts on popularity MemeSequencer | Abhimanyu Dubey (WWW 2018)

Memes as Images Image Macro MemeSequencer | Abhimanyu Dubey (WWW 2018)

Memes evolve during propagation (a) (b) (d) (c) MemeSequencer | Abhimanyu Dubey (WWW 2018)

Types of mutations in memes Content-Preserving Content Altering (a) (c) (b) MemeSequencer | Abhimanyu Dubey (WWW 2018)

Content-preserving mutations (overlays) MemeSequencer | Abhimanyu Dubey (WWW 2018)

Designing an embedding for memes d ( , ) < d ( , ) d ( , ) R n d ( , ) MemeSequencer | Abhimanyu Dubey (WWW 2018)

Key methodological insight What if we could recover the original image (template) from the modified image? If we can decouple the mutations from the original image, we can create a space that preserves the required distances. MemeSequencer | Abhimanyu Dubey (WWW 2018)

Decoupled semantic embeddings template information mutation information MemeSequencer | Abhimanyu Dubey (WWW 2018)

Sparse Matching Consider the set of k template images: We create an augmented set by creating affine transformations of the templates: ... MemeSequencer | Abhimanyu Dubey (WWW 2018)

Sparse Matching Each target image y can then be represented (with error) as a linear combination of the template images s i,j (with α i,j as the weights): In matrix form, we can replace the above by the equation: MemeSequencer | Abhimanyu Dubey (WWW 2018)

Sparse Matching We then want to recover x 0 such that: In the current form, however, this problem is NP-hard (even to approximate), so we solve the following problem: MemeSequencer | Abhimanyu Dubey (WWW 2018)

Constructing decoupled feature vectors The ultimate goal of the method is to create a semantic embedding that enables us to understand virality. The steps of the overall algorithm that provides us with a feature embedding are: - Sparse Matching to decouple the template and mutations - Image feature extraction on template and the mutation separately - CNN-based feature extractors such as AlexNet, ResNet etc - Text feature extraction on template and mutation separately - Run off-the-shelf OCR (OpenCV) to identify text - Extract generic text features (Average Word2Vec and Skip-Thought) to obtain semantic information - Use classifier (SVM) on features to make predictions based on task MemeSequencer | Abhimanyu Dubey (WWW 2018)

Constructing the template set - Typically we have the template set predefined (or manually constructed) - How do we proceed in the case when we do not have a template set? - We proceed without the template set, and construct it on the go - We proceed with an empty template set - For each image we encounter, we run the sparse matching algorithm - If the image doesn’t match with an existing template, we add it to our template set - If the image matches with an existing template, we update the template by median blending: - For all images that match the template, we set the final template image as the median (pixel-wise) MemeSequencer | Abhimanyu Dubey (WWW 2018)

Median Blending MemeSequencer | Abhimanyu Dubey (WWW 2018)

Overall pipeline matched template template image features decoupled overlay Image Feature Extractor (CNN) overlay image features target macro set ( T ) 𝜷 oh you just x overlay text graduated? features you must Sparse Matching OCR know everything. Feature Representation Text Feature Extractor (SkipThought or Word2Vec) template set ( S ) MemeSequencer | Abhimanyu Dubey (WWW 2018)

Experimental Setup We examine our algorithm on 3 different datasets: - Viral Images Dataset (Parikh2015, Lakkaraju2013): 6k training, 500 test image pairs - Memegenerator Dataset (Coscia2013): 326,181 images (70-10-20 train-val-test split) - Quickmeme Dataset (Coscia2013): 178,801 images (70-10-20 train-val-test split) For baselines, we use the following methods: - Spatial Transformer Networks (Dubey2017) - Low level vision features (Parikh2015) - CNN based feature extractors (AlexNet, VGGNet, ResNet) - Text based feature extractors (Word2Vec, SkipThought) MemeSequencer | Abhimanyu Dubey (WWW 2018)

How separable is our representation? We extracted sparse matching features for all 500k images - - Clustered these images based on K-means with varying number of clusters - Calculated Silhouette Score (SS) - Measures how much intra-cluster dissimilar exists: lower is better for tight clusters - Calculated Davies-Bouldin Index (DBI) - Measures how similar different clusters are : lower is better for good separation MemeSequencer | Abhimanyu Dubey (WWW 2018)

How separable is our representation? MemeSequencer | Abhimanyu Dubey (WWW 2018)

Benefits in nearest neighbor retrieval MemeSequencer | Abhimanyu Dubey (WWW 2018)

Benefits in pairwise virality prediction We extracted sparse matching features for all images (per dataset) - - For extracted features, we train a RankSVM to solve the following task: - Given a pair of images, which image is likely to go viral? MemeSequencer | Abhimanyu Dubey (WWW 2018)

Benefits in pairwise virality prediction MemeSequencer | Abhimanyu Dubey (WWW 2018)

Uncovering mutation patterns via phylogenetic trees MemeSequencer | Abhimanyu Dubey (WWW 2018)

Take-home summary We propose the study of image virality based on content alone - - Most memes and viral images are types of image macros - predefined image templates with added images or text (overlays) - We develop an algorithm that exploits this structure of macros to create a robust semantic embedding, which provides benefits on - Semantic clustering - Image retrieval - Virality, topic and popularity prediction - Uncovering mutation patterns in memes MemeSequencer | Abhimanyu Dubey (WWW 2018)