Learning from Unlabeled Video Carl Vondrick Columbia University

Survivor Bias of Video Data Large-scale Video Classification with Convolutional Neural Networks, CVPR 2014

Felix Warneken, Max Plank Institute

The Oops ! dataset

Oops! Predicting Unintentional Action CVPR 2020 oops.cs.columbia.edu Epstein, Chen, Vondrick. CVPR 2020.

Learning from unlabeled video

Example Videos

Perceptual Clues 1) Predictability Ranzato 2014, Han 2019, … 2) Temporal Order Misra 2016, Wei 2018, …

3) Video speed as self-supervised clue Epstein, Chen, Vondrick. CVPR 2020.

Speed of Action Alters Perceptual Judgement

3) Video speed as self-supervised clue Epstein, Chen, Vondrick. CVPR 2020.

Visualizing Features Epstein, Chen, Vondrick. CVPR 2020.

Fit linear model to classify intentionality + - ++ - -

What’s missing? Environmental Unexpected Multi-agent Limited Skill Planning Error Single-agent Execution Error Limited Visibility Human Ours (self-supervised) Limited Knowledge Kinetics (supervised) 0 5 10 15 20 25 Error (lower is better)

oops.cs.columbia.edu Tuesday 10am PST Poster 93 Epstein, Chen, Vondrick. CVPR 2020.

Natural Synchronization Vision Speech

Ackee seems to be: • edible • white/yellow • washable • sticky • larger than cherry tomato “I’m going to go in with the actual ackee I rinsed off earlier”

Word Learning from Vision VisualBERT, VILBERT, VideoBERT, LXMERT, … “stir” Transformer stack … “I turn on the fire and then I [???] the pasta” Learn what Learn how to learn � “stir” means what “stir” means

Learning to Learn Words Suris, Epstein, Ji, Chang, Vondrick. arXiv.

Transformers as Meta-Learners Suris, Epstein, Ji, Chang, Vondrick. arXiv.

Transformers as Meta-Learners Implement with cross entropy loss Suris, Epstein, Ji, Chang, Vondrick. arXiv.

Meta-Learning Episodes New Words Episode … Composition Episode Suris, Epstein, Ji, Chang, Vondrick. arXiv.

Mode 1: Language Modeling Mode 2: Word Acquisition Suris, Epstein, Ji, Chang, Vondrick. arXiv.

Language Modeling 75 18% drop 60 Seen 19% 45 Accuracy drop Seen New 30 New 15 Seen Composition New Composition 0 BERT pretrained BERT + vision Meta-Learned Suris, Epstein, Ji, Chang, Vondrick. arXiv.

Language Modeling 75 11% 18% drop drop 60 Seen Seen 19% New 45 Accuracy drop Seen New 30 New 15 Seen Composition New Composition 0 BERT pretrained BERT + vision Meta-Learned Suris, Epstein, Ji, Chang, Vondrick. arXiv.

Word Acquisition Training Set Test Example get avocado still taking skin off stir rice into pan avocado fish with a knife new word Suris, Epstein, Ji, Chang, Vondrick. arXiv.

Word Acquisition Training Set Test Example open the wash plates switch off oven on oven close oven cupboard with rag the bottom right new word Suris, Epstein, Ji, Chang, Vondrick. arXiv.

Novel word acquisition Suris, Epstein, Ji, Chang, Vondrick. arXiv.

Visualizing Learned Process Suris, Epstein, Ji, Chang, Vondrick. arXiv.

Visualizing Attention Green boxes impact green prediction the most Training Set cut cherry tomatoes put spoon close food container … … … Test chop sun-dried rinse container spoon container put spoon tomatoes tomatoes Suris, Epstein, Ji, Chang, Vondrick. arXiv.

expert.cs.columbia.edu Suris, Epstein, Ji, Chang, Vondrick. arXiv.

Learning from Unlabeled Video Carl Vondrick Columbia University

Learning from Unlabeled Video Carl Vondrick Columbia University - PowerPoint PPT Presentation

Learning from Unlabeled Video Carl Vondrick Columbia University Survivor Bias of Video Data Large-scale Video Classification with Convolutional Neural Networks, CVPR 2014 Survivor Bias of Video Data Large-scale Video Classification with

Learning How to Move and Where to Look from Unlabeled Video Kristen Grauman Department of

Visual Learning with Unlabeled Video and Look-Around Policies Kristen Grauman Department of

from unlabeled video Kristen Grauman Department of Computer Science The University of T exas at

10701 Semi supervised learning Can Unlabeled Data improve supervised learning? Important

Ac#ve Learning Aarti Singh Machine Learning 10-601 Dec 6, 2011 Slides Courtesy: Burr

Learning from Unlabeled Data INFO-4604, Applied Machine Learning University of Colorado Boulder

Learning from Limited Labeled Data (but a lot of unlabeled data) NELL as a case study Tom M.

Clustering Aarti Singh Slides courtesy: Eric Xing Machine Learning 10-701/15-781 Oct 25, 2010

Principal Component Analysis 4/7/17 PCA: the setting Unsupervised learning Unlabeled data

UnTran: Recognizing Unseen Activities with Unlabeled data using Transfer Learning ACM/IEEE

Active Learning with Active Learning with Model Selection Neil Rubens Sugiyama Lab / Tokyo

Principal Component Ananalysis 4-8-2016 PCA: the setting Unsupervised learning Unlabeled

Learning to Extract Entities from Labeled and Unlabeled Text Rosie Jones Language Technologies

GAN-based Photo Video Synthesis Summary of Generating Videos with Scene Dynamics Lei Zhang CS

K-Means Clustering 3/3/17 Unsupervised Learning We have a collection of unlabeled data

Mimicking Word Embeddings using Subword RNNs Yuval Pinter, Robert Guthrie, Jacob Eisenstein

10 Steps to Counting Unlabeled Planar Graphs: 20 Years Later Manuel Bodirsky October 2007

Co-Training Based on Combining Labeled and Unlabeled Data with Co-Training by A. Blum

1 Video of Demo Q-Learning Auto Cliff Grid Exploration vs. Exploitation How to Explore? Video

Clustering Clustering is an unsupervised classification method, i.e. unlabeled data is partitioned

Learning Perceptual Causality from Video Amy Fire and Song-Chun Zhu Center for Vision,

A Framework for Learnig Predictive Structures from Multiple Tasks and Unlabeled Data Rie Kubota

Video is Popular in Korea The Rise of Shared Online Video, the Fall of Traditional Learning Dr.

Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild Presenters: Nikhil Kannan, Ying