Patch to the Future: Unsupervised Visual Prediction Jacob Walker, - PowerPoint PPT Presentation

Feb 15, 2023 •271 likes •876 views

Patch to the Future: Unsupervised Visual Prediction Jacob Walker, Abhinav Gupta, Martial Hebert The Robotics Institute Carnegie Mellon University Visual Prediction Goal Both the what and the how Goal Both the what and the how Goal Both the

Patch to the Future: Unsupervised Visual Prediction Jacob Walker, Abhinav Gupta, Martial Hebert The Robotics Institute Carnegie Mellon University
Visual Prediction
Goal Both the what and the how
Goal Both the what and the how
Goal Both the what and the how
Goal Both the what and the how
Goal Both the what and the how
Goal Both the what and the how
Background Data-Driven Yuen et al. 2010
Background Data-Driven Yuen et al. 2010
Background Data-Driven Yuen et al. 2010
Background Agent-Centric Kitani et al. 2012, Koppula et al. 2013, etc.
Background Agent-Centric Kitani et al. 2012, Koppula et al. 2013, etc.
Our Approach Data-Driven
Our Approach Data-Driven + Agent-Centric
Our Approach Unsupervised
Limitations Domain-Dependent Train Test
Limitations Goal-Driven
Limitations No Inter-Element Prediction
Overview
Representation Singh et al. 2012
Action Space
Scene Interaction
Scene Interaction
Scene Interaction High Low
Expected Reward P(Transition) Reward(X,Y,C) E(Reward) = P(T) * R
Planning
Planning
Planning
Planning
Planning
Planning
Planning
Planning
Planning
Planning
Planning
Planning
Planning
Planning
Training Transitions Scene Interaction
Training Transitions Scene Interaction
Training Transitions
Training Patch Transitions
Training Transitions Scene Interaction
Training Scene Interaction
Training Scene Interaction
Training Scene Interaction
Training Scene Interaction
Training Scene Interaction
Datasets • 183 Videos • 139 Training • 44 Testing • ~300 Minutes
Qualitative Results
Qualitative Results
Quantitative Results
Quantitative Results Data-Driven Active Entity Error (Top 6) NN + Sift-Flow Ours Mean 22.34 14.38 Median 16.68 10.91
Quantitative Results Human-Chosen Active Entity Error (Top 1) NN+Sift-Flow Kitani et al. Ours Mean 27.55 37.94 21.55 Median 23.77 30.23 14.98
VIRAT Second Dataset
Conclusion • Unsupervised method for prediction • No explicit modeling of semantics • Models appearance changes • Code will be available!
Thank You!

Recommend

Unsupervised Visual Representation Learning by Context Prediction Berkan Demirel Most slides in

Unsupervised Visual Representation Learning by Context Prediction Berkan Demirel Most slides in this representation are adopted from authors' original presentation at ICCV 2015 ImageNet + Deep Learning Beagle - Image Retrieval - Detection

2.18k views • 33 slides

Un Unsupervised Visu Visual Re Representation Le Learn rning by by Co Context Pr

Un Unsupervised Visu Visual Re Representation Le Learn rning by by Co Context Pr Prediction Carl Doersch, Abhinav Gupta, Alexei A. Efros Presenter: Yiming Pang Outline Motivation Approach Experiment Low-level

499 views • 34 slides

PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu,

PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz, Thomas Funkhouser Princeton University National University of Defense Technology Technical University of Munich

417 views • 38 slides

Towards Assumption-free Unsupervised Domain Adaptation for Visual recognition

Towards Assumption-free Unsupervised Domain Adaptation for Visual recognition Outline Background and Motivations Unsupervised Domain Adaptation - Estimating label information in

1.51k views • 47 slides

Transformation Equivariance vs. Invariance: Unsupervised Learning of Visual Representations

Transformation Equivariance vs. Invariance: Unsupervised Learning of Visual Representations Guo-Jun Qi guojunq@gmail.com Laboratory for MA chine P erception and LE arning ( MAPLE ) & Futurewei Technologies (Huawei Research USA) Laboratory

1.24k views • 58 slides

Overview Neural networks and visual processing So far we have discussed unsupervised learning up

Overview Neural networks and visual processing So far we have discussed unsupervised learning up to V1 For most technology applications (except perhaps compression), Mark van Rossum V1 description is not enough. Yet it is not clear how to

188 views • 18 slides

Unsupervised Learning of Visual Representation by Solving Jigsaw Puzzles, ECCV 16 2018/11/27

Unsupervised Learning of Visual Representation by Solving Jigsaw Puzzles, ECCV 16 2018/11/27 20173130 Jaeyoon Kim CS688 Paper Presentation Image Retrieval with Mixed initiative and Multimodal Feedback, BMVC 18 The system based on

336 views • 18 slides

Unsupervised learning of visual representations using videos X. Wang and A. Gupta ICCV 2015

Unsupervised learning of visual representations using videos X. Wang and A. Gupta ICCV 2015 Experiment presentation by Ashish Bora Motivation Supervised methods work very well But labels are expensive Lot of unlabeled data is

681 views • 32 slides

Fragment binding prediction using unsupervised learning of ligand substructure binding sites

Fragment binding prediction using unsupervised learning of ligand substructure binding sites Grace Tang Altman Lab July 20, 2013 Berlin, Germany 1 Structure Based Virtual Screening Virtual Databases End Goal Screening ZINC 1 21 million

491 views • 15 slides

Unsupervised Learning of Visual Structure Using Predictive Generative Networks William Lotter,

Unsupervised Learning of Visual Structure Using Predictive Generative Networks William Lotter, Gabriel Kreiman & David Cox Harvard University, Cambridge, USA Article overview by Ilya Kuzovkin Computational Neuroscience Seminar University

964 views • 45 slides

A Hybrid Deep Learning Approach For Chaotic Time Series Prediction Based On Unsupervised Feature

A Hybrid Deep Learning Approach For Chaotic Time Series Prediction Based On Unsupervised Feature Learning Norbert Ayine Agana Advisor: Abdollah Homaifar Autonomous Control & Information Technology Institute (ACIT), Departmenent of

379 views • 35 slides

Detecting and Characterizing Events Unsupervised Machine Learning for Social Science Allison

Detecting and Characterizing Events Unsupervised Machine Learning for Social Science Allison J.B. Chaney Princeton University supervised learning unsupervised learning methods input output prediction exploration goals 9 3 1 ? 5 6

920 views • 59 slides

An Analysis of Patch Plausibility and Correctness of

An Analysis of Patch Plausibility and Correctness of Generate-And- Validate Patch Genera8on System Zichao Qi , Fan Long, Sara Achour, and Mar8n Rinard MIT

323 views • 29 slides

Combining Visual Analytics and Machine Learning for Route Choice Prediction Application to Pre

Combining Visual Analytics and Machine Learning for Route Choice Prediction Application to Pre Tactical Traffic Forecast Rodrigo Marcos Data Scientist, Nommon Solutions and Technologies SESAR Innovation Days Beograd, 29 th November 2017 Scope

461 views • 22 slides

Summary of part I: prediction and RL Prediction is important for action selection The

Summary of part I: prediction and RL Prediction is important for action selection The problem: prediction of future reward The algorithm: temporal difference learning Neural implementation: dopamine dependent learning in BG A

939 views • 60 slides

UNSUPERVISED LEARNING, CLUSTERING UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised

UNSUPERVISED LEARNING, CLUSTERING UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised learning: X - y pairs, f(x) function approximation Unsupervised learning: only X, no y Exploring the space of X measurements,

440 views • 14 slides

Introduction to PCA Unsupervised Learning in R Unsupervised learning Two methods of

UNSUPERVISED LEARNING IN R Introduction to PCA Unsupervised Learning in R Unsupervised learning Two methods of clustering finding groups of homogeneous items Next up, dimensionality reduction Find structure in features

904 views • 29 slides

Prediction and Odds 18.05 Spring 2017 Probabilistic Prediction Also called probabilistic

Prediction and Odds 18.05 Spring 2017 Probabilistic Prediction Also called probabilistic forecasting. Assign a probability to each outcome of a future experiment. Prediction: It will rain tomorrow. Probabilistic prediction: Tomorrow it

480 views • 26 slides

La Larg rge-Scal Scale Patch Patch Reco comme mmendati ation at at Alibab aba Xindong

La Larg rge-Scal Scale Patch Patch Reco comme mmendati ation at at Alibab aba Xindong Zhang 1 , Chenguang Zhu 2 , Yi Li 3 , Jianmei Guo 1 , Lihua Liu 1 , and Haobo Gu 1 1. Alibaba Group 2. University of Texas at Austin 3. Nanyang

462 views • 7 slides

Prediction and Odds 18.05 Spring 2014 January 1, 2017 1 / 20 Probabilistic Prediction Also

Prediction and Odds 18.05 Spring 2014 January 1, 2017 1 / 20 Probabilistic Prediction Also called probabilistic forecasting. Assign a probability to each outcome of a future experiment. Prediction: It will rain tomorrow. Probabilistic

349 views • 20 slides

Prediction and Odds 18.05 Spring 2014 January 1, 2017 1 / 26 Probabilistic Prediction Also

Prediction and Odds 18.05 Spring 2014 January 1, 2017 1 / 26 Probabilistic Prediction Also called probabilistic forecasting. Assign a probability to each outcome of a future experiment. Prediction: It will rain tomorrow. Probabilistic

692 views • 26 slides

A Comprehensive Survey on Deep Future Frame Video Prediction by Javier Selva Castell

Final Master Thesis Master on Artificial Intelligence A Comprehensive Survey on Deep Future Frame Video Prediction by Javier Selva Castell Supervised by Sergio Escalera Guerrero and Marc Oliu Simn Future Frame Prediction Given a video

761 views • 41 slides

Information Structure Prediction for Visual-World Referring Expressions Micha Elsner Hannah

Information Structure Prediction for Visual-World Referring Expressions Micha Elsner Hannah Rohde, Alasdair Clarke The Ohio State University University of Edinburgh University of Aberdeen Describe the person in the box so that someone

783 views • 28 slides

Non-Inferiority Assessment of Patch Adhesion Non-Inferiority Assessment of Patch Adhesion and and

5/21/2015 Non-Inferiority Assessment of Patch Adhesion Non-Inferiority Assessment of Patch Adhesion and and Dermatology Irritation of Post-Market Dermatology Irritation Evaluation Change and Generic Drug Evaluation Presented by Mark Liu,

529 views • 22 slides