Learning Spatiotemporal Features with 3D Convolutional Networks Du - PowerPoint PPT Presentation

Nov 14, 2023 •136 likes •355 views

Learning Spatiotemporal Features with 3D Convolutional Networks Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri ada BAK 29.03.16 Effective Video Descriptor Generic Can represent different types Compact

Learning Spatiotemporal Features with 3D Convolutional Networks Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri Çağdaş BAK 29.03.16
Effective Video Descriptor • Generic – Can represent different types • Compact – Processing, storage • Efficient – computation • Simple – implementation
3D Convolution and Pooling • 3D Convolution is better than 2D Convolution to model temporal information. – 2D CONV : performed only spatially, lose temporal information. – 3D CONV : performed spatio-temporally, preserve temporal information. • Same phenomena is applicable for pooling.
2D Convolution On 1-ch Input • Result : 2D Image.
2D Convolution On n-ch Input • Result : 2D Image.
3D Convolution On n-ch Input • Result : Volume
Identify Best Architecture For 3D ConvNets (On UCF101) • Common network settings – All video frames resized into 128x171. – Videos are split into non-overlapped 16 frame clip. – Input : 3x16x128x171. – 5 Convolution and Pooling layer – 2 Fully Connected layer – Softmax Loss layer to predict action labels
Identify Best Architecture For 3D ConvNets (On UCF101) • Varying Network Architecture – Homogeneous temporal depth. • Depth –d for 1,3,5,7 – Varying temporal depth. • Increasing : 3-3-5-5-7 • Decreasing : 7-7-5-5-3-3
3D Convolution Kernel Temporal Depth Search
Spatiotemporal Feature Learning • Best Network Architecture – With 3x3x3 kernel
Spatiotemporal Feature Learning • Dataset for training – Sports 1M Dataset • Largest video classification benchmark • 1.1 million sports videos • 487 categories
Sports 1M Classification Results
C3D Video Descriptor • C3D Model can be used as a feature extractor for various video analysis tasks. – Action recognition – Action similarity – Scene and Object recognition • Using with fc6 activations – 4096 dimension
Action Recognition • Dataset : UCF101 – 13.320 video – 101 human action
Action Similarity Labeling • Dataset : ASLAN – 3,631 video – 432 action class
Scene Object Recognition • Dataset : YUPENN – 420 video – 14 scene • Dataset : Maryland – 130 video – 13 scene
Why C3D Features? • Generic • Compact • Efficient • Simple
What Does C3D Learn ?
Useful Links • http://vlg.cs.dartmouth.edu/c3d/ • https://github.com/facebook/C3D
Learning Spatiotemporal Features with 3D Convolutional Networks Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri Çağdaş BAK 29.03.16

Recommend

Convolutional Neural Networks for Language CS 6956: Deep Learning for NLP Features from text

Convolutional Neural Networks for Language CS 6956: Deep Learning for NLP Features from text Example: Sentiment classification The goal: Is the sentiment of a sentence positive, negative or neutral? The film is fun and is host to some truly

1.37k views • 88 slides

Probabilistic Palm Rejection Using Spatiotemporal Touch Features and Iterative Classification

Probabilistic Palm Rejection Using Spatiotemporal Touch Features and Iterative Classification Julia Schwarz, Robert Xiao, Jennifer Mankoff, Scott E. Hudson, Chris Harrison ? ? ? ? pen palm palm palm Prior Software-Only Approaches

590 views • 36 slides

Using Spatiotemporal Features for Butterfly Classification MARTA SKRETA, SASHA LUCCIONI, DAVID

Using Spatiotemporal Features for Butterfly Classification MARTA SKRETA, SASHA LUCCIONI, DAVID ROLNICK Climate Change and Butterflies BUTTERFLIES ECOSYSTEM Temperature/weather impact Predators of butterflies/caterpillars Indirect via habitat

585 views • 11 slides

AttentionNAS: Spatiotemporal Attention Cell Search for Video Classification Xiaofang Wang, Xuehan

AttentionNAS: Spatiotemporal Attention Cell Search for Video Classification Xiaofang Wang, Xuehan Xiong, Maxim Neumann, AJ Piergiovanni, Michael S. Ryoo, Anelia Angelova, Kris M. Kitani, Wei Hua Convolutional networks are dominant C3D [ICCV

746 views • 26 slides

Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural

Transfer Learning with Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural Networks A breakthough Convolutional Neural Networks VGG-16 example Layers of Convolutional filters Bottleneck

625 views • 23 slides

Hierarchical Convolutional Features for Visual Tracking Chao Ma Jia-Bin Huang Xiaokang Yang

Hierarchical Convolutional Features for Visual Tracking Chao Ma Jia-Bin Huang Xiaokang Yang Ming-Husan Yang SJTU UIUC SJTU UC Merced ICCV 2015 Background Given the initial state

509 views • 31 slides

Convolutional Networks Lecture slides for Chapter 9 of Deep Learning Ian Goodfellow 2016-09-12

Convolutional Networks Lecture slides for Chapter 9 of Deep Learning Ian Goodfellow 2016-09-12 Convolutional Networks Scale up neural networks to process very large images / video sequences Sparse connections Parameter sharing

552 views • 27 slides

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided Backpropagation, Deep Dream, Deep Art, Fooling Convolutional Neural Networks Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute

1.07k views • 72 slides

Supervised Learning Given: a set of inputs features X 1 , . . . , X n a set of target features Y 1

Supervised Learning Given: a set of inputs features X 1 , . . . , X n a set of target features Y 1 , . . . , Y k a set of training examples where the values for the input features and the target features are given for each example a new

323 views • 18 slides

From Feedforward-Designed Convolutional Neural Networks (FF-CNNs) to Successive Subspace Learning

From Feedforward-Designed Convolutional Neural Networks (FF-CNNs) to Successive Subspace Learning (SSL) January 30, 2020 C.-C. Jay Kuo University of Southern California 1 Introduction Deep Learning provides an effective solution when

961 views • 60 slides

Neural Networks + Convolutional Neural Networks Last Class Global Features The perceptron

CS4501: Introduction to Computer Vision Neural Networks + Convolutional Neural Networks Last Class Global Features The perceptron model Neural Networks multilayer perceptron model (MLP) Backpropagation Todays Class

789 views • 34 slides

Lecture 6: Convolutional NN Princeton University COS 495 Instructor: Yingyu Liang Review:

Deep Learning Basics Lecture 6: Convolutional NN Princeton University COS 495 Instructor: Yingyu Liang Review: convolutional layers Convolution: two dimensional case Input Kernel/filter a b c d w x e f g h y z i j k l wa + bx

847 views • 36 slides

Convolutional Autoencoder (CAE) Prof. Seungchul Lee Industrial AI Lab. Convolutional Autoencoder

Convolutional Autoencoder (CAE) Prof. Seungchul Lee Industrial AI Lab. Convolutional Autoencoder Motivation: image to autoencoder ? Convolutional autoencoder extends the basic structure of the simple autoencoder by changing the fully

349 views • 15 slides

Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning Qimai Li, Zhichao

Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning Qimai Li, Zhichao Han, Xiao-Ming Wu Department of Computing The Hong Kong Polytechnic University Supervised Learning Tons of labeled data A good model 2 Image

880 views • 13 slides

Clustering ! Unsupervised learning: learning without a teacher ! Only features and no labels

So far in the course Supervised learning: learning with a teacher ! You had training data which was (feature, label) pairs and the goal was to learn a mapping from features to labels Clustering ! Unsupervised learning: learning without a

164 views • 12 slides

Learning and transferring mid-level image representions using convolutional neural networks

Willow project-team Learning and transferring mid-level image representions using convolutional neural networks Maxime Oquab, Lon Bottou, Ivan Laptev, Josef Sivic 1 mardi 5 aot 14 Image classification (easy) Is there a car ? Source :

776 views • 61 slides

LEARNING Slides adapted from Towards Data Science Outline Overview Architecture

Convolutional Networks CSCI 447/547 MACHINE LEARNING Slides adapted from Towards Data Science Outline Overview Architecture Intuition Example Visualization Overview Detects low level features Uses these to form

631 views • 23 slides

Convolutional Knowledge Tracing: Modeling Individualization in Student Learning Proces Shuanghong

Convolutional Knowledge Tracing: Modeling Individualization in Student Learning Proces Shuanghong Shen, Qi Liu, Enhong Chen, Han Wu, Zhenya Huang, Weihao Zhao, Yu Su, Haiping Ma, Shijin Wang SIGIR 2020 Anhui Province Key Lab. of Big Data

246 views • 13 slides

Convolutional dictionary learning based auto-encoders for natural exponential-family

Convolutional dictionary learning based auto-encoders for natural exponential-family distributions Bahareh Tolooshams *1 , Andrew H. Song *2 , Simona Temereanca 3 , and Demba Ba 1 1 Harvard University 2 Massachusetts Institute of Technology 3

494 views • 24 slides

ECE 6504: Deep Learning for Perception Topics: (Finish) Backprop Convolutional Neural

ECE 6504: Deep Learning for Perception Topics: (Finish) Backprop Convolutional Neural Nets Dhruv Batra Virginia Tech Administrativia Presentation Assignments https://docs.google.com/spreadsheets/d/

924 views • 67 slides

Hebbian Learning Algorithms for Training Convolutional Neural Networks Gabriele Lagani Computer

Hebbian Learning Algorithms for Training Convolutional Neural Networks Gabriele Lagani Computer Science PhD University of Pisa Outline SGD vs Hebbian learning Hebbian learning variants Training CNNs with Hebbian + WTA approach on

781 views • 25 slides

A spatiotemporal stochastic model for tropical precipitation and water vapor dynamics. Scott

A spatiotemporal stochastic model for tropical precipitation and water vapor dynamics. Scott Hottovy and Sam Stechmann (UW) shottovy@math.wisc.edu University of Wisconsin ONR DURIP grant N00014-14-1-0251 S. Hottovy, UW Spatiotemporal

381 views • 16 slides

Deep Convolutional Neural Nets COMPSCI 371D Machine Learning COMPSCI 371D Machine

Deep Convolutional Neural Nets COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Deep Convolutional Neural Nets 1 / 25 Outline 1 Why Neural Networks? 2 Circuits 3 Neurons, Layers, and Networks 4 Correlation and Convolution 5

287 views • 25 slides

Fully Convolutional Network (FCN) Prof. Seungchul Lee Industrial AI Lab. Deep Learning for

Fully Convolutional Network (FCN) Prof. Seungchul Lee Industrial AI Lab. Deep Learning for Computer Vision: Review Source: 6.S191 Intro. to Deep Learning at MIT 2 Segmentation Segmentation task is different from classification task because

522 views • 15 slides