End-to-end Learning of Action Detection from Frame Glimpses in - PowerPoint PPT Presentation

Nov 27, 2023 •128 likes •286 views

End-to-end Learning of Action Detection from Frame Glimpses in Videos CVPR 2016 Serena Yeung, Olga Russakovsky, Greg Mori , Li Fei-Fei Presenter: Wei-Jen Ko 1 Action detection Predict which and when action occurs in the video. 2 Related

End-to-end Learning of Action Detection from Frame Glimpses in Videos CVPR 2016 Serena Yeung, Olga Russakovsky, Greg Mori , Li Fei-Fei Presenter: Wei-Jen Ko 1
Action detection • Predict which and when action occurs in the video. 2
Related Work Motion features: Dense Trajectories Apearance features: CNN+SIFT+ COLOR Audio features: MFCC+ASR Classified by SVM over exhaustive segments with varying scale and temporal position. D. Oneata, J. Verbeek, and C. Schmid. The lear submission at thumos 2014. L. Wang, Y. Qiao, and X. Tang. Action recognition and detection by combining motion and appearance features J. Yuan, Y. Pei, B. Ni, P. Moulin, and A. Kassim. Adsc submission at thumos challenge 2015 3
Related Work Dynamic feature prioritization Predictive corrective networks Y-C. Su and K. Grauman. Leaving Some Stones Unturned: Dynamic A. Dave, O. Russakovsky, D. Ramanan. Predictive-Corrective Networks Feature Prioritization for Activity Detection in Streaming Video , ECCV for Action Detection, CVPR 2017. 2016. 4
Proposed method • Recurrent neural network-based end-to-end model • Decides which frame to observe next and when to emit a prediction. 5
Observation Network Video frame V ln Image fearure VGG On FC Frame location ln
Recurrent Network s n : start location of the action e n : end location of the action l n+ 1 : location of the video frame to observe next c n : confidence level of the prediction P n : prediction indicator S n, e n, l n+ 1 normalized to [0,1] 7
8
Loss function L cls (d n ): Cross-entropy loss on confidence Cn L loc (d n , g m ) : L2- regression loss minimizing the distance 9
p n and l n+1 trained by REINFORCE Reward function negative reward if did not emit predictions for videos containg instances 10
THUMOS’14 Results 11
If observed frames are not be determined dynamically, it does not provide sufficient resolution to localize action boundaries. 12
ActivityNet Results 13
Strengths: • First End-to-end training approach • Select important frames to observe, no exhaustive searching • Better results 14
15

Recommend

What is Action Learning? What is an ALC? 1 What is Action Learning? Learning by doing

What is Action Learning? What is an ALC? 1 What is Action Learning? Learning by doing Process for bringing together a group of people with varied levels of skills and experience (but with influence, passion, etc. to impact an

191 views • 5 slides

Reinforcement learning with restrictions on the action set Mario Bravo Universidad de Chile

Reinforcement learning with restrictions on the action set Reinforcement learning with restrictions on the action set Mario Bravo Universidad de Chile Joint work with Mathieu Faure (AMSE-GREQAM) Reinforcement learning with restrictions on the

976 views • 68 slides

Shared Action Learning: Supporting Collaboration and Critical Thinking Stephen McCauley &

Shared Action Learning: Supporting Collaboration and Critical Thinking Stephen McCauley & Scott Jiusto Worcester Polytechnic Institute NAFSA May 27, 2015 Workshop Introduction Workshop Objectives Introduce Shared Action Learning as

659 views • 44 slides

Learning Transferable Distance Functions For Human Action Recognition and Detection Weilong Yang

Learning Transferable Distance Functions For Human Action Recognition and Detection Weilong Yang Simon Fraser University 1 Action Recognition and Detection Walking Running Jogging Boxing Waving T X Y 2 Applications Action related

702 views • 33 slides

Action Recognition and Detection with Deep Learning Yue Zhao Multimedia Lab, CUHK

Action Recognition and Detection with Deep Learning Yue Zhao Multimedia Lab, CUHK https://zhaoyue-zephyrus.github.io Why do we need to understand action? Various real-world applications Anomaly detection in video surveillance

751 views • 30 slides

PATIENT ENGAGEMENT Learning Session LEARNING AND ACTION 2 NETWORK RACHELLE DUBOSE CARUTHERS,

4/13/2015 PATIENT ENGAGEMENT Learning Session LEARNING AND ACTION 2 NETWORK RACHELLE DUBOSE CARUTHERS, LMSW OUTREACH COORDINATOR RCARUTHERS@NW14.ESRD.NET 469-916-3800 OFFICE WELCOME AND INTRODUCTIONS Network 14 Staff 126 Dialysis

363 views • 13 slides

Learning for Action Recognition Yemin Shi shiyemin@pku.edu.cn 2018-03 1 Background Action

GPU Accelerated Sequence Learning for Action Recognition Yemin Shi shiyemin@pku.edu.cn 2018-03 1 Background Action Recognition Object Recognition (Video Classification) (Image Classification) Action Recognition VS Object Recognition

508 views • 49 slides

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B.

7. Motor Control and Reinforcement Learning Outline A. Action Selection and Reinforcement B. Temporal Difference Reinforcement Learning C. PVLV Model D. Cerebellum and Error-driven Learning 2/23/18 COSC 494/594 CCN 2 Sensory-Motor Loop

790 views • 56 slides

Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring

Foundations of Machine Learning Reinforcement Learning Reinforcement Learning Agent exploring environment. Interactions with environment: action state Agent Environment reward Problem: find action policy that maximizes cumulative reward

828 views • 66 slides

Reinforcement Learning for Continuous State and Action Spaces Gradient Methods 1 MACHINE LEARNING

MACHINE LEARNING 2012 MACHINE LEARNING TECHNIQUES AND APPLICATIONS Reinforcement Learning for Continuous State and Action Spaces Gradient Methods 1 MACHINE LEARNING 2012 Reinforcement Learning (RL) Supervised Learning Unsupervised

777 views • 51 slides

PATIENT ENGAGEMENT Learning Session LEARNING AND ACTION 4 NETWORK RACHELLE DUBOSE CARUTHERS,

7/21/2015 PATIENT ENGAGEMENT Learning Session LEARNING AND ACTION 4 NETWORK RACHELLE DUBOSE CARUTHERS, LMSW OUTREACH COORDINATOR RCARUTHERS@NW14.ESRD.NET 469-916-3800 OFFICE WELCOME AND INTRODUCTIONS Network 14 Staff Rachelle

702 views • 8 slides

Ron Weisinger An Action Learning Success Story Leading Effective Teams Greater Boston ASTD

Ron Weisinger An Action Learning Success Story Leading Effective Teams Greater Boston ASTD Meeting March 15, 2011 An Action Learning Success Story: Leading Effective Teams Pre-training Assessment An Action Learning Success Story: Leading

627 views • 17 slides

Learning and Adaptation for Sensorimotor Control October 24-26 2018 Action on-based ed l

Learning and Adaptation for Sensorimotor Control October 24-26 2018 Action on-based ed l learning i in sensor orimot otor or systems Per P Pet etersso son n Ume Um e/Lund Uni Universi sity What can we learn from

333 views • 30 slides

NJ SEPSIS LEARNING ACTION COLLABORATIVE, STATE AND CMS UPDATE Christa A. Schorr DNP, RN, NEA-BC,

1 NJ SEPSIS LEARNING ACTION COLLABORATIVE, STATE AND CMS UPDATE Christa A. Schorr DNP, RN, NEA-BC, FCCM Associate Professor of Medicine Nurse Scientist, Cooper Research Institute-Critical Care Outline 1. Discuss NJ learning collaborative plan

226 views • 19 slides

The Engineering Design Process In Action: Learning through MAKING (ocMakerChallenge) Jack

The Engineering Design Process In Action: Learning through MAKING (ocMakerChallenge) Jack Gupton, Educator, Walker Jr. High STEAM, Industrial Arts Jillian Johnson-Sharp, OCDE/CTE Partnership Eva Wolf, CEO, Airwolf 3D Learning through MAKING

362 views • 15 slides

Connecting Visual Representations and Action Research in the Teaching and Learning of Mathematics

Connecting Visual Representations and Action Research in the Teaching and Learning of Mathematics Classroom Action Research What is Classroom Action Research (CAR) ? n Systemic inquiry with the aim of informing practice in a particular

518 views • 38 slides

Douglas Academy School Response to Learning Together Scotland's National Action Plan on

Douglas Academy School Response to Learning Together Scotland's National Action Plan on parental involvement, parental engagement, family learning and learning at home 2018-2021 (August 2018) S Sinclair Overarching Themes Shift

333 views • 10 slides

The Reinforcement Learning Problem Robert Platt Northeastern University Agent Action Agent

The Reinforcement Learning Problem Robert Platt Northeastern University Agent Action Agent World Observation Reward On a single time step, agent does the following: 1. observe some information 2. select an action to execute 3. take note

374 views • 11 slides

Learning Semantic Visual Codebook for Action Recognition by Embedding into Concept Space Behrouz

Learning Semantic Visual Codebook for Action Recognition by Embedding into Concept Space Behrouz Saghafi Using Spatio-temporal Features Action recognition using silhouettes or optical flow encounters difficulties when dealing with non-uniform

556 views • 27 slides

Action Robust Reinforcement Learning and Applications in Continuous Control Chen Tessler *,

Action Robust Reinforcement Learning and Applications in Continuous Control Chen Tessler *, Yonathan Efroni* and Shie Mannor *equal contribution Poster #272 Action Robust Reinforcement Learning and Applications in Continuous Control Robust MDPs

502 views • 8 slides

Service-Learning VT Engage: Community Learning Collaborative Action.Reflection.Knowledge

Service-Learning VT Engage: Community Learning Collaborative Action.Reflection.Knowledge engage.vt.edu What is Service-Learning? A teaching and learning approach that integrates meaningful community service with instruction and reflection

324 views • 16 slides

Deep RL Robert Platt Northeastern University Q-learning Q-function Q action argmax state

Deep RL Robert Platt Northeastern University Q-learning Q-function Q action argmax state action World e t a t s Update rule Q-learning Q-function Q action argmax state action World e t a t s Update rule Deep Q-learning

971 views • 73 slides

Approximate Q-Learning 2/24/17 State Value V(s) vs. Action Value Q(s,a) " # X t r t

Approximate Q-Learning 2/24/17 State Value V(s) vs. Action Value Q(s,a) " # X t r t V = E t =0 Either way, value is the sum of future discounted rewards (assuming the agent behaves optimally): V(s) : after being in state s

404 views • 17 slides

Learning Action Representations for Reinforcement Learning Georgios Scott Yash James Philip

Learning Action Representations for Reinforcement Learning Georgios Scott Yash James Philip Theocharous Jordan Chandak Kostas Thomas Reinforcement Learning Problem Statement Thousands of possible actions! Problem Statement Thousands of

611 views • 31 slides