Video based Animation Synthesis with the Essential Graph Adnane - PowerPoint PPT Presentation

Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes

Goal Given a set of 4D models, how to generate realistic motion from user specified constraints ? ¡ Output: Input: • Video based 4D models of elementary • Novel, user guided shape and appearance movements animation 2 ¡

Motivation Human animation generation: Where: • Digital media production : – Video Game industry – Motion Picture industry – Virtual Reality applications Rise of the Tomb Raider, Cristal Dynamics Game Of Thrones, HBO 3 ¡

Motivation Human animation generation: Where: • Digital media production : – Video Game industry – Motion Picture industry Virtual Reality applications – How: Rise of the Tomb Raider, Cristal Dynamics Game Of Thrones, HBO • Physical modeling: – Computationally expensive – Model limitations 4 ¡

Motivation Human animation generation: Where: • Digital media production : – Video Game industry – Motion Picture industry – Virtual Reality applications How: Rise of the Tomb Raider, Cristal Dynamics Game Of Thrones, HBO • Physical modeling: – Computationally expensive – Model limitations • Example data Reuse: – Key-framing data: • Cost-wise expensive: 100-250$ / character second • Time-wise expensive: 2-3 seconds of finished character animation per day 5 ¡

Motivation Human animation generation : Where: • Digital media production : – Video Game industry – Motion Picture industry – Virtual Reality applications How: Rise of the Tomb Raider, Cristal Dynamics Game Of Thrones, HBO • Physical modeling: Synthetic Motion – Computationally expensive – Model limitations • Example data Reuse: – Key-framing data: • Cost-wise expensive: 100-250$ / character second • Time-wise expensive: 2-3 seconds of finished character animation per day 6 ¡

Motivation Human animation generation: Where: • Digital media production : – Video Game industry – Motion Picture industry – Virtual Reality applications How: Rise of the Tomb Raider, Cristal Dynamics Game Of Thrones, HBO • Physical modeling: Synthetic Motion – Computationally expensive – Model limitations • Example data Reuse: – Key-framing data: • Cost-wise expensive: 100-250$ / character second CMU mocap dataset • Time-wise expensive: 2-3 seconds of finished character animation per day – Motion capture data: ü Real Motion 7 ¡

Motivation Human animation generation: Where: • Digital media production : – Video Game industry – Motion Picture industry – Virtual Reality applications How: Rise of the Tomb Raider, Cristal Dynamics Game Of Thrones, HBO • Physical modeling: Synthetic Motion – Computationally expensive – Model limitations • Example data Reuse: – Key-framing data: • Cost-wise expensive: 100-250$ / character second CMU mocap dataset • Time-wise expensive: 2-3 seconds of finished character animation per day – Motion capture data: Synthetic Shape ü Real Motion 3D character animated with mocap 8 ¡

Motivation Human animation generation: Where: • Digital media production : – Video Game industry – Motion Picture industry – Virtual Reality applications How: Rise of the Tomb Raider, Cristal Dynamics Game Of Thrones, HBO • Physical modeling: Synthetic Motion – Computationally expensive – Model limitations • Example data Reuse: – Key-framing data: • Cost-wise expensive: 100-250$ / character second CMU mocap dataset • Time-wise expensive: 2-3 seconds of finished character animation per day – Motion capture data: Synthetic Shape ü Real Motion – Surface capture data : ü Real Motion ü Real Shape ü Real Appearance 3D character animated with mocap 4D model, Thomas dataset 9 ¡

Surface Capture Reuse Surface Appearance Surface Multi-view temporal projection on videos Reconstruction tracking geometry • Mesh sequence with • Mesh sequence • 4D textured time-consistent with time-variant model topology topology 10 ¡

Surface Capture Reuse Surface Appearance Surface Multi-view temporal projection on videos Reconstruction tracking geometry • Mesh sequence with • Mesh sequence • 4D textured time-consistent with time-variant model topology topology Data: • Cyclic human movements: walk, run, jump, etc. • Acyclic human movements: dance. Thomas, Cathy 11 ¡

Surface Capture Reuse Surface Appearance Surface Multi-view temporal projection on videos Reconstruction tracking geometry • Mesh sequence with • Mesh sequence • 4D textured time-consistent with time-variant model topology topology Data: • Cyclic human movements: walk, run, jump, etc. • Acyclic human movements: dance. Data Reuse: Thomas, Cathy Generate continuous motion stream using basic operations on motion segments : • Rigid transformations • Concatenation • Smooth transition generation 12 ¡

Issues and challenges User control: Ÿ Intuitive formulation of user defined constraints Data organization: Numerical realism criterion: Ÿ A data structure organizing the input sequences and encoding selected transitions between them Ÿ Transition evaluation Motion synthesis: Ÿ Generating synthetic motion transitions. Ÿ Concatenating real and synthetic motion segments. 13 ¡

Issues and challenges User control: Ÿ Intuitive formulation of user defined constraints Data organization: Numerical realism criterion: Ÿ A data structure organizing the input sequences and encoding selected transitions between them Ÿ Transition evaluation Motion synthesis: Ÿ Generating synthetic motion transitions. Ÿ Concatenating real and synthetic motion segments. Challenges : • Limited data: • Sensitive data: • Make exhaustive use of it • Robust mesh processing technique • User perceptual acuity: • Complex dynamics: • Reliable numerical realism criterion • Robust transition generation technique ¡ • Optimal results in terms of said criterion ¡ 14 ¡

Comparison of Graph Based Approaches Automatic/ Input data Data organization supervised Motion Motion Graph(kovar02)(Arikan02) automatic Motion graph Capture 3D Surface Surface Motion Graph(Huang09) automatic Motion graph Capture 4D Surface Parametric 4D Parametric Motion Graph(Casas13) supervised Capture motion graph 4D Surface Essential Essential Graph(Boukhayma et. Boyer15) automatic Capture graph 15 ¡

Contributions • An optimal structure for motion data organization and reuse: The essential graph • Improving realism in synthetic motion transitions through dynamic time warping and variable length blended segments • A novel high-level constraint formulation for motion synthesis: 3D behavioral path synthesis 16 ¡

Approach Motion synthesis pipeline : 17 ¡

Input Data Organization 18 ¡

Input Data Organization Graph structure: • Node = frame/pose • Edge = transition • Edge weight = transition cost Input sequences 19 ¡

Input Data Organization Graph structure: • Node = frame/pose • Edge = transition • Edge weight = transition cost Input sequences We need to add new transitions : 20 ¡

Input Data Organization Graph structure: • Node = frame/pose • Edge = transition • Edge weight = transition cost Input sequences Motion Graph(kovar02) • Adding edges – Local minima in similarity matrix – Threshloding Similarity matrix, 21 ¡

Input Data Organization Graph structure: • Node = frame/pose • Edge = transition • Edge weight = transition cost Input sequences Motion Graph(kovar02) • Adding edges – Local minima in similarity matrix – Threshloding Similarity matrix, 22 ¡

Input Data Organization Graph structure: • Node = frame/pose • Edge = transition • Edge weight = transition cost Input sequences Motion Graph(kovar02) • Adding edges – Local minima in similarity matrix – Threshloding Similarity matrix, 23 ¡

Input Data Organization Graph structure: • Node = frame/pose • Edge = transition • Edge weight = transition cost Input sequences Motion Graph(kovar02) • Adding edges – Local minima in similarity matrix – Threshloding • Improvements – Interpolated Motion Graph(Sofanova07) Similarity matrix, – Well-Connected Motion Graph(Zhao08) – Optimization-based Motion Graph(Ren10) 24 ¡

Essential graph Essential Graph • Creating a complete digraph – Connecting all nodes together with directed edges and transition costs as weights Input sequences Complete digraph 25 ¡

Essential graph Essential Graph • Creating a complete digraph – Connecting all nodes together with directed edges and transition costs as weights Input sequences • Extracting the essential sub-graph – For each node • Calculate the shortest path tree rooted at said node Complete digraph 26 ¡