AI Animation Team Clip1 Clip2 Clip4 Clip3 Clip5 Clip6 Clip7 - PowerPoint PPT Presentation

AI Animation Team

Clip1 Clip2 Clip4 Clip3 Clip5 Clip6 Clip7 Clip8 clip9

Automatically Music Dance Animation

Neural Music Animation Network Noise In Output Lack of Paired Data

Paired match • 3D Skeleton Animation • Music Paired Choreography • 2D Skeleton Animation Video • Music huge number of them available

“Unlimited” Music 2D Animation Feature Neural Animation Match Network （ 3D ） Pre-recorded 3D Skeleton Animation Break into smaller pieces Limited then reconstruct

How do we match 2D animation with 3D animation ？ • Position (Pose) • Velocity Velocity • Acceleration • Motion Beats Motion Beats

Melody Tabs 1.5 0.25 0.5 0.75 1.25 0 1.0 Phase Phase Phase Phase Phase Phase Phase Phase Beat 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Counts 16 Beats One Cycle Choreography Labanotation

Segmented Dance Clips # 1 • Cut dance cycle into smaller sections # 2 # 2 base on motion beats # 3 • # 4 # 4 Reconstruct dance cycle from these smaller sections based on phase # 5 # 6 # 6 # 7 # 7

Music 2D Animation Feature Phase-based Neural Animation Network Match （ 3D ） Pre-recorded 3D Skeleton Animation

Dance Matching System Overview Train Dance Video 1 Dance 2 Matched 3 Motion Dance Audio-Motion Feature Sequence Phase Based Animation Dance Audio Track Interpreter Dance Matching Warping Animation Network Dance Motion Motion Feature Recorded Dance Clip Curve Animation Segments Analysis

System Overview Intelligent Choreography Pipeline Structure Audio Features Audio Track Video Clip 1 Audio Motion Audio-Motion train Track Feature Interpreter … … Network Choreography Choreography Video Video Playlist Phase Video Track Dance Motion Features 2 Human Phase Based Pose Dance Matching Detect Clip Segments Fine Grained Matched Sequence Matching Clip Segments 3 … … 3D Dance Animation 1 hour mocap dataset Sequence Warping

Audio-Motion Interpreter Network Database Structure YouTube – Choreography Playlists ~ 200 videos 3-6 minutes / video • Professional Choreography Studios • K-pop/Urban/Chinese Traditional • Varying bpm 120-200 Challenges Multiple Dancers Moving Camera Sliced Scenes 2D Pose Extraction 16 hours 36 min 500 hours

Audio-Motion Interpreter Network Automatic Pipeline And the Model Choice Mel-spec Music CQT Beat Info ConvNets Audio Librosa Stream Aubio Video Filter Mapping Network Dance Video Phase-based Slice and Clean Conv1D/LSTM+Conv2 LSTM D • Dancer-focused Mixture Density • Stable Camera Network Video 2D Pose • Continuous Steam Extraction Beat Speed Info Mixture Density Networks

Audio-Motion Interpreter Network Modelling The Mapping From Audio Feature to Motion Features Problem Definition Audio Audio Track Features Audio-Motion Dance Motion Phase Based Interpreter Features Dance Matching Network Mel-spec Temporal and Frequency Zero-Padding Fixed Length Different Length based on bpm

Audio-Motion Interpreter Network Modelling The Mapping From Audio Feature to Motion Features Problem Definition Audio Audio Track Features Audio-Motion Dance Motion Phase Based Interpreter Features Dance Matching Network Mel-spec One Motion Curve For each Joint time Dance type Motion delta angle Features max duration vel 8 Major Joints Shoulder/Elbow/Keen/Ankle

Audio-Motion Interpreter Network Modelling The Mapping From Audio Feature to Motion Features Problem Definition

Audio-Motion Interpreter Network Modelling The Mapping From Audio Feature to Motion Features Problem Definition Audio Audio Track Features Audio-Motion Dance Motion Phase Based Interpreter Features Dance Matching Network Mel-spec Training Time 6 hours 40mins 84hours 40mins

Audio-Motion Interpreter Network Modelling The Mapping From Audio Feature to Motion Features Problem Definition Audio Audio Track Features Audio-Motion Dance Motion Phase Based Interpreter Features Dance Matching Network Mel-spec Inference Time for 16-beat 18 ms 1500 ms

Audio-Motion Interpreter Network Network Structure and Training procedure Online Choreography Video Playlist New Audio Audio Track Track Music Waveform Fragments Waveform Fragments Mel Spectrograms Audio-Motion Video Track Interpreter ConvNets Dance Motion Features Training ConvNets 2D Motion Fine Grained Dance Matching Motion Curve Analysis Module Dance Motion 3D Dance Features Sequence

Phase-based Dance Matching 2.1 Motion Curve Analysis and Retrieval Module Beat Max Velocity Detected Video Beat 3D Dance 2D Motions Frames Strength Animation Clips Global Guideline Beats Delta Distance Curves Motion Beat 3D Dance Beats Distribution Breakpoints Clip Curve Retrieval Analysis Lower Body Thigh Calf Rhythm Local Fore-arm Feature Upper-arm Upper Body Curves Rhythm Local Angle Speed Curves

Phase-based Dance Matching 2.2 Choreography Phase Circle Melody Tabs 1.5 0.25 0.5 0.75 1.25 0 1.0 Phase Phase Phase Phase Phase Phase Phase Phase Beat 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Counts 16 Beats One Phase Choreography Labanotation

Phase-based Dance Matching 2.3 Phase-based Fine-grained Data Preparation Segmented Dance Clips Phase-based Slicing Automatic Dataset Preprocess # 1 # 1 # 2 # 2 # 3 # 3 Motion Fine- Inside Phase Curve grained Breakpoints # 4 # 4 Analysis Slicing Distribution # 5 # 5 Breakpoint Breakpoint # 6 # 6 #1 #2 # 7 # 7 # 8 # 8 Mocap Phase-based Clips Animation Clip 256 frames

Orientation Phase-based Pose Similarity Dance Matching Velocity Vector Angel Velocity 2.4 Phase-based Fine-grained Motion Matching Segmented Dance Clips Audio-Motion # 1 Interpreter Dance Phase Breakpoint #0 Network # 2 # 2 Clip # 2 Segment # 1 2/16 Bests # 3 Dance Motion Features Dance Phase Breakpoint #1 Guided Match # 4 Clip # 4 # 4 Phase # 2 # 5 12/16 Bests Breakpoint #3 # 6 6/16 Bests Dance Phase Breakpoint #2 # 7 # 7 Clip # 8 Clip # 7 Phase # 2 Phase # 3 # 8 # 8 Matched Phase Potential Candidates

Animation Warping Enhance the Emotional Impact Original Playback 1.0x Speed Original Animation Beat Sequence Time Warp Music Mechanism Beat Sequence Adjusted Align Animation Animation Beats with Music Beats Beat Sequence Adjusted Playback Speed 1.0x

Animation Warping Enhance the Emotional Impact Original Playback 1.0x Speed Matching Beats Original According to Strength Animation Beat Sequence Time Warp Music Mechanism Beat Sequence Adjusted Animation Beat Sequence Align Animation Beats with Music Beats

Animation Warping Enhance the Emotional Impact Time Warping Curve

Animation Warping Enhance the Emotional Impact Time Warping Curve Enhanced Beats

Animation Warping Enhance the Emotional Impact Before Beats Aligned

Huge Pipeline Efficiency Boost From GPU Computation Dance Video 0 2D Pose Extraction For Training 1 Dance 2 Matched Motion Dance Audio-Motion Feature Sequence Phase Based Animation Dance Audio Track Interpreter Dance Matching Warping Animation Network Phase 0 Phase 1 Phase 2 Training Time 16 hour 6 hour 2 hour Phase 1 Phase 2 Phase 0 84 hour 20 hour 500 hour

System Performance on UGC platform Music > {neural network}>{matching}> Animation

Dance Matching Online Feedback 10000000+ request UGC Widely Spread in Multiple Video Platform

Dance Matching Online UGC on Bilibiili

This System is Currently Powering