Annotation of Video and Film Matthias Zeppelzauer St. Plten - PowerPoint PPT Presentation

Symposium: Film Rechnen Computerbasierte Methoden in der Filmanalyse July 3, 2017 Automated Analysis, Retrieval and Annotation of Video and Film Matthias Zeppelzauer St. Pölten University of Applied Sciences

Motivation Manual annotation = time-consuming & tedious task July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Motivation Manual annotation = time-consuming & tedious task Goal: Annotate content automatically! Automated Film Analysis July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Outline ‣ Methods for automated film analysis ‣ Temporal segmentation ‣ Motion composition retrieval ‣ Visual composition retrieval ‣ Montage analysis ‣ Outline of future research July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Temporal Segmentation July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Step 1: Shot Segmentation ‣ Shot = continuous sequence of frames recorded from a single camera ‣ Basic building block for high-level film analysis ‣ Motion analysis, montage patterns, rhythm analysis, … ‣ Shot boundary types ‣ Abrupt transitions (shot cuts) ‣ Gradual transitions (dissolves, fades. …) ... ... July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Step 1: Shot Segmentation 400 time similarity of frame 400 and frame 250 250 time 7 July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Step 2: Scene Segmentation July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Step 2: Scene Segmentation ‣ Bottom-up approach ‣ Basis: shots ‣ Audio + visual similarity  links ‣ Fuse linked shots into scenes Scene 1 Scene 2 July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Motion Composition Retrieval ‣ Retrieve scenes with particular compositions of camera and object motion ‣ Step 1: motion tracking and segmentation ‣ Step 2: retrieval of motion compositions July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Step 1: Motion Tracking and Segmentation ‣ Motion trajectories  clustering  motion components Y X Time July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Step 2: Retrieve Motion Compositions use segments for retrieval! July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Step 2: Retrieve Motion Compositions ‣ Goal: Search and retrieval of user-defined motion compositions ‣ Input = Query: Sketch motion components as vectors ‣ Camera motion, object motion, groups of objects pan/large object diagonal motion e.g. hammering July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Step 2: Retrieve Motion Compositions traveling right, object to left, group to left people, horses, tractors move diagonally hammering, working people, trumpeter July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Motion Continuity Retrieval ‣ “Motion Continuity refers to the matching of individual scenic elements from shot to shot so that details and actions, filmed at different times will edit together without error” ‣ Consistent screen direction ‣ Example: chasing scene ‣ Retrieve user-specified combinations of motion continuity July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Retrieval of Visual Composition July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Retrieval of Visual Composition ‣ Visual composition = spatial arrangement of visual elements inside an image ‣ Manual search: time consuming + subjective ‣ Automatic retrieval possible? July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Retrieval of Visual Composition ‣ Global image analysis ‣ User-study for evaluation: ‣ User define composition to search for.. July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Retrieval of Visual Composition ‣ Example results July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at) July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at) 20

Montage Analysis ‣ Synchronous Montage: ‣ Correlations between soundtrack and cutting of movie ‣ Stylistic means to highlight important scenes and events ‣ Goal: extract sequences automatically ‣ Benefit: basis for video summarization, abstraction, annotation… July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Extraction of Synchronous Montage Sequences ‣ Correlate shot boundaries with audio onsets July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Extraction of Synchronous Montage Sequences ‣ Step 1: Shot cut detection ‣ Step 2: Audio analysis ‣ Step 3: Correlation extraction ‣ Step 4: Sequence extraction July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Extraction of Synchronous Montage Sequences - Results ‣ Example ‣ “October” (Eisenstein, 1928): protest on the street July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Extraction of Synchronous Montage Sequences ‣ Retrieved sequences: dialogue sequences, action scenes, parallel montage  rich semantics ‣ Applications: ‣ highlight extraction ‣ scene segmentation ‣ summarization July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Conclusion ‣ Automatic film analysis has great potential, e.g. ‣ Temporal segmentation  extraction of shots / scenes ‣ Motion composition retrieval  retrieve typical camera / object motions ‣ Visual composition retrieval  find similar image compositions / framings ‣ Synchronous montage extraction  extract highlights / montage patterns / cross-cutting July 3, 2017 Matthias Zeppelzauer (m.zeppelzauer@fhstp.ac.at)

Thank you for your Attention! Questions? Contact: Matthias.Zeppelzauer@fhstp.ac.at