Camera Motion Identification in the Rough Indexing Paradigm Petra KRÄMER and Jenny BENOIS-PINEAU LaBRI – University Bordeaux I, France {petra.kraemer,jenny.benois}@labri.fr Camera Motion Identification in the Rough Indexing Paradigm – p.1/21 TRECVID 2005 – P . KRÄMER and J.BENOIS-PINEAU
Introduction Task: Given the shot boundary reference Identify the shots in which a certain camera motion (pan, tilt, zoom) is present Rough Indexing Paradigm: Work on a lower spatial and temporal resolution i.e. P-Frames Aim: Reuse motion low-level descriptors from the compressed stream Main challenge in TRECVID 2005: Jitter camera motion due to hand-carried cameras Camera Motion Identification in the Rough Indexing Paradigm – p.2/21 TRECVID 2005 – P . KRÄMER and J.BENOIS-PINEAU
Overview P-Frames 1 Global Motion Estimation ˆ θ j 2 Signifi cance Value Computation s j 3 Motion Segmentation ¯ s m 4 Thresholding ¯ ζ m 5 Classifi cation Motion feature j related to frames, m related to segments of homogeneous motion Camera Motion Identification in the Rough Indexing Paradigm – p.3/21 TRECVID 2005 – P . KRÄMER and J.BENOIS-PINEAU
Overview P-Frames 1 Global Motion Estimation ˆ θ j 2 Signifi cance Value Computation s j 3 Motion Segmentation ¯ s m 4 Thresholding ¯ ζ m 5 Classifi cation Motion feature j related to frames, m related to segments of homogeneous motion Camera Motion Identification in the Rough Indexing Paradigm – p.3/21 TRECVID 2005 – P . KRÄMER and J.BENOIS-PINEAU
Global Motion Estimation Robust global motion estimator for P-Frames [DBP01]: P Estimation of the affi ne 2D motion model: 1 � � � � � � � � ˆ θ j dx i a 1 a 2 a 3 x i = + 2 dy i a 4 a 5 a 6 y i s j 3 Based on the weighted least squares method: ¯ s m 4 θ = ( H T WH ) − 1 H T WZ ˆ ¯ ✛ ✘ ζ m 5 ˆ = ( a 1 , a 2 , a 3 , a 4 , a 5 , a 6 ) T θ Mf MPEG motion compensation vectors Z macroblock centers H ✚ ✙ weights defi ned by the derivative of the Tukey function W Camera Motion Identification in the Rough Indexing Paradigm – p.4/21 TRECVID 2005 – P . KRÄMER and J.BENOIS-PINEAU
Global Motion Estimation P The derivative of the Tukey function: 1 r ( r 2 − λ 2 � r ) 2 if | r | < λ r ˆ θ j ψ ( r, λ r ) = 0 otherwise 2 s j The weights are [OB95]: 3 ¯ s m w i = ψ ( r i ) 4 ¯ r i ✛ ✘ ζ m ψ 5 10 8 threshold λ r 6 4 Mf residuals r i = z i − ˆ z i 2 0 i -th MPEG motion vector z i -2 ✚ ✙ -4 estimation of z i z i ˆ -6 -8 -10 -4 -3 -2 -1 0 1 2 3 4 Camera Motion Identification in the Rough Indexing Paradigm – p.5/21 TRECVID 2005 – P . KRÄMER and J.BENOIS-PINEAU
Global Motion Estimation Motion Compensation Vectors (29087) Estimated Vectors (29087) P 150 150 100 100 1 50 50 ˆ θ j 0 0 2 -50 -50 s j -100 -100 3 ✬ ✩ a) b) ¯ -150 -150 s m -200 -150 -100 -50 0 50 100 150 200 -200 -150 -100 -50 0 50 100 150 200 4 a) P-Frame motion vectors ¯ ζ m b) Estimated vectors 5 c) Macroblocks: Mf Outliers Dominant estimation support D ✫ ✪ ( w i > 0 ) c) Camera Motion Identification in the Rough Indexing Paradigm – p.6/21 TRECVID 2005 – P . KRÄMER and J.BENOIS-PINEAU
Global Motion Estimation Problem: P The global motion parameters are noisy due to jitter motions. 1 The global motion parameters have different meanings. ˆ θ j 2 Solution: s j Signifi cance test of the motion parameters: 3 Thresholding of likelihood values ¯ s m 4 ¯ ζ m 5 Mf Camera Motion Identification in the Rough Indexing Paradigm – p.7/21 TRECVID 2005 – P . KRÄMER and J.BENOIS-PINEAU
Significance Value Computation Based on [BGG99]: P Change to another basis of elementary motion-subfi elds: 1 ˆ φ = ( pan, tilt, zoom, rot, hyp 1 , hyp 2) with θ j 2 zoom = 1 rot = 1 2 ( a 2 + a 6 ) 2 ( a 5 − a 3 ) s j hyp 1 = 1 hyp 2 = 1 2 ( a 2 − a 6 ) 2 ( a 3 + a 5 ) 3 ¯ s m Consider two hypotheses H 0 and H 1 4 H 0 : the considered component of φ is signifi cant ¯ ζ m with ˆ φ 0 as the corresponding motion model 5 H 1 : the considered component of φ is not signifi cant ( = 0 ) Mf with ˆ φ 1 as the corresponding motion model Camera Motion Identification in the Rough Indexing Paradigm – p.8/21 TRECVID 2005 – P . KRÄMER and J.BENOIS-PINEAU
Significance Value Computation P Likelihood function associated to each hypothesis: 1 � �� 1 � − 1 f (ˆ � i Σ − 1 2( r T φ l ) = exp r i ) ˆ θ j l � 2 π det(Σ l ) 2 i ∈ D 1 s j = (2 πσ x,l σ y,l ) || D || exp ( −|| D || ) , l = 0 , 1 3 ¯ s m Assumption: � � 4 σ 2 0 x,l Σ l = ¯ ζ m σ 2 0 y,l 5 ✗ ✔ Mf covariance matrix Σ σ x , σ y variances for x and y ✖ ✕ dominant estimation support D Camera Motion Identification in the Rough Indexing Paradigm – p.9/21 TRECVID 2005 – P . KRÄMER and J.BENOIS-PINEAU
Significance Value Computation P The signifi cance value s is: 1 � � f (ˆ φ 1 ) s = ln = || D || (ln( σ x, 0 σ y, 0 ) − ln( σ x, 1 σ y, 1 )) ˆ θ j f (ˆ φ 0 ) 2 = ∗ � ln( σ 2 0 ) − ln( σ 2 � || D || 1 ) s j 3 ∗ assuming that σ x = σ y ¯ s m 4 Aim: Use s to test the signifi cance ¯ ζ m Idea: 5 If a motion feature (pan, zoom, tilt) is present in a shot, its corresponding motion parameter is signifi cant during a suffi cient Mf number of frames. Camera Motion Identification in the Rough Indexing Paradigm – p.10/21 TRECVID 2005 – P . KRÄMER and J.BENOIS-PINEAU
Significance Value Computation Problem: P The signifi cance values can be noisy due to jitter motions. 1 The motion models ˆ θ can be inaccurate. ˆ θ j 2 Solution: s j 3 Smooth the signifi cance value along the time and take decision on ¯ s m the temporal mean value. 4 –> Segment shots into subshots of homogeneous motion ¯ ζ m 5 Introduce confi dence measures in order to reject frames with an inaccurate motion model. Mf Camera Motion Identification in the Rough Indexing Paradigm – p.11/21 TRECVID 2005 – P . KRÄMER and J.BENOIS-PINEAU
Significance Value Computation Two reasons for inaccurate motion models: P Failure of the MPEG encoder 1 –> Confi dence measure c D ≈ || D || ˆ θ j Failure of the global motion estimation algorithm 2 –> Confi dence measure c σ ≈ σ 2 s j 0 3 ¯ s m Reject of the frame if: c D < λ D || c σ > λ σ 4 ¯ ζ m ✎ ☞ 5 threshold λ D Mf ✍ ✌ threshold λ σ Camera Motion Identification in the Rough Indexing Paradigm – p.12/21 TRECVID 2005 – P . KRÄMER and J.BENOIS-PINEAU
Motion Segmentation Hinkley test to detect changes on the temporal mean value ¯ s ( t ) : P Downward jump: k � � s + δ min � U k = s t − ¯ ( k ≥ 0) 1 2 ˆ t =0 θ j = 0 ≤ i ≤ k U i ; detection if M k − U k > λ H max M k 2 s j 3 Upward jump: k s m ¯ � � s − δ min � = s t − ¯ ( k ≥ 0) V k 4 2 ¯ t =0 ζ m = 0 ≤ i ≤ k V i ; detection if V k − N k > λ H min N k 5 ✗ ✔ Mf temporal mean value ¯ s minimal jump magnitude δ min ✖ ✕ predefi ned threshold λ H Camera Motion Identification in the Rough Indexing Paradigm – p.13/21 TRECVID 2005 – P . KRÄMER and J.BENOIS-PINEAU
Motion Segmentation Principle of the Hinkely test: P 1 s and ¯ s ˆ θ j 2 s j 3 Down s m ¯ 4 M k − U k ¯ ζ m 5 Up Mf V k − N k Camera Motion Identification in the Rough Indexing Paradigm – p.14/21 TRECVID 2005 – P . KRÄMER and J.BENOIS-PINEAU
Thresholding P Selection of the hypothesis: H 0 1 t = T 1 < ˆ � θ j s ( t ) = ¯ s ( t ) λ s T − t 0 > 2 t = t 0 H 1 s j 3 And relative thresholding to determine the dominant motion: ¯ s m 4 � ¯ s ( t ) ¯ if ¯ s ( t ) < α · min { ¯ s pan , ¯ s tilt , ¯ s zoom , ¯ s rot , ¯ s hyp 1 , ¯ s hyp 2 } ζ m ¯ ζ ( t ) = 5 0 otherwise ✗ ✔ Mf segment of homogeneous motion T − t 0 threshold λ s ✖ ✕ constant α Camera Motion Identification in the Rough Indexing Paradigm – p.15/21 TRECVID 2005 – P . KRÄMER and J.BENOIS-PINEAU
Classification The following classifi cation scheme is applied to the thresholded mean P signifi cance values ¯ ζ = (¯ ζ pan , ¯ ζ tilt , ¯ ζ zoom , ¯ ζ rot , ¯ ζ hyp 1 , ¯ ζ hyp 2 ) : 1 ˆ ¯ motion feature θ j ζ 2 1 static camera/ no signifi cant motion (0 , 0 , 0 , 0 , 0 , 0) s j (¯ 2 pan ζ pan , 0 , 0 , 0 , 0 , 0) 3 (0 , ¯ ¯ s m 3 tilt ζ tilt , 0 , 0 , 0 , 0) 4 (¯ ζ pan , ¯ ζ tilt , ¯ 4 zoom ζ zoom , 0 , 0 , 0) ¯ ζ m 5 others complex camera motion 5 Mf Camera Motion Identification in the Rough Indexing Paradigm – p.16/21 TRECVID 2005 – P . KRÄMER and J.BENOIS-PINEAU
Classification Postprocessing: P Join neighbored segments with the same motion feature Reject segments with a duration shorter than t min frames 1 ˆ θ j t min t 2 s j 3 ¯ s m 4 ¯ ζ m 5 If a motion feature is still present: Mf The shot is identifi ed to contain the motion feature. Camera Motion Identification in the Rough Indexing Paradigm – p.17/21 TRECVID 2005 – P . KRÄMER and J.BENOIS-PINEAU
Recommend
More recommend
