MCG-ICT-CAS TRECVID 2008 Automatic Video 2008 Automatic Video - PowerPoint PPT Presentation

MCG-ICT-CAS TRECVID 2008 Automatic Video 2008 Automatic Video Retrieval System Retrieval System Juan Cao, Yong-dong Zhang, , g g g, Bai-lan Feng, Xiu-feng Hua, Lei Bao, Xu Zhang INSTITUT INS TE O E OF CO Multimedia Computing Group COMPUTIN Institute of Computing Technology MPUTING Chinese Academy of Sciences G TE TECHN CHNOLOGY NIST TRECVID Workshop November 17,2008

Outline INSTITUTE OF COMPUTING TECHNOLOGY � Overall system � Review of baseline retrieval Review of baseline retrieval � Performance analysis � Concept-based retrieval � Re-ranking � Dynamic fusion � Conclusion � Conclusion

INSTITUTE OF COMPUTING TECHNOLOGY System Overview

Review of baseline retrieval R i f b li t i l INSTITUTE OF COMPUTING TECHNOLOGY � Text-based retrieval 0.009 � ASR shot matching g A window of 3 shots � Pre-processing � Pre processing Stop words removing stemming � Indexing � Indexing lucence

Review of baseline retrieval R i f b li t i l INSTITUTE OF COMPUTING TECHNOLOGY 0.009 � Text-based retrieval � Visual-based retrieval 0.033 � Feature extraction EH CM Sift EH CM Sift-visual-keywords i l k d Early fusion and LDA embedding � Retrieval model � Retrieval model Multi-bag SVM cosine-similarity � Fusion SSC dynamic fusion

Review of baseline retrieval R i f b li t i l INSTITUTE OF COMPUTING TECHNOLOGY � Text-based retrieval 0.009 � Visual-based retrieval � Visual based retrieval 0.033 0 033 � HLF-based retrieval 0.029 � Concept detectors C d CU-VIREO374 � � Retrieval Model R t i l M d l Multi-bag svm � [Acknowledgement]: Thank Dr. Yu-Gang Jiang for great help in the experiments. p p

Review of baseline retrieval R i f b li t i l INSTITUTE OF COMPUTING TECHNOLOGY � Text-based retrieval 0.009 � Visual-based retrieval � Visual based retrieval 0.033 0 033 � HLF-based retrieval 0.029 � Concept-based retrieval 0.044 p � Keywords mapping � DBCS mapping � DBCS mapping

Review of baseline retrieval Review of baseline retrieval INSTITUTE OF COMPUTING TECHNOLOGY � Text-based retrieval 0.009 0 009 � Visual-based retrieval 0.033 � HLF-based retrieval 0.029 24% � Concept-based retrieval 0.044 � Re-ranking 0.036 � Face � Face � motion

R Review of baseline retrieval i f b li t i l INSTITUTE OF COMPUTING TECHNOLOGY � Text-based retrieval 0.009 � Visual-based retrieval 0.033 � HLF-based retrieval 0.029 � Concept-based retrieval 0.044 � Re-ranking 0.036 � SSC Dynamic fusion � SSC Dynamic fusion

Query-to-concept mapping Q t t i INSTITUTE OF COMPUTING TECHNOLOGY Semantic similarity retrieve the Data: most similar most similar query(textual description,visual examples) Related work concepts Aim: Max{similarity( query, concept)} Statistic similarity Statistic similarity reduce the d th most Data: Collection(ASR text, concept distribution) co-occurrence concepts Aim: Max {p(query , concept)}

Query-to-concept mapping Q t t i INSTITUTE OF COMPUTING TECHNOLOGY Semantic similarity retrieve the Data: most similar most similar query(textual description,visual examples) Correct useful Related work concepts to describe the query to identify the query Aim: Max{similarity( query, concept)} Statistic similarity Statistic similarity reduce the d th most Data: Collection(ASR text, concept distribution) co-occurrence concepts Aim: Max {p(query , concept)}

Wh t i What is useful ? f l ? INSTITUTE OF COMPUTING TECHNOLOGY � Discriminability-ranking � The distributions fluctuate widely between � The distributions fluctuate widely between the given category and the others, but remain stable within this one. i t bl ithi thi � Factors � Factors � Difference of the concept distribution � Detector performance � Collection characteristic � Collection characteristic

Distribution Based Concept Distribution Based Concept Selection Framework(DBCS) INSTITUTE OF COMPUTING TECHNOLOGY VAC: the difference between categories 2 2 ← − − ∑ ∑ VAC t VAC t c ( , ( ) ) sign F t c i ( ( F t ( ( , ) ) F t F t c ( ( , ))( ))( F t F t c ( ( , ) ) F t F t c ( ( , )) )) i i j i j ≠ j i VIC: the difference within the given category g g y 1 ∑ ← − 2 V IC ( , t c ) ( F ( , t s ) F ( , t c )) i i n ∈ s c i i Discriminability-score = Score t ( ) ( ) VAC t c ( , ( , ) / ) VIC t c ( , ( , ) ) i i i i � Where F(t,s) is the distribution function of concept ( , ) p t in shot s .

Example-1 Discriminability-similarity consistency INSTITUTE OF COMPUTING TECHNOLOGY � Topic248 Find shots of a crowd of people � Topic248 Find shots of a crowd of people , outdoors , outdoors filling more than half of the frame area DBCS approach DBCS approach Text selection approach Text selection approach infAP=0.321 infAP=0.203 Crowd Crowd Crowd Crowd 1 40 1.40 Outdoors People_Marching 0.92 Person Demonstration_Or_ 0.64 Protest Factor 1: outdoors and Factor-1: outdoors and Protesters Protesters 0.55 0 55 person also frequently Dark- 0.52 occur in other case. skinned People skinned_People Factor-3: collection characteristic

Example-2 Discriminability-similarity inconsistency Di i i bilit i il it i i t INSTITUTE OF COMPUTING TECHNOLOGY � Topic261 Find shots of one or more people at a � Topic261 Find shots of one or more people at a table or desk , with a computer visible Text selection approach T t l ti h DBCS approach infAP=0.012 infAP=0.116 Computer 0.55 Attached_Body_Parts Computer_Or_Television_Sc Classroom 0.30 reens reens Medical_Personnel 0.27 person Body Parts Body_Parts 0 25 0.25 Hand 0.23 Factor 2: computer d t detector is not reliable t i t li bl

Re-ranking INSTITUTE OF COMPUTING TECHNOLOGY � face and motion factors f f � shot-level average face size and position � shot-leve principal motion direction and intensity ′ = ′ + × Score S S Score F FactorScore t S F FactorCoefficient t C ffi i t � Shot-level vs. Keyframe-level Extract the stable factor � Re-ranking selection R d Reduce the negative effect th ti ff t

Dynamic fusion INSTITUTE OF COMPUTING TECHNOLOGY � Smoothed Similarity Cluster(SSC) S S C (SSC) A feature undergoes a rapid change in its normalized scores is � likely to perform better than a feature which undergoes a more likely to perform better than a feature which undergoes a more gradual transition. 1 1 1000 ( 1000 ( − + ∑ score n ( ) ( ) score n ( ( 1)) )) = n 1 [ P. Wilkins,2007] 1000 = SC 1 N − + ∑ ( score n ( ) score n ( 1)) = n 1 N N median SC ( ) = SSC SC is unstable in real noisy data. y standard deviation SC t d d d i ti ( ( SC ) ) = ∑ In our system, all fusion Run SSC Score Run Weight Run Weight processes are realized by SSC All SSC Scores method.

Outline INSTITUTE OF COMPUTING TECHNOLOGY � Overall System � Review of baseline retrieval Review of baseline retrieval � Performance analysis � Concept-based retrieval � Re-ranking � Dynamic fusion � conclusion � conclusion

Overview of submitted and unsubmitted runs b itt d INSTITUTE OF COMPUTING TECHNOLOGY Run Description Run Description Mean InfAP Mean InfAP Run 1: Text baseline 0.009 Run2 * : Visual baseline(Multi-bag SVM) Run2 : Visual baseline(Multi bag SVM) 0.024 0.024 Run3 * : Visual baseline(LDA) 0.028 Run4: SSC(Run2, Run3) 0.033 * is the Run 5: HLF baseline(svm, CU-VIREO374) 0.029 unsubmiited run 0.036 Run 6: HLF baseline +re-ranking Run 7 * : Concept retrieval(text map, CU-VIREO374) 0.026 Run 8 * : Concept retrieval(DBCS map, CU-VIREO374) 0.039 Run 9 * : SSC(Run7 + Run8) 0.043 Run 10: SSC(Run5 + Run9) 0.053 Run 11: SSC(Run4 + Run9) 0.067

Overall performance analysis INSTITUTE OF COMPUTING TECHNOLOGY 0.08 0.07 0.07 Fusion of visual and concept-based runs 0.06 Best concept-based run 0.05 0.04 Best visual-based run 0.03 0.02 Our text-based run Our text-based run 0.01 0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 A t Automatic search runs of TRECVID2008 ti h f TRECVID2008

Conclusion-1 INSTITUTE OF COMPUTING TECHNOLOGY � Concept-based retrieval is a promising direction. direction. � DBCS mapping method can achieve a DBCS i th d hi stable good performance. � The difference of the concept distribution is more useful than the distribution itself . � Select concepts independent of the detector performance is not reasonable.

Conclusion-2 INSTITUTE OF COMPUTING TECHNOLOGY � Face and motion based re-ranking Face and motion based re ranking technology is important for some special t topics. i � Shot-level feature is stable � Reducing the negative effect is important