DVMM lab d igital video | multimedia laboratory Coping with Video Domain Change Analysis of Cross-Domain Learning Methods for High-Level Visual Concept Detection Eric Zavesky, Wei Jiang, Akira Yanagawa, Shih-Fu Chang TRECVID HLF 2007
Columbia HLF: TRECVID2006 TRECVID TRECVID TRECVID 2005 2005 2005 (development) (development) (development) Columbia 374 Columbia 374 Columbia 374 (baseline models) (baseline models) (baseline models) Concept fusion Concept fusion (explore concept relations) (explore concept relations) TRECVID BCRF BCRF 2006 (20 improved models) (20 improved models) (test) Coping with Video Domain Change; Zavesky 2007 2
Columbia HLF: TRECVID2007 TRECVID TRECVID TRECVID 2005 2005 2005 (development) (development) (development) Columbia 374 Columbia 374 Columbia 374 (baseline models) (baseline models) (baseline models) Concept fusion Concept fusion Concept fusion 1 1 1 (explore concept relations) (explore concept relations) (explore concept relations) BCRF BCRF BCRF (20 improved models) (20 improved models) (20 improved models) 2 2 TRECVID TRECVID TRECVID New baseline New baseline 2007 2007 2007 TRECVID TRECVID 2007 2007 (test) (test) (test) (development) (development) 3 Model Cross domain adaptation Coping with Video Domain Change; Zavesky 2007 3
cross-domain learning Coping with Video Domain Change; Zavesky 2007 4
Definition: • Domain: set of content with same production/capture method and content quality news (old domain) documentary (new domain) TRECVID 2005 TRECVID 2007 Problem: • Not all data sets are created equal; classifiers trained on one domain often do not work well on others Goal: • Achieve robust detection in new domain with minimal additional complexity Coping with Video Domain Change; Zavesky 2007 5
Cross-Domain Problem: What is it? Approach: • Leverage pre-trained existing models • Optimal weighted combination of data from both domains Data: • TRECVID2005 (broadcast news @ 100 hours), • TRECVID2007 (documentaries @ 60 hours) Coping with Video Domain Change; Zavesky 2007 6
Cross-Domain Problem: Common approaches applicable training data method old new condition increasing time & complexity old domain very use old model all - similar to new domain Coping with Video Domain Change; Zavesky 2007 7
Case 1: old model works best Studio top 5 detection results learn new domain, test new domain learn old domain, test old domain � learn old domain, test new domain 8 Coping with Video Domain Change; Zavesky 2007
Cross-Domain Problem: Common approaches applicable training data method old new condition increasing time & complexity old domain very use old model all - similar to new domain train new new and old domains - all domain model very dissimilar Coping with Video Domain Change; Zavesky 2007 9
Case 2: new model works best Waterscape top 5 detection results � learn new domain, test new domain learn old domain, test old domain learn old domain, test new domain 10 Coping with Video Domain Change; Zavesky 2007
Cross-Domain Problem: Common approaches applicable training data method old new condition increasing time & complexity old domain very use old model all - similar to new domain train new new and old domains - all model very dissimilar adapt old new and old domains small all model slightly dissimilar Coping with Video Domain Change; Zavesky 2007 11
Case 3: old model adaptation works best Chart s top 5 detection results learn new domain, test new domain learn old domain, test old domain � adapt old domain+new domain, test new domain 12 Coping with Video Domain Change; Zavesky 2007
Cross-Domain Problem: Common approaches applicable training data method old new condition increasing time & complexity old domain very use old model all - similar to new domain train new new and old domains - all model very dissimilar adapt old new and old domains small all model slightly dissimilar old and new domains train combined all all similar; sparse new domain new+old model or strong old model Coping with Video Domain Change; Zavesky 2007 13
Case 4: combined model works best Sports top 5 detection results learn new domain, test new domain learn old domain, test new domain � learn old domain+new domain, test new domain 14 Coping with Video Domain Change; Zavesky 2007
Cross-Domain Problem: Common approaches applicable training data method old new condition increasing time & complexity old domain very use old model all - similar to new domain train new new domain and old - all model domains very dissimilar adapt old adapt old new and old domains new and old domains small all small all model slightly dissimilar model slightly dissimilar old and new domains old and new domains train combined train combined all all all all similar; sparse new domain similar; sparse new domain new+old model new+old model or strong old model or strong old model Coping with Video Domain Change; Zavesky 2007 15
Topic Review: Support Vector Machine (SVM) New Feature Space New Feature Space New Feature Space Old Feature Space Old Feature Space Old Feature Space + + + + + + + + decision boundary + + + + + + x x x x x ignored + + + + + + positive new ( hyperplane) NOTE: + + + new samples new samples x x + + + + + + support vector + + + positive positive positive samples + + + new domain + + from old domain margin distance + + + old samples old samples old samples old old x x x x + + + + + + can be sparse x x x 1 support support + + + x x x x x classification errors || w || x x x vectors vectors x x x x x x x x negative negative x negative new + + + (using old hyperplane) helpful x x x x x old samples old samples old support vectors samples adapted support vector x x x + + + (evidence for decision x x + + + x x x hyperplane x x x x x + + from old domain + + + boundary) x x x old new samples samples Coping with Video Domain Change; Zavesky 2007 16
Combined model: Uniform sample importance • Idea: includes all data (new and old) in training of new domain models • Kernel matrix: equal weights for all samples features old old new samples learn new new samples old models 1x new Coping with Video Domain Change; Zavesky 2007 17
Replication model: Kernel matrix replication • Idea: augment feature vector to learn intra-domain weights across many dimensions • Cross-domain training may be quite dissimilar • Trust intra-domain similarity more • Intelligent method for feature expansion features replication old old old new new processor samples new old old 2x 1x samples 1x learn new new 1x 2x new models H. Daume III, “Frustratingly easy domain adaptation”, Proc. the 45th Annual Meeting of the Association of Computational Linguistics, 2007 Coping with Video Domain Change; Zavesky 2007 18
Replication model: Kernel matrix replication • Idea: augment feature vector to learn intra-domain weights across many dimensions D features +1 0.9 0.9 +1 0.2 0.4 M samples N samples ... ... old new D*3 features +1 0.4 0.2 0.4 0.2 0.0 0.0 M + N samples old new ... general features features +1 0.9 0.9 0.0 0.0 0.9 0.9 feats ... H. Daume III, “Frustratingly easy domain adaptation”, Proc. the 45th Annual Meeting of the Association of Computational Linguistics, 2007 Coping with Video Domain Change; Zavesky 2007 19
Adaptive SVM (A-SVM): Constrained model adaptation • Idea: trust old domain model more than new domain • Perturb old model within some tolerance with weighted new samples and a constant offset f ( x ) = f old ( x ) + ∆ f ( x ) old extract old support optimize choice models learn support vectors of new samples new vectors and old support models new vectors samples J. Yang, et al., “Cross-domain video concept detection using adaptive svms”, ACM Multimedia, 2007. Coping with Video Domain Change; Zavesky 2007 20
Adaptive SVM (A-SVM): Constrained model adaptation Old Feature Space + decision boundary x ( hyperplane) New Feature Space + margin distance + 1 + constrained x || w || + adjustment + x old support vectors + + (evidence for decision + + boundary) x margin distance + x 1 x || w || x New Feature Space x + x + + x + x + + x + x x new samples adapted x old hyperplane x x support x vectors J. Yang, et al., “Cross-domain video concept detection using adaptive svms”, ACM Multimedia, 2007. Coping with Video Domain Change; Zavesky 2007 21
Cross-domain SVM (CD-SVM): Adapting prior models • Idea: trust support vectors from trained old domain model as best observations in old domain • Weigh SVs then combine with new data and retrain old extract old compute weighted support samples models similarity features learn support vectors new vectors models samples new samples Submitted: W. Jiang, E. Zavesky, S.F. Chang, A. Loui, “Cross-domain learning methods for high-level concept classification,” ICASSP 2008. Coping with Video Domain Change; Zavesky 2007 22
Recommend
More recommend
