constrained video face clustering using 1nn relations
play

Constrained Video Face Clustering using 1NN Relations Vicky - PowerPoint PPT Presentation

Dec 26, 2023 •766 likes •1.05k views

Constrained Video Face Clustering using 1NN Relations Vicky Kalogeiton Andrew Zisserman Video face clustering In Input Ou Outp tput Video source: [Tapaswi ICVGIP 2014] 1 Why does it matter? Comfort Fun Access Modern applications

  1. Constrained Video Face Clustering using 1NN Relations Vicky Kalogeiton Andrew Zisserman

  2. Video face clustering In Input Ou Outp tput Video source: [Tapaswi ICVGIP 2014] 1

  3. Why does it matter? Comfort Fun Access

  4. Modern applications Automatic story telling Grand entry of the king’s horses Grand entry of the king’s horses and men. AR ARYA , wearing a helm and men. ARYA, wearing a helm and cloak, pushes her way into a and cloak, pushes her way into a tall wagon for a better look…. tall wagon for a better look…. In rides JO , followed by the JOFFREY, HOU HOUND 3

  5. Overview C1C: Constrained 1NN Clustering Cannot-lin Ca link t must-link must-link cannot- Must-lin Mu link cannot-link link 1 st min-cut 1 st neighbor neighbor clust min-cut er cluster 4

  6. Overview C1C: Constrained 1NN Clustering Cannot-lin Ca link t must-link must-link cannot- Must-lin Mu link cannot-link link 1 st min-cut 1 st neighbor neighbor clust min-cut er cluster Friends dataset Contributions • season 3 (10h) ü No training required ü Scalable ~25 episodes • ü Low computational cost 17k head tracks • ü Outperforms state of the art 49 characters • ü Friends: challenging 5

  7. Outline • FINCH clustering method • Self-supervised Constraints • C1C pipeline • Friends dataset • Experimental Results 6

  8. Related work [Everingham BMVC 2006] [Cinbis ICCV 2011] [Bojanowski ICCV 2013] [Wu ICCV 2013,CVPR 2013] S-Siam [Sharma T-BIOM 2020] BCL [Tapaswi ICCV 2019] CCL [Sharma FG 2020] T-Siam [Sharma FG 2019] - Use HAC - Require learning - High computational cost 7

  9. C1C: Constrained 1NN Clustering Hierarchical Self-supervised Clustering method constraints [Code] http://www.robots.ox.ac.uk/~vgg/research/c1c/ 8

  10. C1C – FINCH Hierarchical Self-supervised Clustering method constraints • Pairwise distances between all instances FINCH • At every partition, link all first NN • Merge instances that are first neighbors or have a common first neighbor • Represent a cluster with the average of its [S. Sarfraz CVPR 19] instances [Code] http://www.robots.ox.ac.uk/~vgg/research/c1c/ 9

  11. C1C— Self-supervised Constraints Hierarchical Self-supervised Clustering method constraints Must st-Li Link nk Ca Cannot-lin link t [Code] http://www.robots.ox.ac.uk/~vgg/research/c1c/ 10

  12. C1C: Constrained 1NN Clustering 11

  13. C1C: Constrained 1NN Clustering must-link cannot-link 12

  14. C1C: Constrained 1NN Clustering (a) must-link cannot-link (b) 13

  15. C1C: Constrained 1NN Clustering (a) must-link cannot-link (b) 14

  16. C1C: Constrained 1NN Clustering (a) must-link cannot-link 1 st neighbor (b) 15

  17. C1C: Constrained 1NN Clustering (a) must-link (c) cannot-link 1 st neighbor (b) 16

  18. C1C: Constrained 1NN Clustering (a) must-link cannot-link 1 st neighbor (b) 17

  19. C1C: Constrained 1NN Clustering (a) must-link cannot-link 1 st neighbor min-cut (b) 18

  20. C1C: Constrained 1NN Clustering must-link cannot-link 1 st neighbor min-cut cluster

  21. Friends dataset • Friends season 3 (~10h, 25 episodes) • 17k head tracks, 49 characters (six main, 43 secondary) Rachel Monica Ross Joey Phoebe Main Ma Carol Kate Rachel date Gunther Janice Secondary Se 20

  22. Datasets & Metrics Buffy the Vampire Slayer The Big Bang Theory (BBT) Implementation WCP: Weighted Clustering Purity Tr Trade-of off • Architecture: |,| WCP = 1 ResNet-50 & ' - ( . ( Purity • Pre-trained ()* MS-Celeb-1M . ( :purity of cluster / • Fine-tuned: - ( : #samples in cluster / VGGFace2 # clusters 21

  23. Quantitative Results %W %WCP CP 90.8% BBT BBT 92.9% 95.3% 82.9% ü C1 C1C : better Buffy Bu 86.5% FINCH and BCL 88.1% 69.7% ü Fri Friends : Fr Friends challenging 77.0% FINCH [Sarfraz CVPR 19] BCL [Tapaswi ICCV 19] C1C [Ours] 22

  24. Quantitative Results %W %WCP CP 90.8% BBT BBT 92.9% 95.3% 82.9% ü C1 C1C : better Bu Buffy 86.5% FINCH and BCL 88.1% 69.7% ü Fri Friends : Friends Fr challenging 77.0% FINCH [Sarfraz CVPR 19] BCL [Tapaswi ICCV 19] C1C [Ours] 23

  25. Qualitative results 24

  26. Conclusions & Future work C1C: • – links instances through 1NN relations – must-link and cannot-link constraints Advantages: • – scalable – no training required – low computational cost Friends dataset • State of the art results • Overcome failures by using more context • Automatically estimate #characters • 25

  27. Thank you

Recommend

Week 7 Video 1 Clustering Clustering A type of Structure Discovery algorithm This type of

Week 7 Video 1 Clustering Clustering A type of Structure Discovery algorithm This type of

Week 7 Video 1 Clustering Clustering A type of Structure Discovery algorithm This type of method is also referred to as Dimensionality Reduction , based on a common application Clustering You have a large number of data points You

974 views • 63 slides
Week 7 Video 3 Advanced Clustering Algorithms Today Multiple advanced algorithms for

Week 7 Video 3 Advanced Clustering Algorithms Today Multiple advanced algorithms for

Week 7 Video 3 Advanced Clustering Algorithms Today Multiple advanced algorithms for clustering Gaussian Mixture Models Often called EM-based clustering Kind of a misnomer in my opinion What distinguishes this algorithm

681 views • 21 slides
Automatic Face Recognition in Weakly Constrained Environments Fabien Cardinaux cardinau@idiap.ch

Automatic Face Recognition in Weakly Constrained Environments Fabien Cardinaux cardinau@idiap.ch

Automatic Face Recognition in Weakly Constrained Environments Fabien Cardinaux cardinau@idiap.ch Automatic Face Recognition in Weakly Constrained Environments p.1/33 Outline Problem of Face Recognition in Weakly

372 views • 33 slides
Structuring Typical Evolutions using Temporal-Driven Constrained Clustering 8 November 2012

Structuring Typical Evolutions using Temporal-Driven Constrained Clustering 8 November 2012

Structuring Typical Evolutions using Temporal-Driven Constrained Clustering 8 November 2012 Marian-Andrei Rizoiu ERIC Laboratory Julien Velcin Universit Lumire Lyon 2 Stphane Lallich France M-A. Rizoiu, J. Velcin and S. Lallich

634 views • 45 slides
Shape Co-analysis and constrained clustering Daniel Cohen-Or Tel-Aviv University 1 High-level

Shape Co-analysis and constrained clustering Daniel Cohen-Or Tel-Aviv University 1 High-level

Shape Co-analysis and constrained clustering Daniel Cohen-Or Tel-Aviv University 1 High-level Shape analysis [Mehra et al. 08] [Fu et al. 08] [Kalograkis et al. 10] Shape abstraction Upright orientation Learning segmentation [Kim et al.

1.03k views • 81 slides
RECSM Summer School: Machine Learning for Social Sciences Session 3.4: Hierarchical Clustering

RECSM Summer School: Machine Learning for Social Sciences Session 3.4: Hierarchical Clustering

RECSM Summer School: Machine Learning for Social Sciences Session 3.4: Hierarchical Clustering Reto West Department of Political Science and International Relations University of Geneva 1 Clustering Clustering Hierarchical Clustering

237 views • 22 slides
RECSM Summer School: Machine Learning for Social Sciences Session 3.3: K -Means Clustering Reto

RECSM Summer School: Machine Learning for Social Sciences Session 3.3: K -Means Clustering Reto

RECSM Summer School: Machine Learning for Social Sciences Session 3.3: K -Means Clustering Reto West Department of Political Science and International Relations University of Geneva 1 Clustering Clustering Clustering refers to a set of

352 views • 17 slides
Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering?

Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering?

Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering? Aykut Erdem May 2016 Hacettepe University Last time K-Means An iterative clustering algorithm - Initialize: Pick K random points as cluster

932 views • 64 slides
Clustering Hierarchical clustering, k-mean clustering Genome 559: Introduction to Statistical and

Clustering Hierarchical clustering, k-mean clustering Genome 559: Introduction to Statistical and

Clustering Hierarchical clustering, k-mean clustering Genome 559: Introduction to Statistical and Computational Genomics Elhanan Borenstein A quick review The clustering problem: partition genes into distinct sets with high homogeneity

772 views • 40 slides
Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns

Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns

Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns in data, or grouping similar groups of data-points together into clusters . Clustering algorithms for numeric data: Lloyds K-means, EM

688 views • 19 slides
Clustering Hierarchical clustering and k-mean clustering Genome 373 Genomic Informatics

Clustering Hierarchical clustering and k-mean clustering Genome 373 Genomic Informatics

Clustering Hierarchical clustering and k-mean clustering Genome 373 Genomic Informatics Elhanan Borenstein A quick review The clustering problem: partition genes into distinct sets with high homogeneity and high separation Many

768 views • 43 slides
CS 403X Mobile and Ubiquitous Computing Lecture 9: Face Detection, Widget Catalog, SQLite

CS 403X Mobile and Ubiquitous Computing Lecture 9: Face Detection, Widget Catalog, SQLite

CS 403X Mobile and Ubiquitous Computing Lecture 9: Face Detection, Widget Catalog, SQLite Databases Emmanuel Agu Face Detection Mobile Vision API https://developers.google.com/vision/ Face Detection: Locate face in photos and video and Facial

607 views • 33 slides
Clustering DSE 210 Clustering in R d Two common uses of clustering: Vector quantization Find

Clustering DSE 210 Clustering in R d Two common uses of clustering: Vector quantization Find

Clustering DSE 210 Clustering in R d Two common uses of clustering: Vector quantization Find a finite set of representatives that provides good coverage of a complex, possibly infinite, high-dimensional space. Finding meaningful

396 views • 36 slides
Face detection, local features face alignment, and Face detection

Face detection, local features face alignment, and Face detection

12/6/2013 Lecture overview Brief introduction to Face detection, local features face alignment, and Face detection http://www.ima.umn.edu/2008-2009/MM8.5-14.09/activities/Wohlberg-Brendt/sift.png face image parsing

238 views • 11 slides
Clustering: Hierarchical Clustering and K- Means Clustering Machine

Clustering: Hierarchical Clustering and K- Means Clustering Machine

Clustering: Hierarchical Clustering and K- Means Clustering Machine Learning 10-601B Seyoung Kim Many of these slides are derived from William Cohen,

660 views • 46 slides
Clustering and Dimensionality Reduction Preview Clustering K -means clustering

Clustering and Dimensionality Reduction Preview Clustering K -means clustering

Clustering and Dimensionality Reduction Preview Clustering K -means clustering Mixture models Hierarchical clustering Dimensionality reduction Principal component analysis Multidimensional scaling Isomap

1.61k views • 17 slides
Face Cover Face Coverings In School Guidelines Face Coverings Face Coverings and PPE Cloth

Face Cover Face Coverings In School Guidelines Face Coverings Face Coverings and PPE Cloth

Face Cover Face Coverings In School Guidelines Face Coverings Face Coverings and PPE Cloth face coverings protect others if the wearer is infected with SARS CoV-2 and is not aware. Cloth masks may offer some level of protection for the

197 views • 5 slides
Clustering A Categorization of Major Clustering Methods Partitioning Methods

Clustering A Categorization of Major Clustering Methods Partitioning Methods

Clustering What is Clustering? Types of Data in Cluster Analysis Clustering A Categorization of Major Clustering Methods Partitioning Methods Hierarchical Methods 1 2 What is Clustering? What is Clustering? Typical

647 views • 18 slides
Online Open World Face Recognition From Video Streams ID:23202 Fed ederico Pern ernici,

Online Open World Face Recognition From Video Streams ID:23202 Fed ederico Pern ernici,

IARPA JANUS Online Open World Face Recognition From Video Streams ID:23202 Fed ederico Pern ernici, Federico Bartoli, Matteo Bruni and Alberto Del Bimbo MICC - University of Florence - Italy http://www.micc.unifi.it The effectiveness of

383 views • 26 slides
Subspace Clustering Ensemble Clustering Subspace Clustering, Ensemble Clustering, Alternative

Subspace Clustering Ensemble Clustering Subspace Clustering, Ensemble Clustering, Alternative

LUDWIG- MAXIMILIANS- INSTITUTE DATABASE UNIVERSITT FOR SYSTEMS MNCHEN MNCHEN INFORMATICS INFORMATICS GROUP GROUP Subspace Clustering Ensemble Clustering Subspace Clustering, Ensemble Clustering, Alternative Clustering, Multiview

513 views • 9 slides
Alpha Presentation Cloud Based Video Face Tracking The Capstone Experience Team TechSmith Alex

Alpha Presentation Cloud Based Video Face Tracking The Capstone Experience Team TechSmith Alex

Alpha Presentation Cloud Based Video Face Tracking The Capstone Experience Team TechSmith Alex Cramer Kayla Grotsky Eric Newman Alyssa Werner Ryan Zahm Department of Computer Science and Engineering Michigan State University Spring 2016

241 views • 7 slides
Finding Clusters Types of Clustering Approaches: Linkage Based, e.g. Hierarchical Clustering

Finding Clusters Types of Clustering Approaches: Linkage Based, e.g. Hierarchical Clustering

Finding Clusters Types of Clustering Approaches: Linkage Based, e.g. Hierarchical Clustering Clustering by Partitioning, e.g. k-Means Density Based Clustering, e.g. DBScan Grid Based Clustering Compendium slides for Guide to Intelligent

1.32k views • 82 slides
OTT Video Opportunity, threat or something else? 1 9/21/2015 Rick Paulsen Client Relations

OTT Video Opportunity, threat or something else? 1 9/21/2015 Rick Paulsen Client Relations

9/21/2015 OTT Video Opportunity, threat or something else? 1 9/21/2015 Rick Paulsen Client Relations Manager 2 9/21/2015 A little background info MACC is a BSS/OSS provider based in Blair, Nebraska with a nation wide client base

706 views • 42 slides
Introduction to Machine Learning, Clustering and EM Barnab s P czos Contents Clustering

Introduction to Machine Learning, Clustering and EM Barnab s P czos Contents Clustering

Introduction to Machine Learning, Clustering and EM Barnab s P czos Contents Clustering K-means Mixture of Gaussians Expectation Maximization Variational Methods 2 Clustering 3 K- means clustering What is clustering?

1.32k views • 66 slides

More recommend