Interactive Foreground Segmentation in Images and Videos Suyog - PDF document

2/9/2017 Interactive Foreground Segmentation in Images and Videos Suyog Jain 1 Foreground Segmentation Generate pixel level foreground masks for objects in a given image or video 2 1

2/9/2017 Why is Foreground Segmentation useful? Better Visual Search Results from AlchemyAPI search Many irrelevant images appear in search result 3 Why is Foreground Segmentation useful? Better Visual Search Results from AlchemyAPI search Search can focus on the object of interest 4 2

2/9/2017 Why is Foreground Segmentation useful? Training object recognition systems Training Images Recognition System Can benefit from well segmented objects during training 5 Why is Foreground Segmentation useful? Computer Graphics Image & Video editing Need accurate foreground segmentations 6 3

2/9/2017 Why is Foreground Segmentation useful? Computer Graphics 3D Reconstruction Can benefit from foreground segmentations [Snavely, ICCV 209] 7 Human-Machine Collaboration Good at perception and can easily identify the foreground regions Good at processing large volumes of data at the lowest level of details very efficiently Complementary strengths 8 4

2/9/2017 Human-Machine Collaboration Bringing them together can lead to systems which can be accurate and cost effective 9 Interactive Foreground Segmentation Human Segmentation input algorithm [ Boykov 2001, Zabih 2001, Gulshan 2010, Kohli 2008] 10 5

2/9/2017 MRF-Segmentation Model Image Unary Term Pairwise Term p q Frequency Frequency + Foreground distribution Background distribution Unary Term Pairwise Term Low penalty High penalty 11 [ Boykov 2001, Zabih 2001, Gulshan 2010, Kohli 2008] MRF-Segmentation Model Image Unary Term Pairwise Term p q Combinatorial Optimization Foreground Segmentation Optimal labeling output Background 12 [ Boykov 2001, Zabih 2001, Gulshan 2010, Kohli 2008] 6

2/9/2017 MRF-Segmentation Model Foreground Can be solved efficiently using Max flow algorithms Also known as Graph Cuts Segmentation Background 13 [ Boykov 2001, Zabih 2001, Gulshan 2010, Kohli 2008] Demo 14 7

2/9/2017 Video Object Segmentation Generate pixel level foreground masks for an object(s) across the frames of a video Need only a couple of clicks! How long will it take to do this? 15 Interactive Video Segmentation Bring the complementary strengths of humans and machines together. Segmentation High Level Video Frame Guidance Segmentation Propagation 16 8

2/9/2017 Interactive Video Segmentation Get user input first and then generate a segmentation hypothesis User Input System Output Traditional Methods Bounding Box Scribbles Sloppy Contour [ Boykov 2001, Zabih 2001, Rother 2004, Kohli 2008] Interactive Video Segmentation Bounding Box Scribbles Sloppy Contour vs. Point Clicks? [ Boykov 2001, Zabih 2001, Rother 2004, Kohli 2008] 9

2/9/2017 Our Idea - Flip the process Pre-generate thousands of segmentations with no human input. 19 Our Idea - Flip the process Use boundary clicks to quickly “carve” out the accurate ones. 20 10

2/9/2017 Interactive Video Segmentation More accurate segmentation with less annotation cost. Traditional Methods Ours [ Boykov 2001, Zabih 2001, Rother 2004, Kohli 2008] Overview Region 1. Proposals 2. Click Carving Segmentation 3. Propagation 11

2/9/2017 Region Proposals Use perceptual grouping cues to generate thousands of object segmentations with no human input. Static Boundaries Motion Boundaries Hierarchical segmentation and region grouping ….. Multiscale Combinatorial Grouping [Arbelaez 2014 ] Overview Region 1. Proposals 2. Click Carving Segmentation 3. Propagation 12

2/9/2017 Click Carving 1 1 1 1 1 0 1 1 0 0 Votes 25 Click Carving 1 1 1 1 1 0 1 1 0 0 Votes 26 13

2/9/2017 Click Carving 1 2 1 1 2 0 2 2 0 0 Votes 27 Top Ranked Segmentations 14

2/9/2017 Top Ranked Segmentations Top Ranked Segmentations 15

2/9/2017 Top Ranked Segmentations Top Ranked Segmentations 16

2/9/2017 Top Ranked Segmentations Click Carving – User Interface 34 17

2/9/2017 Overview Region 1. Proposals 2. Click Carving Segmentation 3. Propagation Video Segmentation Propagation 36 [Jain ECCV 2014] 18

2/9/2017 Results 37 Experimental Setup • Evaluate on 3 challenging video segmentation datasets: – Segtrack-v2 [Li et al. 2013] – VSB 100 [Sundber et al. 2011] – iVideoSeg [Nagaraja et al. 2015] • User study: – 3 annotators with a max annotation budget of 10 clicks. – Record number of clicks, time spent and best object mask chosen by the annotator. – Compare with several existing methods which use different amount of human annotation. 38 19

2/9/2017 Click Carving Results Ours Propagate from video frame segmented Propagate from fully human segmented Faster but result in poor segmentation quality. Excellent cost vs. accuracy tradeoff video frame though “Click Carving” Click Carving Results Why use boundary clicks? Interior Clicks Boundary Clicks Boundary clicks are far more discriminative than interior clicks. 40 20

2/9/2017 Click Carving Results Using only 1-2 clicks 41 Additional Features • Negative Clicks 42 21

2/9/2017 22

2/9/2017 Additional Features • Negative Clicks • Biomedical Images 46 23

2/9/2017 Demo • Click Carving Demo – Demo1 – Demo2 • Pixel Objectness – http://vision.cs.utexas.edu/projects/pixelobj ectness/ • FusionSeg – video segmentation – http://vision.cs.utexas.edu/projects/fusionse g/ 48 24

2/9/2017 Questions More details at: https://www.cs.utexas.edu/~suyog/ 49 25