Online Video SEEDS Dr. Michael Van den Bergh
Superpixels Extracted via SEEDS Energy- ECCV 2012 Driven Sampling
What are superpixels? • grouping pixels based on similarity (color) • speeds up segmentation • objects are made up of a small number of superpixels
Existing superpixel methods gradual addition of cuts • high accuracy • very slow (contradictory) • e.g. Entropy Rate Superpixels (Liu et al. )
Existing superpixel methods growing from centers • faster • reduced accuracy (local minima + stray labels) • still not fast enough • e.g. SLIC Superpixels (Achanta et al. )
new approach: SEEDS Superpixels initalization largest block update medium block update smallest block update pixel-level update • initialize with rectangular boundaries • gradually refine boundaries • SEEDS: Superpixels Extracted via Energy-driven Sampling - ECCV 2012
Advantages of SEEDS initalization largest block update medium block update smallest block update pixel-level update • faster than growing centers • only needs to evaluate at the boundaries • highly efficient evaluation using color histograms (1 memory lookup)
Advantages of SEEDS initalization largest block update medium block update smallest block update pixel-level update • faster than growing centers • only needs to evaluate at the boundaries • highly efficient evaluation using color histograms • accuracy matches or exceeds state-of-the-art • avoids local minima • optimization only evaluates valid partitionings
Advantages of SEEDS Undersegmentation Error Boundary Recall Achievable Segmentation Accuracy 3 1 0.98 SEEDS (15Hz) SLIC (5Hz) Entropy Rate (1Hz) Felzenszwalb and Huttenlocher 2.25 0.8 0.95 0.6 0.91 1.5 0.75 0.4 0.88 SEEDS (15Hz) SEEDS (15Hz) SLIC (5Hz) SLIC (5Hz) Entropy Rate (1Hz) Entropy Rate (1Hz) Felzenszwalb and Huttenlocher Felzenszwalb and Huttenlocher 0.2 0.84 0 50 100 200 400 50 100 200 400 50 100 200 400 number of superpixels number of superpixels number of superpixels
Advantages of SEEDS initalization largest block update medium block update smallest block update pixel-level update • faster than state-of-the-art • accuracy matches or exceeds state-of-the-art
Advantages of SEEDS initalization largest block update medium block update smallest block update pixel-level update • faster than state-of-the-art • accuracy matches or exceeds state-of-the-art • control over run-time • whenever the algorithm is stopped, a valid partitioning is available • state-of-the-art accuracy at 30 Hz (single core)
Advantages of SEEDS initalization largest block update medium block update smallest block update pixel-level update • faster than state-of-the-art • accuracy matches or exceeds state-of-the-art • control over run-time • whenever the algorithm is stopped, a valid partitioning is available • state-of-the-art accuracy at 30 Hz (single core) • control over superpixel shape • one or more priors can be applied during boundary updating
(b) SEEDS with 3 × 3 smoothing prior (b) SEEDS with compactness prior (b) SEEDS with edge prior (snap to edges) (b) SEEDS with combined prior (3 × 3 smoothing + compactness + snap to edges)
Advantages of SEEDS • faster • more accurate • control over run-time • control over shape • temporal
Advantages of SEEDS initalization largest block update medium block update smallest block update pixel-level update
Online Video SEEDS
Video SEEDS ������� frame 0 frame 1 frame 2 frame initialization block-updates �� propagation pixel-updates
Video SEEDS t = 0 initialization layer 3 ( blocks ) layer 2 ( blocks ) layer 1 ( pixels ) t = 1 initialization layer 2 ( blocks ) layer 1 ( pixels ) Figure 4. Efficient updating at different block sizes.
Video SEEDS Termination Creation , c A t t t | , |B t m , c A t n n t -1:0 m n m , c A t t :0 , c A t t :0 m m p m time time current current frame frame Figure 5. Termination and creation of superpixels. • superpixels per frame • superpixel rate (time)
Video SEEDS 90 0.9 0.95 GBH (t= ) StreamGBH (t=10) 0.85 0.9 80 StreamGBH (t=1) 3D Undersegmentation Error StreamGB (t=1) 0.8 0.85 70 Video SEEDS (t=1) Meanshift 3D Boundary Recall Explained Variation 0.75 0.8 60 0.7 0.75 50 0.65 0.7 40 0.6 0.65 30 0.55 GBH (t= ) 0.6 GBH (t= ) StreamGBH (t=10) StreamGBH (t=10) 20 StreamGBH (t=1) 0.5 0.55 StreamGBH (t=1) StreamGB (t=1) StreamGB (t=1) 10 Video SEEDS (t=1) 0.45 0.5 Video SEEDS (t=1) Meanshift Meanshift 0 0.4 0.45 200 300 400 500 600 700 800 900 200 300 400 500 600 700 800 900 200 300 400 500 600 700 800 900 Number of Supervoxels Number of Supervoxels Number of Supervoxels • Chen Xiph.org benchmark • t= ∞ means the entire video is analyzed • t=1 means it is online (not streaming) • we are at 30Hz, they are at 0.25 Hz
SEEDS in OpenCV
Randomized SEEDS
(b) SEEDS with 3 × 3 smoothing prior (b) SEEDS with compactness prior (b) SEEDS with edge prior (snap to edges) (b) SEEDS with combined prior (3 × 3 smoothing + compactness + snap to edges)
Randomness Injection labels multiple SEEDS samples
Randomized SEEDS Randomized SEEDS labels multiple SEEDS samples
Temporal Video Objectness Randomized SEEDS labels multiple SEEDS samples objectness score
Temporal Video Objectness
Tubes of Bounding Boxes
Tubes of Bounding Boxes
Temporal Video Objectness (SEEDS) Objectness on still images: s-o-a Video Objectness: temporal window performance Objectness on still images: baselines 1 1 0.9 gPb (auc: 0.473) Objectness [1] (auc: 0.490) Canny (auc: 0.408) 0.9 0.9 van de Sande [16] 0.81 Randomized SEEDS - 1 sample (auc: 0.428) Feng et al. [7] (auc: 0.475) Randomized SEEDS - 5 samples (auc: 0.475) Rahtu et al. [12] (auc: 0.3680) 0.8 0.8 0.72 Randomized SEEDS (auc: 0.475) 0.7 0.7 0.63 Detection Rate Detection Rate Detection Rate 0.6 0.6 0.54 0.5 0.5 0.45 0.4 0.4 0.36 0.3 0.3 0.27 0.2 0.2 0.18 3D edge - 5 samples (auc: 0.652) 3D edge - 1 sample (auc: 0.523) 0.1 0.1 only propagation - 5 samples (auc: 0.628) 0.09 only propagation - 1 sample (auc: 0.309) 0 0 0 0 1 2 3 0 1 2 3 0 1 2 3 4 10 10 10 10 10 10 10 10 10 10 10 10 10 # temporal windows (tubes) # windows # windows
Thank You.
Recommend
More recommend
