Saliency Detection Feiyang Chen July 27, 2019 Background - PowerPoint PPT Presentation

Saliency Detection Feiyang Chen July 27, 2019

Background • Humans are able to detect visually distinctive, so-called salient , scene regions effortlessly and rapidly in a pre- attentive stage • It helps to find the objects or regions that efficiently represent a scene, a useful step in complex vision problems such as scene understanding • Some topics that are closely or remotely related to visual saliency include: salient object detection, fixation prediction, image quality assessment, scene attributes…

Definition A process of two stages: • 1) detecting the most salient object • 2) segmenting the accurate region of that object . Three criteria: • 1) good detection: the probability of missing real salient regions and falsely marking the background as a salient region should be low, • 2) high resolution: saliency maps should have a high or full resolution to accurately locate salient objects and retain original image information, • 3) computational efficiency: as front-ends to other complex processes, these models should detect salient regions quickly.

History The First Wave: • One of the earliest saliency models, proposed by Itti et al. in 1998, generated the first wave of interest across multiple disciplines including cognitive psychology, neuroscience, and computer vision. • This model is an implementation of earlier general computational frameworks and psychological theories of bottom-up attention based on center-surround mechanisms.

History The Second Wave: • The second wave of interest surged with several works who defined saliency detection as a binary segmentation problem. • These authors were inspired by some earlier models striving to detect salient regions or proto-objects.

History The Third Wave: • A third wave of interest has appeared recently with the surge in popularity of CNNs, and in particular with the FCNs. • Unlike the majority of classic methods based on contrast cues, CNN-based methods both eliminate the need for hand-crafted features, and alleviate the dependency on center bias knowledge, and hence have been adopted by many researchers. • Neurons with large receptive fields provide global information that can help better identify the most salient region in an image, while neurons with small receptive fields provide local information that can be leveraged to refine saliency maps produced by the higher layers. This allows highlighting salient regions and refining their boundaries.

Methods • Supervised or Unsupervised • Traditional algorithms or Deep Learning-based methods

Classic Method: ITTI • Itti L, Koch C, Niebur E. A model of saliency-based visual attention for rapid scene analysis[J].

Classic Method: SR • Hou X, Zhang L. Saliency detection: A spectral residual approach[C]//2007 IEEE Conference on Computer Vision and Pattern Recognition. Ieee, 2007: 1-8.

Classic Method: LC Zhai Y, Shah M. Visual attention detection in video sequences using spatiotemporal cues[C]

DL-based Methods Deep Unsupervised Saliency Detection: A Multiple Noisy Labeling Perspective

DL-based Methods An Unsupervised Game-Theoretic Approach to Saliency Detection

DL-based Methods SalGAN: visual saliency prediction with adversarial networks

Experiments

Future Work •Co-Saliency •Improved SalGAN •Unsupervised Pre-Train •Combine other methods

Saliency Survey Traditional Methods 1.[ITTI][1998][PAMI] A Model of Saliency-Based Visual Attention for Rapid Scene Analysis, 2.[AIM][2005][NIPS] Saliency Based on Information Maximization , 3.[SR] [2007][CVPR] Saliency Detection A Spectral Residual Approach , 4.[GB][2007][NIPS] Graph-Based Visual Saliency , 5.[SUN][2008][JOV] SUN: A bayesian framework for saliency using natural statistics , 6.[FT][2009][CVPR] Frequency-tuned Salient Region Detection , 7.[CA][2010][CVPR] Context-Aware Saliency Detection , 8.[SEG][2010][ECCV] Segmenting Salient Objects from Images and Videos , 9.[MSSS][2010][ICIP] Saliency Detection using Maximum Symmetric Surround , 10.[HC,RC][2011][CVPR]Global Contrast based Salient Region Detection, 11.[CB][2011][BMVC] Automatic Salient Object Segmentation Based on Context and Shape Prior , 12.[SF][2012][CVPR] Saliency Filters Contrast Based Filtering for Salient Region Detection , 13.[LR][2012][CVPR] A Unified Approach to Salient Object Detection via Low Rank Matrix Recovery , 14.[BSF][2012][ICIP] Saliency Detection Based on Integration of Boundary and Soft-Segmentation , 15.[GC][2013][ICCV] Efficient Salient Region Detection with Soft Image Abstraction , 16.[MR][2013][CVPR] Saliency Detection via Graph-Based Manifold Ranking , 17.[MC][2013][ICCV] Saliency Detection via Absorbing Markov Chain , 18.[DRFI][2013][CVPR]Salient Object Detection A Discriminative Regional Feature Integration Region

Saliency Survey Traditional Methods 19.[DSR][2013][ICCV] Saliency Detection via Dense and Sparse Reconstruction , 20.[HS][2013][CVPR] Hierarchical Saliency Detection , 21.[PCA][2013][CVPR] What Makes a Patch Distinct , 22.[CRF][2013][CVPR] Saliency Aggregation A Data-driven Approach , 23.[UFO][2013][ICCV] Salient Region Detection by UFO Uniqueness, Focusness and Objectness , 24.[COV][2013][JOV] Visual saliency estimation by nonlinearly integrating features using region covariances , 25.[GR][2013][SPL] Graph-Regularized Saliency Detection With Convex-Hull-Based Center Prior , 26.[LSSC][2013][TIP] Bayesian Saliency via Low and Mid Level Cues , 27.[HDCT][2014][CVPR] Salient Region Detection via High-Dimensional Color Transform , 28.[RBD][2014][CVPR] Saliency Optimization from Robust Background Detection , 29.[MSS][2014][SPL] Saliency Detection with Multi-Scale Superpixels , 30.[GP][2015][ICCV] Generic Promotion of Diffusion-Based Salient Object Detection , 31.[MBS][2015][ICCV] Minimum Barrier Salient Object Detection at 80 FPS , 32.[WSC][2015][CVPR] A Weighted Sparse Coding Framework for Saliency Detection , 33.[RRW][2015][CVPR] Robust saliency detection via regularized random walks ranking , 34.[TLLT][2015][CVPR] Saliency Propagation from Simple to Difficult , 35.[BL][2015][CVPR] Salient Object Detection via Bootstrap Learning

Saliency Survey Traditional Methods 36.[BSCA][2015][CVPR] Saliency Detection via Cellular Automata , 37.[GLC][2015][PR] Salient Object Detection via Global and Local Cues , 38.[LPS][2015][TIP] Inner and Inter Label Propagation Salient Object Detection in the Wild , 39.[MAP][2015][TIP] Saliency Region Detection based on Markov Absorption Probabilities , 40.[BFS][2015][NC] Saliency Detection via Background and Foreground Seed Selection , 41.[MST][2016][CVPR] Real-Time Salient Object Detection with a Minimum Spanning Tree , 42.[PM][2016][ECCV] Pattern Mining Saliency , 43.[DSP][2016][PR] Discriminative saliency propagation with sink points , 44.[WLRR][2017][SPL] Salient Object Detection via Weighted Low Rank Matrix Recovery , 45.[MIL][2017][TIP] Salient Object Detection via Multiple Instance Learning , 46.[SMD][2017][PAMI] Salient Object Detection via Structured Matrix Decomposition , 47.[MDC][2017][TIP]300-FPS Salient Object Detection via Minimum Directional Contrast, 48.[WMR][2018][NC] Saliency detection via affinity graph learning and weighted manifold ranking , 49.[RCRR][2018][TIP] Reversion correction and regularized random walk ranking for saliency detection , 50.[WFD][2018][PR] Water flow driven salient object detection at 180 fps

Saliency Survey Deep-Learning Methods 1. LEGS: Deep networks for saliency detection via local estimation and global search, Wang, L. et al, CVPR, 2015. 2. MC: Saliency detection by multi-context deep learning , Zhao, R., et al, CVPR, 2015. 3. MDF: Visual saliency based on multiscale deep features , Li, G., et al, CVPR, 2015. 4. DCL: Deep Contrast Learning for Salient Object Detection , Li, G.,et al, CVPR, 2016. 5. ELD: Deep Saliency with Encoded Low level Distance Map and High Level Features , Gayoung, L., et al, CVPR, 2016. 6. DHS: DHSNet: Deep Hierarchical Saliency Network for Salient Object Detection , Liu, N., et al, CVPR, 2016. 7. RFCN: Saliency detection with recurrent fully convolutional networks , Wang, L., et al, ECCV, 2016. 8. CRPSD: Saliency Detection via Combining Region-Level and Pixel-Level Predictions with CNNs , ECCV, 2016. 9. DISC: DISC: Deep image saliency computing via progressive representation learning , Chen, T., et al, TNNLS, 2016 10. DS: DeepSaliency: Multi-task deep neural network model for salient object detection , Li, X., et al, TIP, 2016. 11. IMC: Deep Salient Object Detection by Integrating Multi-level Cues , Zhang, J., et al, WACV 2017. 12. DSS: Deeply supervised salient object detection with short connections, Hou, Q., et al, CVPR, 2017/ TPAMI, 2018. 13. NLDF: Non-local deep features for salient object detection , Luo, Z., et al, CVPR, 2017. 14. AMU: Amulet: Aggregating multi-level convolutional features for salient object detection, Zhang, P., et al, ICCV, 2017. 15. UCF: Learning Uncertain Convolutional Features for Accurate Saliency Detection , Zhang, P., et al, ICCV, 2017.