ucf mit

New directions in saliency research:
Developments in architectures, datasets, and evaluation

ECCV 2016 (Oct. 8, 2016: 9 AM - 12 PM)

Oudemanhuispoort 4-6, 1012 CN Amsterdam




Description of Tutorial

The application of deep neural architectures (CNNs, RNNs) for saliency modeling has driven progress in the last few years. Out of 66 saliency models evaluated to-date, the 8 top-performing models on the MIT Saliency Benchmark are all neural networks. As progress continues to be made, new problems surface:
  • Which are the successful model architectures?
  • Which datasets and loss functions can be used for training models?
  • Which metrics are most informative?
  • What are the remaining failure modes of models?
    • As model predictions begin to approach human ground-truth, many applications of saliency become feasible. In the service of computer vision, recent saliency models have been found to be useful for applications including: semantic segmentation, object detection, object proposals, image clustering and retrieval, and cognitive saliency applications such as image captioning and high-level image understanding. With new models and applications, there is an increasing need for large datasets, opening up opportunities for new data-collection methodologies, including crowdsourcing attention using web cameras and other user behaviors such as mouse movements and clicks. New architectures and datasets are making real-time, accurate, and specialized applications possible.
    Can individuals or groups (e.g., healthy vs. patient) be predicted based on what and where they look? Can we decode tasks and intentions from eye movements? Can predictions provide real-time design and user feedback?



    Topics

  • new neural network model architectures
  • new methodologies for data collection
  • new saliency benchmarks and datasets
  • new metrics and evaluation procedures
  • new problems in saliency modeling
  • applications to image recognition
  • topics in cognitive psychology and human perception


    • Organizers:

    Ali Borji
    Center for Research in Computer Vision,
    University of Central Florida
    Zoya Bylinskii
    Computer Science and Artificial Intelligence Laboratory,
    Massachusetts Institute of Technology
    Tilke Judd
    Product Manager, Google

    (Members of the MIT Saliency Benchmark)



    Tutorial Schedule

    9:00 - 9:15 Intro & Overview: New directions in saliency
    Tutorial organizers    		 [slides]
    9:15 - 9:45 Deep networks for saliency map prediction
    Naila Murray, Xerox Research Centre Europe    		 [slides]
    9:45 - 10:15 Evaluating saliency models in a probabilistic framework
    Matthias Kümmerer, University of Tuebingen    		 [slides]
    10:15 - 10:45 Saliency for image understanding and manipulation
    Ming-Ming Cheng, Nankai University    		 [slides]
    10:45 - 11:00 Coffee break
    11:00 - 11:30 Towards cognitive saliency
    Zoya Bylinskii, Massachusetts Institute of Technology    		 [slides]   [slides with transcript]   [ppt of slides]
    11:30 - 12:00 Research Panel [sample questions]

    Relevant References

    New Evaluation Methods

  • M Kümmerer, T Wallis, M Bethge Information-theoretic model comparison unifies saliency metrics PNAS, 112(52), 16054-16059, 2015
  • M Kümmerer, T Wallis, and M Bethge How close are we to understanding image-based saliency? arXiv preprint arXiv:1409.7686, 2014
  • Z Bylinskii, T Judd, A Oliva, A Torralba, F Durand What do different evaluation metrics tell us about saliency models? arXiv preprint arXiv:1604.03605, 2016
  • S Rahman, N Bruce Visual Saliency Prediction and Evaluation across Different Perceptual Tasks PloS one 10 (9), e0138053, 2015

    • New Data Collection Methodologies

  • M Jiang, S Huang, J Duan, Q Zhao SALICON: Saliency in context CVPR, 2015
  • NW Kim, Z Bylinskii, MA Borkin, A Oliva, KZ Gajos, H Pfister A Crowdsourced Alternative to Eye-tracking for Visualization Understanding CHI EA, 2015
  • K Krafka, A Khosla, P Kellnhofer, H Kannan, S Bhandarkar, W Matusik, A Torralba Eye Tracking for Everyone CVPR, 2016
  • A Papoutsaki, P Sangkloy, J Laskey, N Daskalova, J Huang, J Hays WebGazer: Scalable Webcam Eye Tracking Using User Interactions IJCAI, 2016
  • P Xu, KA Ehinger, Y Zhang, A Finkelstein, SR Kulkarni, J Xiao TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking arXiv preprint arXiv:1504.06755, 2015

    • New (Deep) Model Architectures

  • S Jetley, N Murray, E Vig End-to-End Saliency Mapping via Probability Distribution Prediction CVPR, 2016
  • M Kümmerer, L Theis, M Bethge Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet ICLR Workshop, 2015
  • M Cornia, L Baraldi, G Serra, R Cucchiara A Deep Multi-Level Network for Saliency Prediction ICPR, 2016
  • X Huang, C Shen, X Boix, Q Zhao Salicon: Reducing the semantic gap in saliency prediction by adapting deep neural networks ICCV, 2015
  • SS Kruthiventi, K Ayush, R Venkatesh Babu DeepFix: A Fully Convolutional Neural Network for predicting Human Eye Fixations arXiv preprint arXiv:1510.02927, 2015
  • E Vig, M Dorr, D Cox Large-Scale Optimization of Hierarchical Features for Saliency Prediction in Natural Images CVPR, 2014

    • New Applications

  • A Borji, MM Cheng, H Jiang, J Li Salient Object Detection: A Benchmark IEEE Transactions on Image Processing (TIP), 2015
  • W Qi, MM Cheng, A Borji, H Lu, LF Bai SaliencyRank: Two-stage manifold ranking for salient object detection Computational Visual Media 1 (4), 309-320, 2015
  • MM Cheng, NJ Mitra, X Huang, SM Hu SalientShape: Group saliency in image collections The Visual Computer 30 (4), 443-453, 2014
  • Y Wei, X Liang, Y Chen, X Shen, MM Cheng, Y Zhao, S Yan STC: A Simple to Complex Framework for Weakly-supervised Semantic Segmentation arXiv preprint arXiv:1509.03150
  • C Shen, X Huang, Q Zhao Predicting Eye Fixations on Webpage With an Ensemble of Early Features and High-Level Representations from Deep Network IEEE Transactions on Multimedia 17 (11), 2084-2093, 2015
  • A Recasens, A Khosla, C Vondrick, A Torralba Where are they looking? NIPS, 2015
  • S Wang, M Jiang, XM Duchesne, EA Laugeson, DP Kennedy, R Adolphs, Q Zhao Atypical visual saliency in autism spectrum disorder quantified through model-based eye tracking Neuron 88 (3), 604-616, 2015
  • M Jiang, X Boix, G Roig, J Xu, L Van Gool, Q Zhao Learning to Predict Sequences of Human Visual Fixations IEEE TNNLS, 27 (6), 1241-1252, 2016
  • A Borji, L Itti Defending Yarbus: Eye movements reveal observers' task Journal of vision 14 (3), 1-22, 2014
  • JFG Boisvert, NDB Bruce Predicting task from eye movements: On the importance of spatial distribution, dynamics, and image features Neurocomputing, 2016
  • Z Bylinskii, P Isola, A Torralba, A Oliva How you look at a picture determines if you will remember it SUNw Scene Understanding Workshop, CVPR, 2015

    • Where should saliency models look next?

  • Z Bylinskii, A Recasens, A Borji, A Oliva, A Torralba, F Durand Where Should Saliency Models Look Next? ECCV, 2016
  • NDB Bruce, C Wloka, N Frosst, S Rahman, JK Tsotsos On computational modeling of visual saliency: Examining what’s right, and what’s left Vision Research 116, 95-112, 2015
  • NDB Bruce, C Catton, S Janjic A Deeper Look at Saliency: Feature Contrast, Semantics, and Beyond CVPR, 2016
  • D Parks, A Borji, L Itti Augmented saliency model using automatic 3D head pose detection and learned gaze following in natural scenes Vision research 116, 113-126, 2015
  • M Jiang, J Xu, Q Zhao Saliency in crowd ECCV, 2014
  • M Feng, A Borji, H Lu Fixation prediction with a combined model of bottom-up saliency and vanishing point IEEE WACV, 2016