What Do Different Evaluation Metrics Tell Us About Saliency Models?
Top Cited Papers
- 28 February 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in Ieee Transactions On Pattern Analysis and Machine Intelligence
- Vol. 41 (3), 740-757
- https://doi.org/10.1109/TPAMI.2018.2815601
Abstract
How best to evaluate a saliency model's ability to predict where humans look in images is an open research question. The choice of evaluation metric depends on how saliency is defined and how the ground truth is represented. Metrics differ in how they rank saliency models, and this results from how false positives and false negatives are treated, whether viewing biases are accounted for, whether spatial deviations are factored in, and how the saliency maps are pre-processed. In this paper, we provide an analysis of 8 different evaluation metrics and their properties. With the help of systematic experiments and visualizations of metric computations, we add interpretability to saliency scores and more transparency to the evaluation of saliency models. Building off the differences in metric properties and behaviors, we make recommendations for metric selections under specific assumptions and for specific applications.Keywords
Other Versions
Funding Information
- Natural Sciences and Engineering Research Council of Canada
- Toyota Research Institute/MIT CSAIL Joint Research Center
This publication has 77 references indexed in Scilit:
- Methods for comparing scanpaths and saliency maps: strengths and weaknessesBehavior Research Methods, 2012
- Measures and Limits of Models of Fixation SelectionPLOS ONE, 2011
- Fixations on low-resolution imagesJournal of Vision, 2011
- Learning a saliency map using fixated locations in natural scenesJournal of Vision, 2011
- Scene and screen center bias early eye movements in scene viewingVision Research, 2010
- Objects predict fixations better than early saliencyJournal of Vision, 2008
- Improved seam carving for video retargetingACM Transactions on Graphics, 2008
- The central fixation bias in scene viewing: Selecting an optimal viewing position independently of motor biases and image feature distributionsJournal of Vision, 2007
- Seam carving for content-aware image resizingACM Transactions on Graphics, 2007
- A survey of content-based image retrieval with high-level semanticsPattern Recognition, 2006