Modelling what users see when they look at images: a cognitive viewpoint

Abstract

Analysis of user viewing and query‐matching behavior furnishes additional evidence that the relevance of retrieved images for system users may arise from descriptions of objects and content‐based elements that are not evident or not even present in the image. This investigation looks at how users assign pre‐determined query terms to retrieved images, as well as looking at a post‐retrieval process of image engagement to user cognitive assessments of meaningful terms. Additionally, affective/emotion‐based query terms appear to be an important descriptive category for image retrieval. A system for capturing (eliciting) human interpretations derived from cognitive engagements with viewed images could further enhance the efficiency of image retrieval systems stemming from traditional indexing methods and technology‐based content extraction algorithms. An approach to such a system is posited.