Modelling what users see when they look at images: a cognitive viewpoint

Analysis of user viewing and query‐matching behavior furnishes additional evidence that the relevance of retrieved images for system users may arise from descriptions of objects and content‐based elements that are not evident or not even present in the image. This investigation looks at how users assign pre‐determined query terms to retrieved images, as well as looking at a post‐retrieval process of image engagement to user cognitive assessments of meaningful terms. Additionally, affective/emotion‐based query terms appear to be an important descriptive category for image retrieval. A system for capturing (eliciting) human interpretations derived from cognitive engagements with viewed images could further enhance the efficiency of image retrieval systems stemming from traditional indexing methods and technology‐based content extraction algorithms. An approach to such a system is posited.

[1]  Susanne Ornager.  Image Retrieval: Theoretical Analysis and Empirical User Studies on Accessing Information in Images. , 1997 .

[2]  Alberto Del Bimbo,et al.  Image retrieval by elastic matching of shapes and image patterns , 1996, Proceedings of the Third IEEE International Conference on Multimedia Computing and Systems.

[3]  Alex Pentland,et al.  Photobook: Content-based manipulation of image databases , 1996, International Journal of Computer Vision.

[4]  C.-C. Jay Kuo,et al.  Wavelet descriptor of planar curves: theory and applications , 1996, IEEE Trans. Image Process..

[5]  B. S. Manjunath,et al.  Texture Features for Browsing and Retrieval of Image Data , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  J. Frisby.  Seeing: Illusion, Brain and Mind , 1979 .

[7]  Elaine Svenonius.  Access to nonbook materials: the limits of subject indexing for visual and aural languages , 1994 .

[8]  Brian C. O'Connor,et al.  Browsing A Framework for Seeking Functional Information , 1993 .

[9]  Calvin C. Gotlieb,et al.  Texture descriptors based on co-occurrence matrices , 1990, Comput. Vis. Graph. Image Process..

[10]  Philip N. Klein,et al.  Indexing based on edit-distance matching of shape graphs , 1998, Other Conferences.

[11]  Sara Shatford,et al.  Analyzing the Subject of a Picture: A Theoretical Approach , 1986 .

[12]  June Abbas,et al.  User Reactions as Access Mechanism: An Exploration Based on Captions for Images , 1999, J. Am. Soc. Inf. Sci..

[13]  Sara Shatford Layne.  Some issues in the indexing of images , 1994 .

[14]  Serge J. Belongie,et al.  Region-based image querying , 1997, 1997 Proceedings IEEE Workshop on Content-Based Access of Image and Video Libraries.

[15]  Dragutin Petkovic,et al.  Content-Based Representation and Retrieval of Visual Media: A State-of-the-Art Review , 1996 .

[16]  Rodney A. Brooks,et al.  Symbolic Reasoning Among 3-D Models and 2-D Images , 1981, Artif. Intell..

[17]  I. Biederman.  Recognition-by-components: a theory of human image understanding. , 1987, Psychological review.

[18]  Thomas Craven.  Modern Art: The Men, the Movements, the Meaning , 1976 .

[19]  Anil K. Jain,et al.  Image retrieval using color and shape , 1996, Pattern Recognit..

[20]  François Mathey.  The world of the impressionists , 1961 .

[21]  Raya Fidel,et al.  The image retrieval task: implications for the design and evaluation of image databases , 1997, New Rev. Hypermedia Multim..

[22]  Z. Pylyshyn.  Is vision continuous with cognition? The case for cognitive impenetrability of visual perception. , 1999, The Behavioral and brain sciences.

[23]  E. Gombrich.  ART AND ILLUSION: A STUDY IN THE PSYCHOLOGY OF PICTORIAL REPRESENTATION. , 1960 .

[24]  Chin-Wan Chung,et al.  An Indexing and Retrieval Mechanism for Complex Similarity Queries in Image Databases , 1999, J. Vis. Commun. Image Represent..

[25]  Benjamin B. Kimia,et al.  Shock-based approach for indexing of image databases using shape , 1997, Other Conferences.

[26]  Markus A. Stricker,et al.  Color indexing with weak spatial constraints , 1996, Electronic Imaging.

[27]  Dragutin Petkovic,et al.  Efficient query by image content for very large image databases , 1993, Digest of Papers. Compcon Spring.

[28]  Rajiv Mehrotra,et al.  Similar-Shape Retrieval in Shape Data Management , 1995, Computer.

[29]  Hideyuki Tamura,et al.  Textural Features Corresponding to Visual Perception , 1978, IEEE Transactions on Systems, Man, and Cybernetics.

[30]  Corinne Jörgensen,et al.  Attributes of Images in Describing Tasks , 1998, Inf. Process. Manag..

[31]  Fang Liu,et al.  Periodicity, Directionality, and Randomness: Wold Features for Image Modeling and Retrieval , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  E. Panofsky.  Meaning in the Visual Arts , 1970 .

[33]  Howard Greisdorf,et al.  Relevance thresholds : A conjunctive/disjunctive model of end-user cognition as an evaluative process , 2000 .

[34]  Per Roland,et al.  The engine of reason, the seat of the soul , 1996 .

[35]  A. Damasio.  The Feeling of What Happens: Body and Emotion in the Making of Consciousness , 1999 .

[36]  Edward E. Smith,et al.  Categories and concepts , 1984 .

[37]  Stephen J. Brown.  The world of imagery , 1927 .

[38]  K. Holyoak,et al.  Induction of category distributions: a framework for classification learning. , 1984, Journal of experimental psychology. Learning, memory, and cognition.

[39]  Markus A. Stricker,et al.  Similarity of color images , 1995, Electronic Imaging.

[40]  Peter L. Stanchev,et al.  GRIM_DBMS: a GRaphical IMage DataBase Management System , 1989, VDB.

[41]  Gary Marchionini,et al.  Information Seeking in Electronic Environments , 1995 .

[42]  E. Rosch.  ON THE INTERNAL STRUCTURE OF PERCEPTUAL AND SEMANTIC CATEGORIES1 , 1973 .

[43]  Samantha Kelly Hastings,et al.  Query Categories in a Study of Intellectual Access to Digitized Art Images. , 1995 .

[44]  Thomas M. Strat,et al.  Context-Based Vision: Recognizing Objects Using Information from Both 2D and 3D Imagery , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[45]  Andrew Hollingworth,et al.  Eye Movements, Visual Memory, and Scene Representation , 2000 .

[46]  Peter G. B. Enser,et al.  Visual image retrieval: seeking the alliance of concept-based and content-based paradigms , 2000, J. Inf. Sci..

[47]  Toshikazu Kato,et al.  Database architecture for content-based image retrieval , 1992, Electronic Imaging.

[48]  M. Markkula,et al.  Searching for Photos - Journalists' Practices in Pictorial IR , 1998 .

[49]  D. Hinkle,et al.  Applied statistics for the behavioral sciences , 1979 .

[50]  Shih-Fu Chang,et al.  Image Retrieval: Current Techniques, Promising Directions, and Open Issues , 1999, J. Vis. Commun. Image Represent..

[51]  B. Dervin.  AN OVERVIEW OF SENSE-MAKING RESEARCH: CONCEPTS, METHODS AND RESULTS TO DATE , 1983 .

[52]  Peter G. B. Enser,et al.  Progress in Documentation Pictorial Information Retrieval , 1995, J. Documentation.

[53]  Peter G. B. Enser,et al.  Analysis of user need in image archives , 1997, J. Inf. Sci..

[54]  Michael Brady,et al.  Generating and Generalizing Models of Visual Objects , 1987, Artif. Intell..

[55]  Hayit Greenspan,et al.  Finding Pictures of Objects in Large Collections of Images , 1996, Object Representation in Computer Vision.

[56]  John H. Holland,et al.  Induction: Processes of Inference, Learning, and Discovery , 1987, IEEE Expert.

[57]  Alex Pentland,et al.  Photobook: tools for content-based manipulation of image databases , 1994, Electronic Imaging.

[58]  Sun-Yuan Kung,et al.  A hierarchical algorithm for image retrieval by sketch , 1997, Proceedings of First Signal Processing Society Workshop on Multimedia Signal Processing.

[59]  Michael J. Swain,et al.  Color indexing , 1991, International Journal of Computer Vision.

[60]  Hsin-Liang Chen.  An analysis of image queries in the field of art history , 2001 .

[61]  Peter G. B. Enser.  Pictorial information retrieval , 1995 .