A framework for understanding user interaction with content-based image retrieval: model, interface and users

User interaction is essential to the communication between users and content-based image retrieval (CBIR) systems. User interaction covers three key elements: an interaction model, an interactive interface and users. The three key elements combine to enable effective interaction to happen. Many studies have investigated different aspects of user interaction. However, there is lack of research in combining all three elements in an integrated manner, especially through well-principled data analysis based on a systematic user study. In this thesis, we investigate the combination of all three elements for interactive CBIR. We first propose uInteract - a framework including a novel four-factor user interaction model (FFUIM) and an interactive interface. The FFUIM aims to improve interaction and search accuracy of the relevance feedback mechanism for CBIR. The interface delivers the FFUIM visually, aiming to support users in grasping how the interaction model functions and how best to manipulate it. The framework is tested in three task-based and user-oriented comparative evaluations, which involves 12 comparative systems, 12 real life scenario tasks and 50 subjects. The quantitative data analysis shows encouraging observations on ease of use and usefulness of the proposed framework, and also reveals a large variance of the results depending on different user types. Accordingly, based on Information Foraging Theory, we further propose a user classification model along three user interaction dimensions: information goals (I), search strategies (S) and evaluation thresholds (E) of users. To our best knowledge, this is the first principled user classification model in CBIR. The model is operated and verified by a systematic qualitative data analysis based on multi linear regression on the real user interaction data from comparative user evaluations. From final quantitative and qualitative data analysis based on the ISE model, we have established what different types of users like about the framework and their preferences for interactive CBIR systems. Our findings offer useful guidelines for interactive search system design, evaluation and analysis.

[1]  Andreas Nürnberger,et al.  Summarizing Video Information Using Self-Organizing Maps , 2006, 2006 IEEE International Conference on Fuzzy Systems.

[2]  N. Boujemaa,et al.  Relevance Feedback for Image Retrieval : a Short Survey , 2004 .

[3]  William A. Brenneman.  Statistics for Research , 2005, Technometrics.

[4]  Stuart K. Card,et al.  Information foraging in information access environments , 1995, CHI '95.

[5]  Ryen W. White,et al.  A study of interface support mechanisms for interactive information retrieval , 2006, J. Assoc. Inf. Sci. Technol..

[6]  Mounia Lalmas,et al.  A survey on the use of relevance feedback for information access systems , 2003, The Knowledge Engineering Review.

[7]  Peter Wegner,et al.  Why interaction is more powerful than algorithms , 1997, CACM.

[8]  Peter Ingwersen,et al.  Information Retrieval Interaction , 1992 .

[9]  John P. Eakins,et al.  Shape Feature Matching for Trademark Image Retrieval , 2003, CIVR.

[10]  Victoria S. Uren,et al.  Enabling Effective User Interactions in Content-Based Image Retrieval , 2009, AIRS.

[11]  Nicholas J. Belkin,et al.  Relationships between categories of relevance criteria and stage in task completion , 2007, Inf. Process. Manag..

[12]  Marcus Jerome Pickering,et al.  Evaluation of key frame-based retrieval techniques for video , 2003, Comput. Vis. Image Underst..

[13]  Paul Browne.  Video information retrieval using objects and ostensive relevance feedback , 2004, SAC '04.

[14]  Melody Y. Ivory,et al.  Search result exploration: a preliminary study of blind and sighted users' decision making and performance , 2004, CHI EA '04.

[15]  Marti A. Hearst,et al.  Scatter/gather browsing communicates the topic structure of a very large text collection , 1996, CHI.

[16]  Frank Hopfgartner,et al.  Simulated Testing of an Adaptive Multimedia Information Retrieval System , 2007, 2007 International Workshop on Content-Based Multimedia Indexing.

[17]  J M Lauritsen,et al.  [Data collection and analysis]. , 1999, Ugeskrift for laeger.

[18]  Peter Pirolli,et al.  Information Scent and Web Navigation : Theory , Models , and Automated Usability Evaluation , 2006 .

[19]  Dan Morris,et al.  Investigating the querying and browsing behavior of advanced search engine users , 2007, SIGIR.

[20]  Ben Shneiderman,et al.  Users can change their web search tactics: Design guidelines for categorized overviews , 2008, Inf. Process. Manag..

[21]  Gerard Salton,et al.  Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer , 1989 .

[22]  Wei-Pang Yang,et al.  A two-level relevance feedback mechanism for image retrieval , 2008, Expert Syst. Appl..

[23]  Stuart K. Card,et al.  The effects of information scent on visual search in the hyperbolic tree browser , 2003, TCHI.

[24]  Thomas S. Huang,et al.  Relevance feedback in image retrieval: A comprehensive review , 2003, Multimedia Systems.

[25]  Guojun Lu,et al.  Evaluation of similarity measurement for image retrieval , 2003, International Conference on Neural Networks and Signal Processing, 2003. Proceedings of the 2003.

[26]  Gary Marchionini,et al.  Exploratory search , 2006, Commun. ACM.

[27]  Chaomei Chen,et al.  Similarity-Based Image Browsing , 2000 .

[29]  Michael McGill,et al.  A performance evaluation of similarity measures, document term weighting schemes and representations in a Boolean environment , 1980, SIGIR '80.

[30]  Peter Pirolli,et al.  Information Foraging , 2009, Encyclopedia of Database Systems.

[31]  Gary Marchionini,et al.  Exploratory search and HCI: designing and evaluating interfaces to support exploratory search interaction , 2007, CHI Extended Abstracts.

[32]  Paul Clough,et al.  The IAPR TC-12 Benchmark: A New Evaluation Resource for Visual Information Systems , 2006 .

[33]  Arthur H. M. ter Hofstede,et al.  Query Formulation as an Information Retrieval Problem , 1996, Comput. J..

[34]  Matti Pietikäinen,et al.  A comparative study of texture measures with classification based on featured distributions , 1996, Pattern Recognit..

[35]  C. J. van Rijsbergen,et al.  Incorporating user search behavior into relevance feedback , 2003, J. Assoc. Inf. Sci. Technol..

[36]  Hanqing Lu,et al.  WillHunter: Interactive Image Retrieval with Multilevel Relevance Measurement , 2004, ICPR 2004.

[37]  Alexander Dekhtyar,et al.  Information Retrieval , 2018, Lecture Notes in Computer Science.

[38]  Ryen W. White,et al.  An implicit feedback approach for interactive information retrieval , 2006, Inf. Process. Manag..

[39]  Zi Huang,et al.  Dissimilarity measures for content-based image retrieval , 2008, 2008 IEEE International Conference on Multimedia and Expo.

[40]  Borko Furht,et al.  Content-Based Image and Video Retrieval , 2002, Multimedia Systems and Applications Series.

[41]  Ryen W. White,et al.  A study of factors affecting the utility of implicit relevance feedback , 2005, SIGIR '05.

[42]  Daniel Heesch,et al.  Performance boosting with three mouse clicks - Relevance feedback for CBIR , 2003 .

[43]  Ryen W. White,et al.  Supporting exploratory search , 2006 .

[44]  Bettina Berendt,et al.  Inf Retrieval , 2008 .

[45]  Joachim M. Buhmann,et al.  Non-parametric similarity measures for unsupervised texture segmentation and image retrieval , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[46]  Joachim M. Buhmann,et al.  Empirical evaluation of dissimilarity measures for color and texture , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[47]  Peter Ingwersen,et al.  The development of a method for the evaluation of interactive information retrieval systems , 1997, J. Documentation.

[48]  Chaur-Chin Chen,et al.  Similarity Measurement Between Images , 2005, COMPSAC.

[49]  Jana Urban,et al.  An adaptive approach for image organisation and retrieval , 2003 .

[50]  Stefan M. Rüger,et al.  Fractional Distance Measures for Content-Based Image Retrieval , 2005, ECIR.

[51]  Mika Käki,et al.  Enhancing Web Search Result Access with Automatic Categorization , 2005 .

[52]  Iain Campbell,et al.  Interactive Evaluation of the Ostensive Model Using a New Test Collection of Images with Multiple Relevance Assessments , 2000, Information Retrieval.

[53]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[54]  Thomas S. Huang,et al.  Relevance feedback: a power tool for interactive content-based image retrieval , 1998, IEEE Trans. Circuits Syst. Video Technol..

[55]  Victoria S. Uren,et al.  Comparing Dissimilarity Measures for Content-Based Image Retrieval , 2008, AIRS.

[56]  Mark D. Dunlop.  The effect of accessing nonmatching documents on relevance feedback , 1997, TOIS.

[57]  Hermann Ney,et al.  FIRE - Flexible Image Retrieval Engine: ImageCLEF 2004 Evaluation , 2004, CLEF.

[58]  Peter Pirolli,et al.  Computational models of information scent-following in a very large browsable text collection , 1997, CHI.

[59]  Thierry Pun,et al.  Strategies for positive and negative relevance feedback in image retrieval , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[60]  Stefan M. Rüger,et al.  Performance Comparison of Different Similarity Models for CBIR with Relevance Feedback , 2003, CIVR.

[61]  Jana Urban,et al.  AN ADAPTIVE APPROACH TOWARDS CONTENT-BASED IMAGE RETRIEVAL , 2003 .

[62]  Paul Mulholland,et al.  Investigating the Effects of Exploratory Semantic Search on the Use of a Museum Archive , 2008 .

[63]  Joemon M. Jose,et al.  Spatial querying for image retrieval: a user-oriented evaluation , 1998, SIGIR '98.

[64]  Daniel Heesch.  The NN k technique for image searching and browsing , 2005 .

[65]  Nicholas J. Belkin,et al.  The TREC Interactive Tracks: Putting the User into Search , 2005 .

[66]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[67]  Norbert Fuhr,et al.  A probability ranking principle for interactive information retrieval , 2008, Information Retrieval.

[68]  Joemon M. Jose,et al.  EGO: A personalized multimedia management and retrieval tool , 2006, Int. J. Intell. Syst..

[69]  Ed H. Chi,et al.  Using information scent to model user information needs and actions and the Web , 2001, CHI.

[70]  Pamela Effrein Sandstrom,et al.  Information Foraging Theory: Adaptive Interaction with Information , 2010, J. Assoc. Inf. Sci. Technol..

[71]  Joemon M. Jose,et al.  An adaptive technique for content-based image retrieval , 2006, Multimedia Tools and Applications.

[72]  Leonidas J. Guibas,et al.  The Earth Mover's Distance as a Metric for Image Retrieval , 2000, International Journal of Computer Vision.

[73]  Mika Käki,et al.  Controlling the complexity in comparing search user interfaces via user studies , 2008, Information Processing & Management.

[74]  J. Nielsen.  Information foraging : Why google makes people leave your site faster , 2003 .

[75]  Jan Puzicha,et al.  Distribution-based Image Similarity , 1999, State-of-the-Art in Content-Based Image and Video Retrieval.

[76]  Amanda Spink,et al.  From Highly Relevant to Not Relevant: Examining Different Regions of Relevance , 1998, Inf. Process. Manag..

[77]  Stefan Rüger,et al.  Robust texture features for still-image retrieval , 2005 .

[78]  Keith van Rijsbergen,et al.  Ranking expansion terms with partial and ostensive evidence , 2002 .

[79]  Victoria S. Uren,et al.  A Four-Factor User Interaction Model for Content-Based Image Retrieval , 2009, ICTIR.

[80]  Marcia J. Bates,et al.  Where should the person stop and the information search interface start? , 1990, Inf. Process. Manag..

[81]  Joemon M. Jose,et al.  Evaluating a workspace’s usefulness for image retrieval , 2006, Multimedia Systems.

[82]  Ryen W. White,et al.  Exploratory Search: Beyond the Query-Response Paradigm , 2009, Exploratory Search: Beyond the Query-Response Paradigm.

[83]  B. N. Chatterji,et al.  Comparison of similarity metrics for texture image retrieval , 2003, TENCON 2003. Conference on Convergent Technologies for Asia-Pacific Region.

[84]  Kalervo Järvelin,et al.  Explaining User Performance in Information Retrieval: Challenges to IR Evaluation , 2009, ICTIR.