Exploratory Search in an Audio-Visual Archive: Evaluating a Professional Search Tool for Non-Professional Users

As archives are opening up and publishing their content online, the general public can now directly access archive collections. To support access, archives typically provide the public with their internal search tools that were originally intended for professional archivists. We conduct a small-scale user study where non-professionals perform exploratory search tasks with a search tool originally developed for media professionals and archivists in an audio visual archive. We evaluate the tool using objective and subjective measures and find that non-professionals find the search interface difficult to use in terms of both. Analysis of search behavior shows that non-professionals often visiting the description page of individual items in a result list are more successful on search tasks than those who visit fewer pages. A more direct presentation of entities present in the metadata fields of items in a result list can be beneficial for non-professional users on exploratory search tasks.

[1]  Laura Hollink,et al.  Search behavior of media professionals at an audiovisual archive: A transaction log analysis , 2010 .

[2]  Mark J. Huiskes,et al.  The MIR flickr retrieval evaluation , 2008, MIR '08.

[3]  G. A. Miller.  THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[4]  Gary Marchionini,et al.  Exploratory search , 2006, Commun. ACM.

[5]  Szymon Rusinkiewicz,et al.  Rotation Invariant Spherical Harmonic Representation of 3D Shape Descriptors , 2003, Symposium on Geometry Processing.

[6]  Ben Shneiderman,et al.  Direct manipulation for comprehensible, predictable and controllable user interfaces , 1997, IUI '97.

[7]  Joaquim A. Jorge,et al.  Retrieving ClipArt Images by Content , 2004, CIVR.

[8]  Maarten de Rijke,et al.  Search behavior of media professionals at an audiovisual archive: A transaction log analysis , 2010, J. Assoc. Inf. Sci. Technol..

[9]  Thomas Mandl,et al.  The Influence of Expectation and System Performance on User Satisfaction with Retrieval Systems , 2010, EVIA@NTCIR.

[10]  Bruce Archer,et al.  Design as a discipline , 1979 .

[11]  Donna K. Harman,et al.  The TREC Test Collections , 2005 .

[12]  Andries van Dam,et al.  Post-WIMP user interfaces , 1997, CACM.

[13]  Robin Jeffries,et al.  Orienteering in an information landscape: how information seekers get from here to there , 1993, INTERCHI.

[14]  Thierry Pun,et al.  Performance evaluation in content-based image retrieval: overview and proposals , 2001, Pattern Recognit. Lett..

[15]  Thomas S. Huang,et al.  3D MARS: immersive virtual reality for content-based image retrieval , 2001, IEEE International Conference on Multimedia and Expo, 2001. ICME 2001..

[16]  Bryan Lawson,et al.  How Designers Think: The Design Process Demystified , 1990 .

[17]  S. Nakade,et al.  Mosaicture: Image Mosaic Generating System Using CBIR Technique , 2007, International Conference on Computational Intelligence and Multimedia Applications (ICCIMA 2007).

[18]  David P. Dobkin,et al.  A search engine for 3D models , 2003, TOGS.

[19]  Shamkant B. Navathe,et al.  Generalized formal models for faceted user interfaces , 2009, JCDL '09.

[20]  A. Strauss,et al.  The discovery of grounded theory: strategies for qualitative research aldine de gruyter , 1968 .

[21]  Vid Petrovic,et al.  The future of the CAVE , 2011 .

[22]  Monica M. C. Schraefel,et al.  mSpace: improving information access to multimedia domains with multimodal exploratory search , 2006, CACM.

[23]  Donald A. Norman,et al.  Logic versus usage: the case for activity-centered design , 2006, INTR.

[24]  Bill Buxton,et al.  Sketching User Experiences: Getting the Design Right and the Right Design , 2007 .

[25]  Thomas Mandl,et al.  Evaluation of five web search engines in Arabic language , 2010, LWA.

[26]  Robert J. K. Jacob,et al.  Using fNIRS brain sensing in realistic HCI settings: experiments and guidelines , 2009, UIST '09.

[27]  Marti A. Hearst.  Search User Interfaces , 2009 .

[28]  Ben Shneiderman,et al.  The eyes have it: a task by data type taxonomy for information visualizations , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[29]  Monica M. C. Schraefel,et al.  Backward highlighting: enhancing faceted search , 2008, UIST '08.

[30]  Marcia J. Bates,et al.  Information search tactics , 1979, J. Am. Soc. Inf. Sci..

[31]  Irwin King,et al.  Montage: An Image Database for the Fashion, Textile, and Clothing Industry in Hong Kong , 1998, ACCV.

[32]  Michael Burmester,et al.  AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität , 2003, MuC.

[33]  Robert Silvers,et al.  Photomosaics : putting pictures in their place , 1996 .

[34]  Ishani Chakraborty,et al.  Correlating low-level image statistics with users - rapid aesthetic and affective judgments of web pages , 2009, CHI.

[35]  S. Greenberg,et al.  The Psychology of Everyday Things , 2012 .

[36]  Donald A. Norman,et al.  Natural user interfaces are not natural , 2010, INTR.

[37]  Todd C. Kelley,et al.  The Art of Innovation: Lessons in Creativity from IDEO, America's Leading Design Firm , 2001 .

[38]  Julio Gonzalo,et al.  WePS-3 Evaluation Campaign: Overview of the Web People Search Clustering and Attribute Extraction Tasks , 2010, CLEF.

[39]  Ann Blandford,et al.  Exploring the Impact of Search Interface Features on Search Tasks , 2010, ECDL.

[40]  Samuel S. Silva,et al.  Head-mounted display versus desktop for 3D navigation in virtual reality: a user study , 2008, Multimedia Tools and Applications.

[41]  Nicholas J. Belkin,et al.  Some(what) grand challenges for information retrieval , 2008, SIGF.

[42]  Gerard Salton,et al.  Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer , 1989 .

[43]  Ryen W. White,et al.  Finding relevant documents using top ranking sentences: an evaluation of two alternative schemes , 2002, SIGIR '02.

[44]  Sharon L. Oviatt,et al.  Human-centered design meets cognitive load theory: designing interfaces that help people think , 2006, MM '06.

[45]  Jaana Kekäläinen,et al.  Cumulated gain-based evaluation of IR techniques , 2002, TOIS.

[46]  Thomas Mandl,et al.  Mehrwertpotenziale von Online-Social-Business-Netzwerken für die Personalbeschaffung von Fach- und Führungskräften , 2009, GI Jahrestagung.

[47]  Jaana Kekäläinen,et al.  Intuition-supporting visualization of user's performance based on explicit negative higher-order relevance , 2008, SIGIR '08.

[48]  Gary Marchionini,et al.  Effects of structure and interaction style on distinct search tasks , 2007, JCDL '07.

[49]  Daniel A. Keim,et al.  Visual Analytics: Definition, Process, and Challenges , 2008, Information Visualization.

[50]  Andrei Broder,et al.  A taxonomy of web search , 2002, SIGF.

[51]  Giuseppe Santucci,et al.  Visual Comparison of Ranked Result Cumulated Gains , 2011, EuroVA@EuroVis.

[52]  Sandra G. Hart,et al.  Nasa-Task Load Index (NASA-TLX); 20 Years Later , 2006 .

[53]  H. Simon,et al.  The Processes of Creative Thinking , 1959 .

[54]  Josep Blat,et al.  Parameterized User-Centered Design for Interacting with Multimedia Repositories , 2009, 2009 First International Conference on Advances in Multimedia.

[55]  R. J. Bogumil,et al.  The reflective practitioner: How professionals think in action , 1985, Proceedings of the IEEE.

[56]  Stephen E. Robertson,et al.  On the history of evaluation in IR , 2008, J. Inf. Sci..

[57]  O. Reiser,et al.  Principles Of Gestalt Psychology , 1936 .

[58]  Peter Brusilovsky,et al.  Semantic annotation based exploratory search for information analysts , 2010, Inf. Process. Manag..

[59]  Daniel Vogel,et al.  Interactive public ambient displays: transitioning from implicit to explicit, public to personal, interaction with multiple users , 2004, UIST '04.

[60]  Jing Huang,et al.  Image indexing using color correlograms , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[61]  Max L. Wilson,et al.  The Revisit Rack: Grouping web search thumbnails for optimal visual recognition , 2010, ASIST.

[62]  James J. Thomas,et al.  Defining Insight for Visual Analytics , 2009, IEEE Computer Graphics and Applications.

[63]  Susanne Bødker,et al.  When second wave HCI meets third wave challenges , 2006, NordiCHI '06.

[64]  Thomas Sikora,et al.  The MPEG-7 visual standard for content description-an overview , 2001, IEEE Trans. Circuits Syst. Video Technol..

[65]  Josep Blat,et al.  A basic framework for integrating social and collaborative applications into learning environments , 2009 .

[66]  Max L. Wilson,et al.  Tag clouds and keyword clouds: evaluating zero-interaction benefits , 2011, CHI EA '11.

[67]  Howard D. Wactlar,et al.  Constant density displays using diversity sampling , 2003, IEEE Symposium on Information Visualization 2003 (IEEE Cat. No.03TH8714).

[68]  Ryen W. White,et al.  Evaluating advanced search interfaces using established information-seeking models , 2009, J. Assoc. Inf. Sci. Technol..

[69]  Daniel A. Keim,et al.  Challenges in Visual Data Analysis , 2006, Tenth International Conference on Information Visualisation (IV'06).

[70]  Thomas Mandl.  Recent Developments in the Evaluation of Information Retrieval Systems : Moving Toward Diversity and Practical Applications , 2005 .

[71]  Johan Redström,et al.  Slow Technology – Designing for Reflection , 2001, Personal and Ubiquitous Computing.

[72]  Barton C. Massey,et al.  DESIGN METHODS , 2002 .

[73]  Joaquim A. Jorge,et al.  Content-based retrieval of technical drawings , 2005, Int. J. Comput. Appl. Technol..

[74]  Thomas Mandl,et al.  Quality in Blogs: How to Find the Best User Generated Content , 2010, BIS.

[75]  Giorgio Maria Di Nunzio,et al.  10 Years of CLEF Data in DIRECT: Where We Are and Where We Can Go , 2010, EVIA@NTCIR.

[76]  Jock D. Mackinlay,et al.  The structure of the information visualization design space , 1997, Proceedings of VIZ '97: Visualization Conference, Information Visualization Symposium and Parallel Rendering Symposium.

[77]  Manuel J. Fonseca,et al.  Geometric matching for clip-art drawing retrieval , 2009, J. Vis. Commun. Image Represent..

[78]  Johnny Chung Lee,et al.  Hacking the Nintendo Wii Remote , 2008, IEEE Pervasive Computing.

[79]  Ryen W. White,et al.  Supporting exploratory search , 2006 .

[80]  Marc Rioux,et al.  Nefertiti: a query by content software for three-dimensional models databases management , 1997, Proceedings. International Conference on Recent Advances in 3-D Digital Imaging and Modeling (Cat. No.97TB100134).

[81]  Thomas Mandl,et al.  The effect of named entities on effectiveness in cross-language information retrieval evaluation , 2005, SAC '05.

[82]  Daniel A. Keim,et al.  Visual Analytics Challenges , 2009 .

[83]  Roy T. Fielding,et al.  Principled design of the modern Web architecture , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[84]  Ryen W. White,et al.  Supporting Exploratory Search, Introduction, Special Issue, Communications of the ACM , 2006 .

[85]  Manuel J. Fonseca,et al.  Sketch-based retrieval of drawings using spatial proximity , 2010, J. Vis. Lang. Comput..

[86]  Mary Madden,et al.  Reputation Management and Social Media: How People Monitor Their Identity and Search for Others Online , 2010 .

[87]  David Ma,et al.  Does domain highlighting help people identify phishing sites? , 2011, CHI.

[88]  Ben Shneiderman,et al.  A Rank-by-Feature Framework for Interactive Exploration of Multidimensional Data , 2005, Inf. Vis..

[89]  Marcia J. Bates,et al.  The design of browsing and berrypicking techniques for the online search interface , 1989 .

[90]  Gerald Schaefer,et al.  Evaluation and Benchmarking of Image Database Navigation Tools , 2009, IPCV.

[91]  Daniel E. Rose,et al.  Understanding user goals in web search , 2004, WWW '04.