User evaluation of Físchlár-News

Technological developments in content-based analysis of digital video information are undergoing much progress, with ideas for fully automatic systems now being proposed and demonstrated. Yet because we do not yet have robust operational video retrieval systems that can be deployed and used, the usual HCI practise of conducting a usage study and an informed iterative system design is thus not possible. Físchlár-News is one of the first automatic, content-based broadcast news analysis and archival systems that process broadcast news video so that users can search, browse, and play it in an easy-to-use manner with a conventional web browser. The system incorporates a number of state-of-the-art research components, some of which are not yet considered mature technology, yet it has been built to be robust enough to be deployed to users who are interested in access to daily news throughout a university campus. In this article we report and discuss a user-evaluation study conducted with 16 users, each of whom utilized the system freely for a one month period. Results from a detailed qualitative analysis are presented, looking at collected questionnaires, incident diaries, and interaction-log data. The findings suggest that our users employed the system in conjunction with their other news update methods, such as watching TV news at home and browsing online news websites at their workplace, their major concerns being up-to-dateness and coverage of the news content. They tried to accommodate the system to fit their established web browsing habits, and they found local news content and the ability to play self-contained news stories on their desktop as major values of the system. Our study also resulted in a detailed wishlist of new features which will help in the further development of both our and others' systems.

[1]  Michael J. Pazzani,et al.  A hybrid user model for news story classification , 1999 .

[2]  Zhu Zhang,et al.  NewsInEssence: A System For Domain-Independent, Real-Time News Clustering and Multi-Document Summarization , 2001, HLT.

[3]  Kirsten Swearingen,et al.  Interaction Design for Recommender Systems , 2002 .

[4]  John Riedl,et al.  Is seeing believing?: how recommender system interfaces affect users' opinions , 2003, CHI '03.

[5]  Shingo Uchihashi,et al.  An interactive comic book presentation for exploring video , 2000, CHI.

[6]  Graeme Hirst,et al.  Lexical Cohesion Computed by Thesaural relations as an indicator of the structure of text , 1991, CL.

[7]  David Evans,et al.  Tracking and summarizing news on a daily basis with Columbia's Newsblaster , 2002 .

[8]  Paul Over,et al.  The TREC VIdeo Retrieval Evaluation (TRECVID): A Case Study and Status Report , 2004, RIAO.

[9]  James Allan,et al.  Monitoring the News: a TDT demonstration system , 2001, HLT.

[10]  Shin'ichi Satoh,et al.  Topic Threading for Structuring a Large-Scale News Video Archive , 2004, CIVR.

[11]  Tom Rodden,et al.  At home with the technology: an ethnographic study of a set-top-box trial , 1999, TCHI.

[12]  Russell Swan,et al.  TimeMine: visualizing automatically constructed timelines. , 2000, SIGIR 2000.

[13]  Sean M. McNee,et al.  Confidence Displays and Training in Recommender Systems , 2003, INTERACT.

[14]  David A. Forsyth,et al.  Towards auto-documentary: tracking the evolution of news stories , 2004, MULTIMEDIA '04.

[15]  Claude Seyrat,et al.  Indexing and retrieval of multimedia objects at different levels of granularity , 1998, Other Conferences.

[16]  Alan F. Smeaton,et al.  Dublin City University Video Track Experiments for TREC 2002 , 2001, TREC.

[17]  Alan F. Smeaton,et al.  Designing the User Interface for the Físchlár Digital Video Library , 2006, J. Digit. Inf..

[18]  Michael G. Christel,et al.  Interactive maps for a digital video library , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[19]  Wolfgang Hürst,et al.  Advanced user interfaces for dynamic video browsing , 2004, MULTIMEDIA '04.

[20]  Shih-Fu Chang,et al.  Story boundary detection in large broadcast news video archives: techniques, experience and trends , 2004, MULTIMEDIA '04.

[21]  Alan F. Smeaton,et al.  A generic news story segmentation system and its evaluation , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[22]  Jean-Marc Odobez,et al.  A Hierarchical Keyframe User Interface for Browsing Video over the Internet , 2003, INTERACT.

[23]  Noel E. O'Connor,et al.  Evaluating and combining digital video shot boundary detection algorithms , 2000 .

[24]  Michael J. Witbrock,et al.  Story segmentation and detection of commercials in broadcast news video , 1998, Proceedings IEEE International Forum on Research and Technology Advances in Digital Libraries -ADL'98-.

[25]  Derry O'Sullivan,et al.  Explicit vs Implicit Profiling - A Case-Study in Electronic Programme Guides , 2003, IJCAI.

[26]  S. Satoh,et al.  Topic-Based Structuring of a Very Large-Scale News Video Corpus , 2003 .

[27]  Earl Rennison,et al.  Galaxy of news: an approach to visualizing and understanding expansive news landscapes , 1994, UIST '94.

[28]  Wai On Lee,et al.  Introducing Internet Terminals to the Home: Interaction Between Social, Physical, and Technological Spaces , 2000, BCS HCI.

[29]  Alan F. Smeaton,et al.  Improving the Quality of the Personalized Electronic Program Guide , 2004, User Modeling and User-Adapted Interaction.

[30]  Paul Over,et al.  Boundary Error Analysis and Categorization in the TRECVID News Story Segmentation Task , 2005, CIVR.

[31]  Ryen W. White,et al.  NewsFlash: Adaptive TV News Delivery on the Web , 2003, Adaptive Multimedia Retrieval.

[32]  Tobun Dorbin Ng,et al.  Collages as dynamic summaries for news video , 2002, MULTIMEDIA '02.

[33]  Tobun Dorbin Ng,et al.  Collages as Dynamic Summaries of Mined Video Content for Intelligent Multimedia Knowledge Management , 2003 .

[34]  James Allan,et al.  On-Line New Event Detection and Tracking , 1998, SIGIR.

[35]  Kenton O'Hara,et al.  Dealing with mobility: understanding access anytime, anywhere , 2001, TCHI.

[36]  Barry Smyth,et al.  A personalized television listings service , 2000, CACM.

[37]  John M. Gauch,et al.  Real Time Video Scene Detection and Classification , 1999, Inf. Process. Manag..

[38]  Alan F. Smeaton,et al.  Indexing, browsing, and searching of digital video , 2005, Annu. Rev. Inf. Sci. Technol..

[39]  Alan F. Smeaton,et al.  TRECVID 2004 Experiments in Dublin City University , 2004, TRECVID.

[40]  Stephen W. Smoliar,et al.  Video parsing, retrieval and browsing: an integrated and content-based solution , 1997, MULTIMEDIA '95.

[41]  John R. Smith,et al.  Multimodal video search techniques: late fusion of speech-based retrieval and visual content-based retrieval , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[42]  J. A. Fitzpatrick,et al.  Breakingstory: visualizing change in online news , 2003, CHI Extended Abstracts.

[43]  Alan F. Smeaton,et al.  TRECVid 2006 Experiments at Dublin City University , 2012, TRECVID.

[44]  Jonathan L. Herlocker,et al.  Evaluating collaborative filtering recommender systems , 2004, TOIS.

[45]  Peter J. Thomas,et al.  Introduction to the new usability , 2002, TCHI.

[46]  Alan F. Smeaton,et al.  The Físchlár-News-Stories System: Personalised Access to an Archive of TV News , 2004, RIAO.

[47]  Sean M. McNee,et al.  Interfaces for Eliciting New User Preferences in Recommender Systems , 2003, User Modeling.

[48]  Alan F. Smeaton,et al.  Evaluation of automatic shot boundary detection on a large video test suite , 1999 .

[49]  Alan F. Smeaton,et al.  User evaluation outside the lab: the trial of Físchlár-News , 2005 .

[50]  James Allan,et al.  TimeMine (demonstration session): visualizing automatically constructed timelines , 2000, SIGIR '00.

[51]  Kirsten Swearingen,et al.  Beyond Algorithms: An HCI Perspective on Recommender Systems , 2001 .

[52]  Kenneth Y. Goldberg,et al.  Eigentaste: A Constant Time Collaborative Filtering Algorithm , 2001, Information Retrieval.

[53]  Jaime Teevan,et al.  Implicit feedback for inferring user preference: a bibliography , 2003, SIGF.

[54]  Mark T. Maybury,et al.  Broadcast news navigation using story segmentation , 1997, MULTIMEDIA '97.

[55]  Bernard Mérialdo,et al.  Automatic construction of personalized TV news programs , 1999, MULTIMEDIA '99.

[56]  Gary Marchionini,et al.  From Information Retrieval to Information Interaction , 2004, ECIR.

[57]  Kim Halskov,et al.  The usability of everyday technology: emerging and fading opportunities , 2002, TCHI.

[58]  Paul Over,et al.  TRECVID 2004 - An Overview , 2004, TRECVID.

[59]  Alan F. Smeaton,et al.  Design, implementation and testing of an interactive video retrieval system , 2003, MIR '03.

[60]  Lawrence Wai-Choong Wong,et al.  ANSES: Summarisation of News Video , 2003, CIVR.

[61]  Julian Warner,et al.  Leonardo's Laptop: Human Needs and the New Computing Technologies , 2006, J. Documentation.