Empirical Evaluations of Interactive Systems in Cultural Heritage: A Review

An increasing number of interactive systems aim to enhance the user experience UX of visitors at museums, archaeological places and cultural sites. This paper presents a review of empirical evaluations of interactive systems in cultural heritage CH based on a systematically selected sample of 53 publications from 2012-2016. Empirical evaluations examine the degree an interactive system satisfies user goals and expectations and are inherently complex activities that require careful planning and selection of methods. The review reports on a interactive systems of CH in terms of: purpose, technology, cultural content and location of interaction and b methods of empirical evaluation in terms of: dimensions of UX, general approach, data collection and participants. The paper provides discussion, critique and outlook on issues including: extending the evaluation dimensions towards the cultural value and the essence of interactivity; placing more consideration on CH professionals; identifying new evaluation methods that balance qualitative results to evidence-based approaches.

[1]  Tom Carey,et al.  ACM SIGCHI Curricula for Human-Computer Interaction , 1992 .

[2]  Jeffrey Rubin,et al.  Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests , 1994 .

[3]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[4]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[5]  Michael J. Singer,et al.  Measuring Presence in Virtual Environments: A Presence Questionnaire , 1998, Presence.

[6]  I. Olkin,et al.  Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement , 1999, The Lancet.

[7]  Jennifer L. Berdahl,et al.  The Study of Groups: Past, Present, and Future , 2000 .

[8]  Michael Burmester,et al.  AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität , 2003, MuC.

[9]  George Pavlidis,et al.  Methods for 3D digitization of Cultural Heritage , 2007 .

[10]  Janet C. Read,et al.  Validating the Fun Toolkit: an instrument for measuring children’s opinions of technology , 2008, Cognition, Technology & Work.

[11]  Anna L. Cox,et al.  Research Methods for Human-Computer Interaction , 2008 .

[12]  Harry Budi Santoso,et al.  Measuring the user experience , 2008 .

[13]  John Zimmerman,et al.  An analysis and critique of Research through Design: towards a formalization of a research approach , 2010, Conference on Designing Interactive Systems.

[14]  Kasper Hornbæk,et al.  Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience , 2011, CHI.

[15]  Erik Champion,et al.  Chinese culture approached through touch: Chinese cultural heritage learnt via touch-based games , 2012, 2012 18th International Conference on Virtual Systems and Multimedia.

[16]  Jeff Sauro,et al.  Quantifying the User Experience: Practical Statistics for User Research , 2012 .

[17]  Josef Froschauer,et al.  Analysing user motivation in an art folksonomy , 2012, i-KNOW '12.

[18]  Michael E. Atwood,et al.  Changing perspectives on evaluation in HCI: past, present, and future , 2013, CHI Extended Abstracts.

[19]  Tanguy Coenen,et al.  MuseUs: Case study of a pervasive cultural heritage serious game , 2013, JOCCH.

[20]  Yi-Ting Huang,et al.  An annales school-based serious game creation framework for taiwanese indigenous cultural heritage , 2013, JOCCH.

[21]  Alan Miller,et al.  Exploring canons & cathedrals with Open Virtual Worlds: The recreation of St Andrews Cathedral, St Andrews day, 1318 , 2013, 2013 Digital Heritage International Congress (DigitalHeritage).

[22]  Tom Tullis,et al.  Measuring the User Experience, Second Edition: Collecting, Analyzing, and Presenting Usability Metrics , 2013 .

[23]  Josef Froschauer,et al.  Art history concepts at play with ThIATRO , 2013, JOCCH.

[24]  Chris Creed,et al.  Multi-Touch Tables for Exploring Heritage Content in Public Spaces , 2013, Visual Heritage in the Digital Age.

[25]  Alessandro De Gloria,et al.  A serious game model for cultural heritage , 2012, JOCCH.

[26]  Ben Shneiderman,et al.  Designing The User Interface , 2013 .

[27]  Charles B. Callaway,et al.  Experiments with Mobile Drama in an Instrumented Museum for Inducing Conversation in Small Groups , 2014, TIIS.

[28]  Christian Sandor,et al.  A usability scale for handheld augmented reality , 2014, VRST '14.

[29]  Markku Reunanen,et al.  A Holistic User-Centered Approach to Immersive Digital Cultural Heritage Installations , 2015, ACM Journal on Computing and Cultural Heritage.

[30]  Lorenzo Picinali,et al.  Audio Tactile Maps (ATM) System for the Exploration of Digital Heritage Buildings by Visually-impaired Individuals - First Prototype and Preliminary Evaluation , 2014 .

[31]  Rita Castro,et al.  Design of an Interactive Experience with Medieval Illuminations: A Journey into the Beauty and Meaning of Medieval Portuguese Manuscripts , 2014, JOCCH.

[32]  Torbjörn Svensson,et al.  The Mystery of Elin. Incorporating a City Cultural Program on History and Heritage into a Pervasive Game , 2014, IE.

[33]  Andreas Reichinger,et al.  3D Printing for Cultural Heritage: Preservation, Accessibility, Research and Education , 2014, 3D Research Challenges in Cultural Heritage.

[34]  Jonas Fritsch,et al.  Designing interactive listening situations , 2014, OZCHI.

[35]  Andreas Lanitis,et al.  Presenting Cypriot Cultural Heritage in Virtual Reality: A User Evaluation , 2014, EuroMed.

[36]  Alberto Jaspe Villanueva,et al.  IsoCam: Interactive Visual Exploration of Massive Cultural Heritage Models on Large Projection Setups , 2014, JOCCH.

[37]  Nuno Jardim Nunes,et al.  The state of user experience evaluation practice , 2014, NordiCHI.

[38]  Yi-Ping Hung,et al.  Interacting with the past: Creating a time perception journey experience using kinect-based breath detection and deterioration and recovery simulation technologies , 2014, JOCCH.

[39]  Mary Beth Rosson,et al.  Enhancing community awareness of and participation in local heritage with a mobile application , 2014, CSCW.

[40]  Christos Sintoris,et al.  Do you remember that building? Exploring old Zakynthos through an augmented reality mobile game , 2014, 2014 International Conference on Interactive Mobile Communication Technologies and Learning (IMCL2014).

[41]  Xuan Wang,et al.  Interacting with Traditional Chinese Culture through Natural Language , 2014, JOCCH.

[42]  Martin Wolpers,et al.  Natural Europe educational games suite: using structured museum-data for creating mobile educational games , 2014, Advances in Computer Entertainment.

[43]  Eva Pietroni,et al.  Interacting with Virtual Reconstructions in Museums: The Etruscanning Project , 2014, JOCCH.

[44]  Gabriela Avram,et al.  Interactive exhibitions design: what can we learn from cultural heritage professionals? , 2014, NordiCHI.

[45]  Antti Jylhä,et al.  A Wearable Multimodal Interface for Exploring Urban Points of Interest , 2015, ICMI.

[46]  Charalambos Poullis,et al.  Visualizing and Assessing Hypotheses for Marine Archaeology in a VR CAVE Environment , 2015, ACM Journal on Computing and Cultural Heritage.

[47]  Perla Gianni Falvo Exhibit design with multimedia and cognitive technologies impact assessment on Luca Giordano, Raphael and the Chapel of the Magi in Palazzo Medici Riccardi, Florence , 2015, 2015 Digital Heritage.

[48]  Martin White,et al.  Evaluation of a Cultural Heritage Augmented Reality Game , 2015 .

[49]  Alan Miller,et al.  The Virtual Museums of Caen: A case study on modes of representation of digital historical content , 2015, 2015 Digital Heritage.

[50]  Jens Keil,et al.  A contextualized educational museum experience connecting objects, places and themes through mobile virtual museums , 2015, 2015 Digital Heritage.

[51]  David Benyon,et al.  Using the blended spaces framework to design heritage stories with schoolchildren , 2015, Int. J. Child Comput. Interact..

[52]  Jetmir Xhembulla,et al.  Integrating a Location-Based Mobile Game in the Museum Visit , 2015, ACM Journal on Computing and Cultural Heritage.

[53]  Selma Rizvic,et al.  Reconstructing cultural heritage objects from storytelling , 2015, 2015 Digital Heritage.

[54]  Paloma Díaz,et al.  A See-Through Display for Interactive Museum Showcases , 2015, ITS.

[55]  Massimo Bergamasco,et al.  A Virtual Laboratory An immersive VR experience to spread ancient libraries heritage , 2015, 2015 Digital Heritage.

[56]  Paola Di Giuseppantonio Di Franco,et al.  3D Printing and Immersive Visualization for Improved Perception of Ancient Artifacts , 2015, PRESENCE: Teleoperators and Virtual Environments.

[57]  Emanuel Demetrescu,et al.  Engaging and shared gesture-based interaction for museums the case study of K2R international expo in Rome , 2015, 2015 Digital Heritage.

[58]  Bruno Fanini,et al.  Interface design for serious game visual strategies the case study of “Imago Bononiae” , 2015, 2015 Digital Heritage.

[59]  Shanshan Jiang,et al.  One place, many stories digital storytelling for cultural heritage discovery in the landscape , 2015, 2015 Digital Heritage.

[60]  Hussein Suleman,et al.  Enhancing Digital Heritage Archives Using Gamified Annotations , 2015, ICADL.

[61]  Ju-Ling Shih,et al.  A simulated learning environment of history games for enhancing players’ cultural awareness , 2015, Interact. Learn. Environ..

[62]  P. Brunet,et al.  Through the Loupe: Visitor Engagement With a Primarily Text-Based Handheld AR Application , 2015 .

[63]  Seugnet Blignaut,et al.  Success factors for serious games to enhance learning: a systematic review , 2017, Virtual Reality.

[64]  Daniela Petrelli,et al.  Audio-based narratives for the trenches of World War I: Intertwining stories, places and interaction for an evocative experience , 2016, Int. J. Hum. Comput. Stud..

[65]  Ioannis Paliokas,et al.  The Use of Serious Games in Museum Visits and Exhibitions: A Systematic Mapping Study , 2016, 2016 8th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES).

[66]  Fotis Liarokapis,et al.  Examining User Experiences in a Mobile Augmented Reality Tourist Guide , 2016, PETRA.

[67]  Nick Dulake,et al.  Articulating Co-Design in Museums: Reflections on Two Participatory Processes , 2016, CSCW.

[68]  Maria Roussou,et al.  Families and Mobile Devices in Museums , 2016, ACM Journal on Computing and Cultural Heritage.

[69]  Marcello Carrozzino,et al.  Time-Travelling with Mobile Augmented Reality: A Case Study on the Piazza dei Miracoli , 2016, EuroMed.

[70]  Spyros Vosinakis,et al.  Accessible museum collections for the visually impaired: combining tactile exploration, audio descriptions and mobile gestures , 2016, MobileHCI Adjunct.

[71]  Claus B. Madsen,et al.  Handheld Visual Representation of a Castle Chapel Ruin , 2016, ACM Journal on Computing and Cultural Heritage.

[72]  Damianos Gavalas,et al.  Augmented reality in cultural heritage: Field of view awareness in an archaeological site mobile guide , 2016, J. Ambient Intell. Smart Environ..

[73]  Vittorio Scarano,et al.  Immersivity and Playability Evaluation of a Game Experience in Cultural Heritage , 2016, EuroMed.

[74]  Spyros Vosinakis,et al.  Adult and Children User Experience with Leap Motion in Digital Heritage: The Cycladic Sculpture Application , 2016, EuroMed.

[75]  Spyros Vosinakis,et al.  A pervasive role-playing game for introducing elementary school students to archaeology , 2016, MobileHCI Adjunct.

[76]  Andreas Lanitis,et al.  Paintings Alive: A Virtual Reality-Based Approach for Enhancing the User Experience of Art Gallery Visitors , 2016, EuroMed.

[77]  Asim Evren Yantaç,et al.  Towards a Sustainable Crowdsourced Sound Heritage Archive by Public Participation: The Soundsslike Project , 2016, NordiCHI.

[78]  Katerina Kabassi,et al.  Evaluating websites of museums: State of the art , 2017 .

[79]  Anton Bogdanovych,et al.  Virtual worlds vs books and videos in history education , 2017, Interact. Learn. Environ..

[80]  Alan J. Dix Human-Computer Interaction , 2018, Encyclopedia of Database Systems.