Towards Diverse Media Augmented E-Book Reader Platform

In order to leverage the use of various modalities such as audio-visual-touch in instilling learning behaviour, we present an intuitive approach of annotation based hapto-audiovisual interaction with the traditional digital learning materials such as eBooks. By integrating the traditional home entertainment system and respective media in the user’s reading experience combined with haptic interfaces, we examine whether such augmentation of modalities influence the user’s reading experience in terms of attention, entertainment and retention. The proposed Haptic E-Book (HE-Book) system leverages the haptic jacket, haptic arm band as well as haptic sofa interfaces to receive haptic emotive signals wirelessly in the form of patterned vibrations of the actuators and expresses the learning material by incorporating audio-video based augmentation in order to pave ways for intimate reading experience in the popular eBook platform. We have designed and developed desktop, mobile/tablet based HE-Book system as well as a semi-automated annotation authoring tool. Our system also supports multimedia based diverse quiz augmentations, which can help in learning tracking. We have conducted quantitative and qualitative tests using the developed prototype systems. We have adopted the indirect objective based performance analysis methodology, which is commonly used for multimedia based learning investigation. The user study shows that, there is a positive tendency of accepting multimodal interactions including haptics with traditional eBook reading experience. Though our limited number of laboratory tests reveal, that haptics can be an influencing media in eBook reading experience, but it requires large scale real life tests to provide a concluding remarks.

[1]  Abu Saleh Md. Mahfujur Rahman,et al.  Bridging the Gap between Virtual and Real World by Bringing an Interpersonal Haptic Communication System in Second Life , 2010, 2010 IEEE International Symposium on Multimedia.

[2]  P. Ekman,et al.  Approaches To Emotion , 1985 .

[3]  Wijnand A. IJsselsteijn,et al.  Mediated social touch: a review of current research and future directions , 2006, Virtual Reality.

[4]  Anselmo Lastra,et al.  Immersive electronic books for surgical training , 2005, IEEE MultiMedia.

[5]  Steven J. DeRose,et al.  Document Structure and Markup in the FRESS Hypertext System , 1999, Markup Lang..

[6]  Anne Mangen,et al.  Hypertext fiction reading: haptics and immersion , 2008 .

[7]  Ed Huai-hsin Chi,et al.  Annotating 3D electronic books , 2005, CHI EA '05.

[8]  Ivan Poupyrev,et al.  The MagicBook - Moving Seamlessly between Reality and Virtuality , 2001, IEEE Computer Graphics and Applications.

[9]  Katie Blake,et al.  The electronic book , 1988 .

[10]  Lorna M. Brown,et al.  Tactile feedback for mobile interactions , 2007, CHI.

[11]  Shafiq ur Réhman,et al.  iFeeling: Vibrotactile Rendering of Human Emotions on Mobile Phones , 2008, WMMP.

[12]  Ronald P. Carver,et al.  Reading Rate: Theory, Research, and Practical Implications. , 1992 .

[13]  R. Likert.  “Technique for the Measurement of Attitudes, A” , 2022, The SAGE Encyclopedia of Research Design.

[14]  Bernardo Magnini,et al.  Integrating Subject Field Codes into WordNet , 2000, LREC.

[15]  J. Turner.  Human Emotions: A Sociological Theory , 2007 .

[16]  Yutaka Ishibashi,et al.  Perception of synchronization errors in haptic and visual communications , 2006, SPIE Optics East.

[17]  Naoki Kawakami,et al.  Affective haptics in emotional communication , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[18]  Jock D. Mackinlay,et al.  3Book: a scalable 3D virtual book , 2004, CHI EA '04.

[19]  Eduardo Casilari-Pérez,et al.  An analytical model for estimating the delay in Bluetooth communications with serial port profile , 2009, IWCMC.

[20]  Abdulmotaleb El Saddik,et al.  Spatial resolution of vibrotactile perception on the human forearm when exploiting funneling illusion , 2009, 2009 IEEE International Workshop on Haptic Audio visual Environments and Games.

[21]  Woontack Woo,et al.  Multi-layer Based Authoring Tool for Digilog Book , 2009, ICEC.

[22]  Chung-Hsien Wu,et al.  Emotion recognition from text using semantic labels and separable mixture models , 2006, TALIP.

[23]  W. G. Parrott,et al.  Emotions in social psychology : essential readings , 2001 .

[24]  Vladimir I. Levenshtein,et al.  Binary codes capable of correcting deletions, insertions, and reversals , 1965 .

[25]  Anton Nijholt,et al.  A Lexical Grammatical Implementation of Affect , 2004, TSD.

[26]  Mitsuru Ishizuka,et al.  EmoHeart: Conveying Emotions in Second Life Based on Affect Sensing from Text , 2010, Adv. Hum. Comput. Interact..

[27]  M. Weiser,et al.  Hot topics-ubiquitous computing , 1993 .

[28]  Paul Lemmens,et al.  A body-conforming tactile jacket to enrich movie viewing , 2009, World Haptics 2009 - Third Joint EuroHaptics conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.

[29]  Abdulmotaleb El Saddik,et al.  Haptics Technologies: Bringing Touch to Multimedia , 2011 .

[30]  Federico Casalegno,et al.  The Three Dimensions of Book Evolution in Ubiquitous Computing Age: Digitalization, Augmentation, and Hypermediation , 2010, 2010 IEEE International Conference on Sensor Networks, Ubiquitous, and Trustworthy Computing.

[31]  M. F. Suarez,et al.  The Oxford Companion to the Book , 2010 .

[32]  Paul W. F. Witt.  Audio-visual methods in teaching , 1954 .

[33]  Nicholas Chen,et al.  Navigation techniques for dual-display e-book readers , 2008, CHI.

[34]  B. L. Harrison,et al.  E-books and the future of reading , 2000 .

[35]  Abdulmotaleb El-Saddik,et al.  HugMe: A haptic videoconferencing system for interpersonal communication , 2008, 2008 IEEE Conference on Virtual Environments, Human-Computer Interfaces and Measurement Systems.

[36]  A. Ortony,et al.  What's basic about basic emotions? , 1990, Psychological review.

[37]  Christopher O. Jaynes,et al.  The universal media book: tracking and augmenting moving surfaces with projected information , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[38]  R. Plutchik.  A GENERAL PSYCHOEVOLUTIONARY THEORY OF EMOTION , 1980 .

[39]  Taezoon Park,et al.  Facilitating the Design of Vibration for Handheld Devices , 2009, HCI.

[40]  George G. Robertson,et al.  The WebBook and the Web Forager: an information workspace for the World-Wide Web , 1996, CHI.

[41]  Christopher D. Manning,et al.  Enriching the Knowledge Sources Used in a Maximum Entropy Part-of-Speech Tagger , 2000, EMNLP.

[42]  Blake Hannaford,et al.  Architectures for shared haptic virtual environments , 1997, Comput. Graph..

[43]  Abdulmotaleb El Saddik,et al.  Adding emotional tag to augment context-awareness in social network services , 2011, 2011 IEEE International Instrumentation and Measurement Technology Conference.

[44]  Carlo Strapparava,et al.  WordNet Affect: an Affective Extension of WordNet , 2004, LREC.

[45]  Mark Billinghurst,et al.  The design of a mixed-reality book: Is it still a real book? , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[46]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[47]  Michael Koch,et al.  Ubiquitous Computing , 2001, CSCW-Kompendium.

[48]  Andries van Dam,et al.  Computers and Publishing: Writing, Editing, and Printing , 1970, Adv. Comput..

[49]  Stephen A. Brewster,et al.  Investigating the effectiveness of tactile feedback for mobile touchscreens , 2008, CHI.

[50]  Gitta Salomon,et al.  Articulating a metaphor through user-centered design , 1995, CHI '95.

[51]  D. Leutner,et al.  Direct Measurement of Cognitive Load in Multimedia Learning , 2003 .

[52]  Austin Henderson,et al.  CATALOGUES: A METAPHOR FOR COMPUTER APPLICATION DELIVERY , 1987 .

[53]  Shogo Nishida,et al.  Virtual Pop-Up Book Based on Augmented Reality , 2007, HCI.