Augmenting natural interaction with physical paper in ambient intelligence environments

Physical paper, in its various forms (e.g. books, leaflets, catalogues), is extensively used in everyday activities, despite any advancements in digital technology and ICT. Inspired by the popularity of this medium, several research efforts have envisioned and pursued a new era of interactive paper, however several challenges still remain to be addressed. On the other hand, recent advancements towards Ambient Intelligence (AmI) and Smart environments, bear the promise of seamless integration of the physical and digital worlds in an intuitive and user-friendly manner. This paper presents InPrinted, a systematic and generic framework supporting physical paper augmentation and user interaction in Ambient intelligence environments. InPrinted enables natural multimodal user interaction with any kind of printed matter in smart environments, providing context aware and anticipation mechanisms, as well as tools and interaction techniques that support the development of applications incorporating printed matter augmentation. InPrinted has been put to practice in the development of various systems, including the Interactive Maps system, facilitating interaction with printed maps as well as their augmentation with digital information. The results of an in-situ observation experiment of the Interactive Maps system are reported, highlighting that interacting with augmented paper is quite easy and natural, while the overall User Experience is positive.

[1]  Tara Matthews,et al.  Defining, Designing, and Evaluating Peripheral Displays: An Analysis Using Activity Theory , 2007, Hum. Comput. Interact..

[2]  Nikolaus F. Troje,et al.  Paper windows: interaction techniques for digital paper , 2005, CHI.

[3]  Lynn Wilcox,et al.  Pacer: fine-grained interactive paper via camera-touch hybrid gestures on a cell phone , 2010, CHI.

[4]  Ivan Poupyrev,et al.  The MagicBook - Moving Seamlessly between Reality and Virtuality , 2001, IEEE Computer Graphics and Applications.

[5]  Beat Signer,et al.  Interactive Paper: Past, Present and Future , 2010 .

[6]  Jan Stage,et al.  It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field , 2006, NordiCHI '06.

[7]  Raimund Dachselt Interacting with Printed Books Using Digital Pens and Smart Mobile Projection , 2011 .

[8]  Hideki Koike,et al.  EnhancedDesk: integrating paper documents and digital documents , 1998, Proceedings. 3rd Asia Pacific Computer Human Interaction (Cat. No.98EX110).

[9]  Constantine Stephanidis,et al.  A smart environment for augmented learning through physical books , 2011, 2011 IEEE International Conference on Multimedia and Expo.

[10]  Constantine Stephanidis,et al.  The book of Ellie: An interactive book for teaching the alphabet to children , 2013, 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW).

[11]  Wendy E. Mackay,et al.  Designing interactive paper: lessons from three augmented reality projects , 1998 .

[12]  Michael R. Lyu,et al.  ART: augmented reality table for interactive trading card game , 2006, VRCIA '06.

[13]  Akrivi Katifori,et al.  Creating an Ontology for the User Profile: Method and Applications , 2007, RCIS.

[14]  Constantine Stephanidis,et al.  Study-Buddy: Improving the Learning Process through Technology-Augmented Studying Environments , 2011, HCI.

[15]  Mary Beth Rosson,et al.  Scenario-based design , 2002 .

[16]  Mitsuru Ishizuka,et al.  Keyword extraction from a single document using word co-occurrence statistical information , 2004, Int. J. Artif. Intell. Tools.

[17]  B. L. William Wong,et al.  In Use, In Situ: Extending Field Research Methods , 2007, Int. J. Hum. Comput. Interact..

[18]  Yvonne Rogers,et al.  Why It's Worth the Hassle: The Value of In-Situ Studies When Designing Ubicomp , 2007, UbiComp.

[19]  Harry Chen,et al.  An ontology for context-aware pervasive computing environments , 2003, The Knowledge Engineering Review.

[20]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[21]  Pierre David Wellner,et al.  Interacting with paper on the DigitalDesk , 1993, CACM.

[22]  X. Zabulis,et al.  Augmenting physical books towards education enhancement , 2013, 2013 1st IEEE Workshop on User-Centered Computer Vision (UCCV).

[23]  Beat Signer,et al.  PaperPoint: a paper-based presentation and interactive paper prototyping tool , 2007, Tangible and Embedded Interaction.

[24]  Ioannis Georgalis,et al.  Architectures, methods and tools for creating ambient intelligence environments , 2013 .

[25]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[26]  R. Smith,et al.  An Overview of the Tesseract OCR Engine , 2007, Ninth International Conference on Document Analysis and Recognition (ICDAR 2007).

[27]  John Robinson,et al.  Live paper: video augmentation to simulate interactive paper , 1999, MULTIMEDIA '99.

[28]  Mark Billinghurst,et al.  A Survey of Augmented Reality , 2015, Found. Trends Hum. Comput. Interact..

[29]  Constantine Stephanidis,et al.  Augmented interaction with physical books in an Ambient Intelligence learning environment , 2013, Multimedia Tools and Applications.

[30]  Claudio S. Pinhanez Using a steerable projector and a camera to transform surfaces into interactive displays , 2001, CHI Extended Abstracts.

[31]  Catherine C. Marshall,et al.  Reading and Interactivity in the Digital Library: Creating an experience that transcends paper , 2003 .

[32]  Tovi Grossman,et al.  PenLight: combining a mobile projector and a digital pen for dynamic visual overlay , 2009, CHI.

[33]  Joseph J. LaViola,et al.  ErgoDesk: A Framework for Two- and Three-Dimensional Interaction at the ActiveDesk , 1998 .

[34]  Koen De Bosschere,et al.  Towards an Extensible Context Ontology for Ambient Intelligence , 2004, EUSAI.

[35]  Constantine Stephanidis,et al.  Enhancing education through natural interaction with physical paper , 2014, Universal Access in the Information Society.

[36]  Dieter Schmalstieg,et al.  The Studierstube Augmented Reality Project , 2002, Presence: Teleoperators & Virtual Environments.

[37]  Mark Billinghurst,et al.  Tangible tiles: design and evaluation of a tangible user interface in a collaborative tabletop setup , 2006, OZCHI.

[38]  Constantine Stephanidis,et al.  Interacting with augmented paper maps: a user experience study , 2017, CHItaly.

[39]  Nelson Zagalo,et al.  Bridging book: a not-so-electronic children's picturebook , 2013, IDC.

[40]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[41]  Beat Signer,et al.  Only touching the surface: creating affinities between digital content and paper , 2004, CSCW.

[42]  Abigail Sellen,et al.  The myth of the paperless office , 2001 .

[43]  Mark Billinghurst,et al.  Augmenting the science centre and museum experience , 2004, GRAPHITE '04.

[44]  Constantine Stephanidis,et al.  Ambient educational mini-games , 2012, AVI.

[45]  Ali Mazalek,et al.  Turning a page on the digital annotation of physical books , 2008, Tangible and Embedded Interaction.

[46]  Timo Ojala,et al.  It's a Jungle Out There: Fantasy and Reality of Evaluating Public Displays "in the Wild" , 2011 .

[47]  Claudio S. Pinhanez,et al.  Interacting with steerable projected displays , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[48]  Abigail Sellen,et al.  The myth of the paperless office mit press , 2003 .

[49]  Niels Henze,et al.  Analysis of children's handwriting on touchscreen phones , 2013, MobileHCI '13.

[50]  Andrew Wilson,et al.  MirageTable: freehand interaction on a projected augmented reality tabletop , 2012, CHI.

[51]  James D. Hollan,et al.  Papiercraft: A gesture-based command system for interactive paper , 2008, TCHI.