AR 2.0: Social Media in Mobile Augmented Reality

The goal of this thesis is to design, implement and evaluate novel techniques for integrating social media into handheld Augmented Reality (AR) applications. Augmenting the physical environment with information is an idea prospected for over twenty years within the AR research community. The availability of mobile phones as an affordable platform for mobile AR applications lead for the first time to a wide awareness of the public. For instance, AR browsers, applications that allow users to browse information registered to places or objects in the real world, are now installed on several millions of phones. Despite the fact that AR browsers have arrived in the mass market, we argue that they are solely used for browsing information while at the same time lacking in amount and variety of information that can be accessed. This is caused by the current authoring tools aiming at professionals and do not permit a spontaneous creation of content in place, while also focussing on rather simple content forms. In the first part of this thesis, we want to shed light on the adoption of AR browsers to the public, by presenting results of two conducted studies. The results are later used for developing the concept of Augmented Reality 2.0 as the next evolution of Augmented Reality. Augmented Realities 2.0 utmost concern is to not only see users of AR applications as consumers, but also as a producers of content (”Prosumer”). In preparations for our implemented prototypes, we present fundamental technologies such as a precise tracking for handheld devices and an AR framework that can be used to implement AR 2.0 applications. In the second part of this thesis, we present novel prototypes integrating social media in various forms into mobile AR applications. Lastly, an assessment of these prototypes is done based on user feedback and observations made through user studies. This thesis extends the current understanding on the adoption of AR browsers and gives requirements helping developers to make informed design decision when building next generation AR browsers relying on user-generated content.

[1]  Mark Billinghurst,et al.  Face to face collaborative AR on mobile phones , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[2]  Jeremy R. Cooperstock,et al.  Audio Graffiti: A Location Based Audio-Tagging and Remixing Environment , 2009, ICMC.

[3]  Vincent Lepetit,et al.  Keyframe-based modeling and tracking of multiple 3D objects , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[4]  Pished Bunnun,et al.  OutlinAR: an assisted interactive model building system with reduced computational effort , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[5]  Suya You,et al.  Real-Time Object Tracking for Augmented Reality Combining Graph Cuts and Optical Flow , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[6]  Sean J. Barbeau,et al.  Positional Accuracy of Assisted GPS Data from High-Sensitivity GPS-enabled Mobile Phones , 2011, Journal of Navigation.

[7]  Harald Ganster,et al.  Hybrid Tracking for Outdoor Augmented Reality Applications , 2002, IEEE Computer Graphics and Applications.

[8]  Hideyuki Tamura,et al.  A hybrid registration method for outdoor augmented reality , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[9]  Anthony Steed,et al.  Interactive modelling and tracking for mixed and augmented reality , 2006, VRST '06.

[10]  Dieter Schmalstieg,et al.  Mobile Phones as a Platform for Augmented Reality , 2008 .

[11]  Steven K. Feiner,et al.  Exploring MARS: developing indoor and outdoor user interfaces to a mobile augmented reality system , 1999, Comput. Graph..

[12]  Charlotte Magnusson,et al.  Pointing for non-visual orientation and navigation , 2010, NordiCHI.

[13]  Volker Coors,et al.  Meeting the spirit of history , 2001, VAST '01.

[14]  Wendell Bell,et al.  The Third Wave. , 1982 .

[15]  Jun Rekimoto,et al.  Augment-able reality: situated communication through physical and digital spaces , 1998, Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215).

[16]  Selim Benhimane,et al.  Gravity-aware handheld Augmented Reality , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[17]  Pished Bunnun,et al.  In-Situ interactive image-based model building for Augmented Reality from a handheld device , 2013, Virtual Reality.

[18]  R. Ramakrishnan,et al.  Crowdsourcing systems on the World-Wide Web , 2011, Commun. ACM.

[19]  Dieter Schmalstieg,et al.  The Studierstube Augmented Reality Project , 2002, Presence: Teleoperators & Virtual Environments.

[20]  Dieter Schmalstieg,et al.  Augmented Reality 2.0 , 2008, Virtual Realities.

[21]  Mark Billinghurst,et al.  Using a mobile phone for 6 DOF mesh editing , 2007, CHINZ.

[22]  Venkataraman Sundareswaran,et al.  A novel interface for device diagnostics using speech recognition, augmented reality visualization, and 3D audio auralization , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[23]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Ivan E. Sutherland,et al.  A head-mounted three dimensional display , 1968, AFIPS Fall Joint Computing Conference.

[25]  Steven K. Feiner,et al.  Authoring 3D hypermedia for wearable augmented and virtual reality , 2003, Seventh IEEE International Symposium on Wearable Computers, 2003. Proceedings..

[26]  P. Beardsley,et al.  iLamps: geometrically aware and self-configuring projectors , 2003, SIGGRAPH Courses.

[27]  Berthold K. P. Horn,et al.  Closed-form solution of absolute orientation using unit quaternions , 1987 .

[28]  Jennifer Healey,et al.  Augmented Reality through Wearable Computing , 1997, Presence: Teleoperators & Virtual Environments.

[29]  Wei Liu,et al.  Human Pacman: a mobile wide-area entertainment system based on physical, social, and ubiquitous computing , 2004, ACE '04.

[30]  Blair MacIntyre,et al.  Browsing the Real-World Wide Web: Maintaining Awareness of Virtual Information in an AR Information Space , 2003, Int. J. Hum. Comput. Interact..

[31]  Gerhard Reitmayr,et al.  North-centred orientation tracking on mobile phones , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[32]  Mihran Tuceryan,et al.  Automatic determination of text readability over textured backgrounds for augmented reality systems , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[33]  Uwe D. Hanebeck,et al.  Template matching using fast normalized cross correlation , 2001, SPIE Defense + Commercial Sensing.

[34]  Holger Regenbrecht,et al.  A pragmatic approach to augmented reality authoring , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[35]  Mads Haahr,et al.  Design, implementation and evaluation of audio for a location aware augmented reality game , 2010, Fun and Games '10.

[36]  Tom Drummond,et al.  Rapid 3D modelling from live video , 2010, The 33rd International Convention MIPRO.

[37]  Steven Feiner,et al.  Authoring and presenting situated media in augmented and virtual reality , 2006 .

[38]  G. Ritzer,et al.  Production, Consumption, Prosumption , 2010 .

[39]  Marc Pollefeys,et al.  Unstructured video-based rendering: interactive exploration of casually captured videos , 2010, SIGGRAPH 2010.

[40]  Jukka Häkkinen,et al.  Examining mobile phone text legibility while walking , 2004, CHI EA '04.

[41]  Jun Rekimoto,et al.  Matrix: a realtime object identification and registration method for augmented reality , 1998, Proceedings. 3rd Asia Pacific Computer Human Interaction (Cat. No.98EX110).

[42]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[43]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[44]  Andrew J. Davison,et al.  Live dense reconstruction with a single moving camera , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[45]  T. P. Caudell,et al.  Augmented reality: an application of heads-up display technology to manual manufacturing processes , 1992, Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences.

[46]  Steven K. Feiner,et al.  Situated documentaries: embedding multimedia presentations in the real world , 1999, Digest of Papers. Third International Symposium on Wearable Computers.

[47]  Blair MacIntyre,et al.  Ghosts in the Machine: Integrating 2D Video Actors into a 3D AR System , 2001 .

[48]  Dieter Schmalstieg,et al.  Mobile collaborative augmented reality , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[49]  G. Klein,et al.  Parallel Tracking and Mapping for Small AR Workspaces , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[50]  Holger Regenbrecht,et al.  Audio stickies: visually-guided spatial audio annotations on a mobile augmented reality platform , 2013, OZCHI.

[51]  Thomas Olsson,et al.  Online user survey on current mobile augmented reality applications , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[52]  Anton van den Hengel,et al.  In situ image-based modeling , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[53]  Dieter Schmalstieg,et al.  Exploiting sensors on mobile phones to improve wide-area localization , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[54]  Stephen DiVerdi,et al.  Using aerial photographs for improved mobile AR annotation , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[55]  George W. Fitzmaurice,et al.  Situated information spaces and spatially aware palmtop computers , 1993, CACM.

[56]  Stephen DiVerdi,et al.  Annotation in outdoor augmented reality , 2009, Comput. Graph..

[57]  Dieter Schmalstieg,et al.  Online Creation of Panoramic Augmented Reality Annotations on Mobile Phones , 2012, IEEE Pervasive Computing.

[58]  Dieter Schmalstieg,et al.  Real-time panoramic mapping and tracking on mobile phones , 2010, 2010 IEEE Virtual Reality Conference (VR).

[59]  Tim O'Reilly,et al.  What is Web 2.0: Design Patterns and Business Models for the Next Generation of Software , 2007 .

[60]  Dieter Schmalstieg,et al.  Zooming interfaces for augmented reality browsers , 2010, Mobile HCI.

[61]  Suya You,et al.  A robust hybrid tracking system for outdoor augmented reality , 2004, IEEE Virtual Reality 2004.

[62]  Dima Damen,et al.  Integrating 3D object detection, modelling and tracking on a mobile phone , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[63]  Blair MacIntyre,et al.  Integrating 2-D Video Actors into 3-D Augmented-Reality Systems , 2002, Presence: Teleoperators & Virtual Environments.

[64]  Mark Billinghurst,et al.  A survey of evaluation techniques used in augmented reality studies , 2008, SIGGRAPH 2008.

[65]  Walterio W. Mayol-Cuevas,et al.  Ninja on a Plane: Automatic Discovery of Physical Planes for Augmented Reality Using Visual SLAM , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[66]  Michael Rohs,et al.  USING CAMERA-EQUIPPED MOBILE PHONES FOR INTERACTING WITH REAL-WORLD OBJECTS , 2004 .

[67]  Stephen DiVerdi,et al.  Implicit 3D modeling and tracking for anywhere augmentation , 2007, VRST '07.

[68]  Lorenz Rogge,et al.  Space-time visual effects as a post-production process , 2010, 3DVP '10.

[69]  Holger Regenbrecht,et al.  A mobile Passive Augmented Reality Device - mPARD , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[70]  Simon J. Julier,et al.  Authoring of physical models using mobile computers , 2001, Proceedings Fifth International Symposium on Wearable Computers.

[71]  David W. Murray,et al.  Parallel Tracking and Mapping on a camera phone , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[72]  Stephen A. Brewster,et al.  Audio Bubbles: Employing Non-speech Audio to Support Tourist Wayfinding , 2009, HAID.

[73]  Christian Knöpfle,et al.  Template based authoring for AR based service scenarios , 2005, IEEE Proceedings. VR 2005. Virtual Reality, 2005..

[74]  Selim Benhimane,et al.  Inertial sensor-aligned visual feature descriptors , 2011, CVPR 2011.

[75]  Xin Chen,et al.  City-scale landmark identification on mobile devices , 2011, CVPR 2011.

[76]  Fredrik Espinoza,et al.  GeoNotes: Social and Navigational Aspects of Location-Based Information Systems , 2001, UbiComp.

[77]  Bruce H. Thomas,et al.  A wearable computer system with augmented reality to support terrestrial navigation , 1998, Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215).

[78]  Anton van den Hengel,et al.  Interactive modelling for AR applications , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[79]  Dieter Schmalstieg,et al.  Global pose estimation using multi-sensor fusion for outdoor Augmented Reality , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[80]  Michael Grafe,et al.  The Augmented Reality Personal Digital Assistant , 2001 .

[81]  Jan-Olof Eklundh,et al.  Fast Block Matching with Normalized Cross-Correlation using Walsh Transforms , 2002 .

[82]  Adrian David Cheok,et al.  22nd International Conference on Human-Computer Interaction with Mobile Devices and Services , 2007, Lecture Notes in Computer Science.

[83]  Dieter Schmalstieg,et al.  Pose tracking from natural features on mobile phones , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[84]  Stephen DiVerdi,et al.  "Anywhere Augmentation": Towards Mobile Augmented Reality in Unprepared Environments , 2007, Location Based Services and TeleCartography.

[85]  Blair MacIntyre,et al.  Three Angry Men: An Augmented-Reality Experiment in Point-of-View Drama , 2003 .

[86]  Bruce H. Thomas,et al.  The Tinmith System - Demonstrating New Techniques for Mobile Augmented Reality Modelling , 2002, AUIC.

[87]  Ian D. Reid,et al.  Single View Metrology , 2000, International Journal of Computer Vision.

[88]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[89]  Tom Drummond,et al.  Real-Time Video Annotations for Augmented Reality , 2005, ISVC.

[90]  Steven K. Feiner,et al.  Information at a Glance , 2002, IEEE Computer Graphics and Applications.

[91]  Tobias Höllerer,et al.  Pictorial depth cues for outdoor augmented reality , 2005, Ninth IEEE International Symposium on Wearable Computers (ISWC'05).

[92]  Bruce H. Thomas,et al.  Augmented reality user interfaces and techniques for outdoor modelling , 2003, I3D '03.

[93]  Katashi Nagao,et al.  The world through the computer: computer augmented interaction with real world environments , 1995, UIST '95.

[94]  Roberta L. Klatzky,et al.  Personal guidance system for the visually impaired , 1994, ASSETS.

[95]  Hirokazu Kato,et al.  3D live: real time captured content for mixed reality , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[96]  Dieter Schmalstieg,et al.  Sketching up the world: in situ authoring for mobile Augmented Reality , 2012, Personal and Ubiquitous Computing.

[97]  Shun'ichi Tano,et al.  Multimedia Informal Communication by Wearable Computer based on Real-World Context and Graffiti , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[98]  Simon J. Julier,et al.  BARS: Battlefield Augmented Reality System , 2001 .

[99]  Tom Drummond,et al.  Going out: robust model-based tracking for outdoor augmented reality , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[100]  Steven K. Feiner,et al.  BRIDGING THE GAPS: HYBRID TRACKING FOR ADAPTIVE MOBILE AUGMENTED REALITY , 2004, Appl. Artif. Intell..

[101]  M. Haller,et al.  AMIRE-ES: Authoring Mixed Reality once, run it anywhere , 2005 .

[102]  David W. Murray,et al.  Improving the Agility of Keyframe-Based SLAM , 2008, ECCV.

[103]  Steven M. Seitz,et al.  Photo tourism: exploring photo collections in 3D , 2006, ACM Trans. Graph..

[104]  David Ingram,et al.  Augmented reality in a wide area sentient environment , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[105]  Dieter Schmalstieg,et al.  Image-driven view management for augmented reality browsers , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[106]  Dieter Schmalstieg,et al.  First steps towards handheld augmented reality , 2003, Seventh IEEE International Symposium on Wearable Computers, 2003. Proceedings..

[107]  James R. Lewis,et al.  IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use , 1995, Int. J. Hum. Comput. Interact..

[108]  A. Strauss,et al.  Basics of Qualitative Research , 1992 .

[109]  Venkataraman Sundareswaran,et al.  3D audio augmented reality: implementation and experiments , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[110]  Ian Oakley,et al.  Auditory display design for exploration in mobile audio-augmented reality , 2012, Personal and Ubiquitous Computing.

[111]  Mark Billinghurst,et al.  Mobile phone based AR scene assembly , 2005, MUM '05.

[112]  A. Kaplan,et al.  Users of the world, unite! The challenges and opportunities of Social Media , 2010 .

[113]  James C. Spohrer,et al.  Information in Places , 1999, IBM Syst. J..

[114]  Dieter Schmalstieg,et al.  Handheld Augmented Reality Displays , 2006, IEEE Virtual Reality Conference (VR 2006).

[115]  Blair MacIntyre,et al.  Exploring spatial narratives and mixed reality experiences in Oakland Cemetery , 2005, ACE '05.

[116]  Michael Haller,et al.  Augmenting the reality with 3D sound sources , 2002, SIGGRAPH '02.

[117]  Jens Grubert,et al.  Augmented reality browsers: essential products or only gadgets? , 2013, CACM.

[118]  Matti Karjalainen,et al.  Localization of Amplitude-Panned Virtual Sources I: Stereophonic Panning , 2001 .

[119]  Gilles Simon In-Situ 3D Sketching Using a Video Camera as an Interaction and Tracking Device , 2010, Eurographics.

[120]  Dieter Schmalstieg,et al.  ARToolKitPlus for Pose Trackin on Mobile Devices , 2007 .

[121]  Jens Grubert,et al.  Augmented Reality Browser Survey , 2012 .

[122]  Bruce H. Thomas,et al.  Tinmith-Metro: new outdoor techniques for creating city models with an augmented reality wearable computer , 2001, Proceedings Fifth International Symposium on Wearable Computers.

[123]  Oliver Bimber,et al.  Video see-through AR on consumer cell-phones , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[124]  Hirokazu Kato,et al.  Marker tracking and HMD calibration for a video-based augmented reality conferencing system , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[125]  Steven K. Feiner,et al.  A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment , 1997, Digest of Papers. First International Symposium on Wearable Computers.

[126]  Tom Drummond,et al.  Semi-automatic Annotations in Unknown Environments , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[127]  Tom Drummond,et al.  Machine Learning for High-Speed Corner Detection , 2006, ECCV.

[128]  Dieter Schmalstieg,et al.  Robust and unobtrusive marker tracking on mobile phones , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[129]  Blair MacIntyre,et al.  The Argon AR Web Browser and standards-based AR application environment , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[130]  Ian Oakley,et al.  Urban Sound Gardens: Supporting Overlapping Audio Landmarks in Exploratory Environments , 2010 .

[131]  Johannes Schöning,et al.  Falling asleep with Angry Birds, Facebook and Kindle: a large scale study on mobile application usage , 2011, Mobile HCI.

[132]  Ben Shneiderman,et al.  Visualization methods for personal photo collections: browsing and searching in the PhotoFinder , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[133]  Ronald Azuma,et al.  Tracking requirements for augmented reality , 1993, CACM.

[134]  Tobias Höllerer,et al.  Fast annotation and modeling with a single-point laser range finder , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[135]  Ronald Azuma,et al.  Orientation Tracking for Outdoor Augmented Reality Registration , 1999, IEEE Computer Graphics and Applications.

[136]  Sagaya Aurelia,et al.  Mobile Augmented Reality and Location Based Service , 2014 .

[137]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[138]  Mor Naaman,et al.  From Where to What: Metadata Sharing for Digital Photographs with Geographic Coordinates , 2003, OTM.

[139]  Stephen DiVerdi,et al.  Envisor: Online Environment Map Construction for Mixed Reality , 2008, 2008 IEEE Virtual Reality Conference.

[140]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[141]  Bruce H. Thomas,et al.  ARQuake: an outdoor/indoor augmented reality first person application , 2000, Digest of Papers. Fourth International Symposium on Wearable Computers.

[142]  Yun-Ta Tsai,et al.  Indirect augmented reality , 2011, Comput. Graph..

[143]  Frank Allan Hansen,et al.  Ubiquitous annotation systems: technologies and challenges , 2006, HYPERTEXT '06.

[144]  Dieter Schmalstieg,et al.  Real-time self-localization from panoramic images on mobile devices , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[145]  Dieter Schmalstieg,et al.  Robust detection and tracking of annotations for outdoor augmented reality browsing , 2011, Comput. Graph..

[146]  Enrico Rukzio,et al.  Investigating selection and reading performance on a mobile phone while walking , 2010, Mobile HCI.