Hyperion: A Wearable Augmented Reality System for Text Extraction and Manipulation in the Air

We develop Hyperion a Wearable Augmented Reality (WAR) system based on Google Glass to access text information in the ambient environment. Hyperion is able to retrieve text content from users' current view and deliver the content to them in different ways according to their context. We design four work modalities for different situations that mobile users encounter in their daily activities. In addition, user interaction interfaces are provided to adapt to different application scenarios. Although Google Glass may be constrained by its poor computational capabilities and its limited battery capacity, we utilize code-level offloading to companion mobile devices to improve the runtime performance and the sustainability of WAR applications. System experiments show that Hyperion improves users ability to be aware of text information around them. Our prototype indicates promising potential of converging WAR technology and wearable devices such as Google Glass to improve people's daily activities.

[1]  Pan Hui,et al.  Future Networking Challenges: The Case of Mobile Augmented Reality , 2017, 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS).

[2]  Pan Hui,et al.  ReadMe: A Real-Time Recommendation System for Mobile Augmented Reality Ecosystems , 2016, ACM Multimedia.

[3]  A. Panayiotou,et al.  RGB Slemmings: An augmented reality game in your room , 2016 .

[4]  Gang Hu,et al.  Exploring Viewable Angle Information in Georeferenced Video Search , 2015, ACM Multimedia.

[5]  Pan Hui,et al.  Ubii: Towards Seamless Interaction between Digital and Physical Worlds , 2015, ACM Multimedia.

[6]  Zi Huang,et al.  Joint Modeling of Users' Interests and Mobility Patterns for Point-of-Interest Recommendation , 2015, ACM Multimedia.

[7]  Hugo Fuks,et al.  Landmark identification with wearables for supporting spatial awareness by blind persons , 2015, UbiComp.

[8]  Lorenzo Porzi,et al.  i-Street: Detection, Identification, Augmentation of Street Plates in a Touristic Mobile Application , 2015, ICIAP.

[9]  Tao Yang,et al.  Eye-Wearable Technology for Machine Maintenance: Effects of Display Position and Hands-free Operation , 2015, CHI.

[10]  Mahadev Satyanarayanan,et al.  Cloudlets: at the leading edge of mobile-cloud convergence , 2014, 6th International Conference on Mobile Computing, Applications and Services.

[11]  Mohammed E. Hoque,et al.  A google glass app to help the blind in small talk , 2014, ASSETS.

[12]  Ben D. Sawyer,et al.  Google Glass , 2014, Hum. Factors.

[13]  Nadir Weibel,et al.  Chroma: a wearable augmented-reality solution for color blindness , 2014, UbiComp.

[14]  Mohammed Yeasin,et al.  Expression: a dyadic conversation aid using Google Glass for people with visual impairments , 2014, UbiComp Adjunct.

[15]  Shwetak N. Patel,et al.  AirLink: sharing files between multiple devices using in-air gestures , 2014, UbiComp.

[16]  Swati Rallapalli,et al.  Enabling physical analytics in retail stores using smart glasses , 2014, MobiCom.

[17]  Pan Hui,et al.  CloudRidAR: a cloud-based architecture for mobile augmented reality , 2014, MARS@MobiSys.

[18]  Justin Manweiler,et al.  Cloud-based mobile augmented reality, pitfalls and strategies in a realtime deployment , 2014, MARS@MobiSys.

[19]  Mahadev Satyanarayanan,et al.  Towards wearable cognitive assistance , 2014, MobiSys.

[20]  Mark Billinghurst,et al.  The glass class: Designing wearable interfaces , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[21]  Samia Bouzefrane,et al.  Cloudlets Authentication in NFC-Based Mobile Computing , 2014, 2014 2nd IEEE International Conference on Mobile Cloud Computing, Services, and Engineering.

[22]  Pan Hui,et al.  Mobile augmented reality survey: a bottom-up approach , 2013, ArXiv.

[23]  John Krogstie,et al.  Mobile augmented reality for cultural heritage: A technology acceptance study , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[24]  Michel Denis,et al.  NAVIG: augmented reality guidance system for the visually impaired , 2012, Virtual Reality.

[25]  Tim Verbelen,et al.  Cloudlets: bringing the cloud to the mobile user , 2012, MCS '12.

[26]  Pan Hui,et al.  ThinkAir: Dynamic resource allocation and parallel execution in the cloud for mobile code offloading , 2012, 2012 Proceedings IEEE INFOCOM.

[27]  Kai Wang,et al.  End-to-end scene text recognition , 2011, 2011 International Conference on Computer Vision.

[28]  Hiroaki Kimura,et al.  System support for mobile augmented reality services , 2007, SAC '07.

[29]  Tudor Dumitras,et al.  Eye of the Beholder: Phone-Based Text-Recognition for the Visually-Impaired , 2006, 2006 10th IEEE International Symposium on Wearable Computers.

[30]  Lambert Schomaker,et al.  Text detection from natural scene images: towards a system for visually impaired persons , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[31]  Alan L. Yuille,et al.  Detecting and reading text in natural scenes , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[32]  D Whitaker,et al.  Reliability of the Snellen chart , 1995, BMJ.

[33]  Tong Zhang,et al.  Mobile Camera Based Text Detection and Translation , 2011 .

[34]  김만두,et al.  시력 손상과 시각 장애(Visual Impairment and Blindness) , 2011 .

[35]  Dahe.  The Color Picker , 2008 .

[36]  Robyn Opie,et al.  A magnifying glass , 2004 .