A system for supporting paper-based augmented reality

In this paper, we aim to implement augmented reality (AR) on distant text documents or books. For this purpose, we propose a new paper-based AR system that can detect text documents in real scenes, markerize and identify them, estimate their relative 3D poses to the camera, and augment them with virtual contents. Unlike the previous paper-based AR systems (applicable to only close documents), the proposed system not only requires no detection of words or characters, but allows partial occlusions like the previous systems. In our experiments, the proposed system worked at 24 fps and could consistently achieve high identification rates for both occluded and unoccluded pages.

[1]  Hanchuan Peng,et al.  Document Image Recognition Based on Template Matching of Component Block Projections , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Shiliang Zhang,et al.  USB: Ultrashort Binary Descriptor for Fast Visual Matching and Retrieval , 2014, IEEE Transactions on Image Processing.

[3]  Scott L. Minneman,et al.  Listen reader: an electronically augmented paper-based book , 2001, CHI.

[4]  Hirokazu Kato,et al.  Marker tracking and HMD calibration for a video-based augmented reality conferencing system , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[5]  Hirokazu Kato,et al.  A registration method based on texture tracking using ARToolKit , 2003, 2003 IEEE International Augmented Reality Toolkit Workshop.

[6]  Dieter Schmalstieg,et al.  Robust and unobtrusive marker tracking on mobile phones , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[7]  Vincent Lepetit,et al.  The haunted book , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[8]  Andreas Dünser,et al.  An interactive augmented reality coloring book , 2012, 3DUI.

[9]  Matthew Turk,et al.  TranslatAR: A mobile augmented reality translator , 2011, 2011 IEEE Workshop on Applications of Computer Vision (WACV).

[10]  Ivan Poupyrev,et al.  The MagicBook - Moving Seamlessly between Reality and Virtuality , 2001, IEEE Computer Graphics and Applications.

[11]  Jong-Il Park,et al.  Invisible marker tracking for AR , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[12]  Ning Li.  An Implementation of OCR System Based on Skeleton Matching , 1993 .

[13]  Mark Fiala,et al.  ARTag, a fiducial marker system using digital techniques , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[14]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[15]  Jiřı́ Matas,et al.  Real-time scene text localization and recognition , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Vincent Lepetit,et al.  Scalable real-time planar targets tracking for digilog books , 2010, The Visual Computer.

[17]  Woontack Woo,et al.  Hybrid Document Matching Method for Page Identification of Digilog Books , 2011, Trans. Edutainment.

[18]  Beat Signer,et al.  Content publishing framework for interactive paper documents , 2005, DocEng '05.

[19]  Masakazu Iwamura,et al.  Camera Based Document Image Retrieval with More Time and Memory Efficient LLAH , 2008 .

[20]  Richard O. Duda,et al.  Use of the Hough transformation to detect lines and curves in pictures , 1972, CACM.

[21]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[22]  Charles Baur,et al.  Automatic text detection for mobile augmented reality translation , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[23]  Dimosthenis Karatzas,et al.  Multi-script Text Extraction from Natural Scenes , 2013, 2013 12th International Conference on Document Analysis and Recognition.

[24]  Dieter Schmalstieg,et al.  Pose tracking from natural features on mobile phones , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[25]  Pierre Vandergheynst,et al.  FREAK: Fast Retina Keypoint , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Moshe Mahler,et al.  HideOut: mobile projector interaction with tangible objects and surfaces , 2013, TEI '13.

[27]  Tobias Höllerer,et al.  Evaluation of Interest Point Detectors and Feature Descriptors for Visual Tracking , 2011, International Journal of Computer Vision.

[28]  Roland Siegwart,et al.  BRISK: Binary Robust invariant scalable keypoints , 2011, 2011 International Conference on Computer Vision.

[29]  Christopher Hunt,et al.  Notes on the OpenSURF Library , 2009 .

[30]  Berna Erol,et al.  Paper-Based Augmented Reality , 2007 .

[31]  Gary R. Bradski,et al.  ORB: An efficient alternative to SIFT or SURF , 2011, 2011 International Conference on Computer Vision.

[32]  Hideo Saito,et al.  Augmenting text document by on-line learning of local arrangement of keypoints , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[33]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[34]  Andrew Zisserman,et al.  Multiple View Geometry in Computer Vision (2nd ed) , 2003 .

[35]  Andrew Y. Ng,et al.  Text Detection and Character Recognition in Scene Images with Unsupervised Feature Learning , 2011, 2011 International Conference on Document Analysis and Recognition.

[36]  Ivan Poupyrev,et al.  Virtual object manipulation on a table-top AR environment , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[37]  Mark Billinghurst,et al.  Augmented Reality in the Classroom , 2012, Computer.

[38]  Majid Mirmehdi,et al.  Recognising text in real scenes , 2002, International Journal on Document Analysis and Recognition.

[39]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[40]  Mark Billinghurst,et al.  The mixed reality book: a new multimedia reading experience , 2007, CHI Extended Abstracts.

[41]  Jun Rekimoto,et al.  CyberCode: designing augmented reality environments with visual tags , 2000, DARE '00.

[42]  Masa Inakage,et al.  Little red: storytelling in mixed reality , 2003, SIGGRAPH '03.