Robust Registration of Virtual Objects for Real-Time Augmented Reality

Augmented reality is a technology which allows 2D and 3D computer graphics to be accurately aligned or registered with scenes of the real-world in real-time. The potential uses of this technology are numerous, from architecture and medicine, to manufacturing and entertainment. Vision-based techniques which augment objects onto predetermined planar patterns are considered the most promising approach for achieving accurate registrations, but the majority of the proposed methods fail to provide any robustness to significant changes in pattern scale, orientation, or partial pattern occlusion. This thesis presents the design and implementation of a robust pattern-based augmentation system that addresses these problems, and analyzes its performance using standard consumer-level hardware. Known planar patterns are tracked in a real-time video feed, and virtual 2D and 3D objects are accurately augmented onto these patterns based on the plane's orientation. A method to achieve perspective-correct augmentations without the need for a manual camera calibration procedure is also described. iv Acknowledgments First, I would like to thank my thesis supervisors Gerhard Roth and Jit Bose. Without their support, this thesis would not have been possible. It's not often that the path to a master's thesis is laid out on a yellow brick road, but Gerhard has provided just that. Without his expert guidance and knowledge, his continuous flow of ideas and improvements, his quick turnaround in providing feedback on my writing, and the non-stop stream of technical papers that he would leave for me on my desk, I would not have been able to come this far this quickly. I am especially grateful for the encouragement he has given me throughout this masters experience, as well as the insights he has given me regarding what it takes to become a successful researcher. Additionally, his discussions and opinions regarding government and politics provided a refreshing break from my C++ code. I thank Jit for always being available to discuss thesis issues, course and degree requirements, and potential future PhD research topics and directions, even despite his busy schedule. I am most impressed by his humble demeanor, even though he is an expert in almost everything (including ping-pong and squash). I've always had a bad habit of attempting to finish things as late as possible, and the submission and defense of this thesis was no exception. Both Linda Pfeiffer and Nicki Enouy deserve a huge thank you for scheduling my thesis examination at the last minute, and making …

[1]  Kiriakos N. Kutulakos,et al.  Calibration-Free Augmented Reality , 1998, IEEE Trans. Vis. Comput. Graph..

[2]  Ivan Poupyrev,et al.  Virtual object manipulation on a table-top AR environment , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[3]  G. Roth,et al.  Hand tracking for interactive pattern-based augmented reality , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[4]  George Drettakis,et al.  Interactive Virtual Relighting of Real Scenes , 2000, IEEE Trans. Vis. Comput. Graph..

[5]  Rajeev Sharma,et al.  Real-Time Tracking of Multiple Objects Using Fiducials for Augmented Reality , 2001, Real Time Imaging.

[6]  Axel Pinz,et al.  Building a hybrid tracking system: integration of optical and magnetic tracking , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[7]  Axel Pinz,et al.  Real-Time Optical Edge and Corner Tracking at Subpixel Accuracy , 1999, CAIP.

[8]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Vincent Lepetit,et al.  Handling occlusion in augmented reality systems: a semi-automatic method , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[10]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[11]  Mark de Berg,et al.  Computational geometry: algorithms and applications , 1997 .

[12]  Suya You,et al.  Augmented Reality Tracking in Natural Environments , 1998 .

[13]  Ivan Poupyrev,et al.  The MagicBook: a transitional AR interface , 2001, Comput. Graph..

[14]  Pietro Perona,et al.  Real-time 2-D feature detection on a reconfigurable computer , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[15]  Marie-Odile Berger.  Resolving occlusion in augmented reality: a contour based approach without 3D reconstruction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[16]  Holger Regenbrecht,et al.  A tangible AR desktop environment , 2001, Comput. Graph..

[17]  Hirokazu Kato,et al.  Marker tracking and HMD calibration for a video-based augmented reality conferencing system , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[18]  Mark A. Livingston,et al.  Superior augmented reality registration by integrating landmark tracking and magnetic tracking , 1996, SIGGRAPH.

[19]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[20]  Ronald Azuma,et al.  Hybrid inertial and vision tracking for augmented reality registration , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[21]  Marie-Odile Berger,et al.  Registration with a zoom lens camera for augmented reality applications , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[22]  Thomas Ertl,et al.  Computer Graphics - Principles and Practice, 3rd Edition , 2014 .

[23]  Mongi A. Abidi,et al.  A New Efficient and Direct Solution for Pose Estimation Using Quadrangular Targets: Algorithm and Evaluation , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Mark de Berg,et al.  Computational Geometry: Algorithms and Applications, Second Edition , 2000 .

[25]  Ronald Azuma,et al.  Orientation Tracking for Outdoor Augmented Reality Registration , 1999, IEEE Computer Graphics and Applications.

[26]  M. Carter.  Computer graphics: Principles and practice , 1997 .

[27]  Jun Rekimoto,et al.  Matrix: a realtime object identification and registration method for augmented reality , 1998, Proceedings. 3rd Asia Pacific Computer Human Interaction (Cat. No.98EX110).

[28]  Ivan Poupyrev,et al.  The MagicBook - Moving Seamlessly between Reality and Virtuality , 2001, IEEE Computer Graphics and Applications.

[29]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[30]  Yongduek Seo,et al.  Weakly calibrated video-based augmented reality: embedding and rendering through virtual camera , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[31]  Larry S. Davis,et al.  Iterative pose estimation using coplanar points , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Christopher M. Brown,et al.  Interactive Augmented Reality , 1998 .

[33]  Naokazu Yokoya,et al.  A stereo vision-based augmented reality system with marker and natural feature tracking , 2001, Proceedings Seventh International Conference on Virtual Systems and Multimedia.

[34]  Bruce H. Thomas,et al.  ARQuake: an outdoor/indoor augmented reality first person application , 2000, Digest of Papers. Fourth International Symposium on Wearable Computers.

[35]  Wolfgang Broll,et al.  Interface with angels: the future of VIR and AR interfaces , 2001 .

[36]  Axel Pinz,et al.  The integration of optical and magnetic tracking for multi-user augmented reality , 1999, Comput. Graph..

[37]  Andrew W. Fitzgibbon,et al.  Markerless tracking using planar structures in the scene , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[38]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[39]  Cagatay Basdogan,et al.  Haptics in virtual environments: taxonomy, research status, and challenges , 1997, Comput. Graph..

[40]  Peter F. Sturm,et al.  Algorithms for plane-based pose estimation , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[41]  S. Pizer,et al.  The Image Processing Handbook , 1994 .

[42]  Koichiro Deguchi,et al.  Head pose determination from one image using a generic model , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[43]  Gudrun Klinker,et al.  Augmented Reality: A Problem in Need of Many Computer Vision-Based Solutions , 2000, Confluence of Computer Vision and Computer Graphics.

[44]  Gang Xu,et al.  Epipolar Geometry in Stereo, Motion and Object Recognition , 1996, Computational Imaging and Vision.

[45]  Hideyuki Tamura,et al.  Mixed Reality: Future Dreams Seen at the Border between Real and Virtual Worlds , 2001, IEEE Computer Graphics and Applications.

[46]  Ronald Azuma.  Making Direct Manipulation Work in Virtual Reality , 1997, SIGGRAPH 1997.

[47]  Gerhard Roth,et al.  Hand-based interaction in augmented reality , 2002, IEEE International Workshop HAVE Haptic Virtual Environments and Their.

[48]  Ruzena Bajcsy,et al.  Confluence of Computer Vision and Computer Graphics , 2000, NATO Science Series.

[49]  Hideyuki Tamura,et al.  AR/sup 2/Hockey: a case study of collaborative augmented reality , 1998, Proceedings. IEEE 1998 Virtual Reality Annual International Symposium (Cat. No.98CB36180).

[50]  Emanuele Trucco,et al.  Introductory techniques for 3-D computer vision , 1998 .