High-fidelity visuo-haptic interaction with virtual objects in multi-modal AR systems

The driving force of our research is the precise combination of real and - possibly indistinguishable - virtual objects in an interactive augmented reality environment. This requires real-time, multimodal simulation, as well as stable and accurate overlay of the computer-generated objects. This paper describes several methods to improve accuracy and stability of our hybrid augmented reality system. In a comparison of two approaches to hybrid head pose refinement, we show that Quasi-Newton method enables high performance optimization for image space error minimization. Moreover, a 3D landmark refinement step is proposed, which significantly improves quality and robustness of the overlay process. The enhanced system is demonstrated in an interactive AR environment, which provides accurate haptic feedback from real and virtual deformable objects. Finally, the effect of landmark occlusion on tracking stability during user interaction is also analyzed.

[1]  Mark A. Livingston,et al.  Superior augmented reality registration by integrating landmark tracking and magnetic tracking , 1996, SIGGRAPH.

[2]  K. S. Arun,et al.  Least-Squares Fitting of Two 3-D Point Sets , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Jian L. Zhou,et al.  User's Guide for CFSQP Version 2.0: A C Code for Solving (Large Scale) Constrained Nonlinear (Minimax) Optimization Problems, Generating Iterates Satisfying All Inequality Constraints , 1994 .

[4]  Nassir Navab,et al.  Automated initialization for marker-less tracking: a sensor fusion approach , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[5]  W. A. Ho,et al.  Fusion of data from head-mounted and fixed sensors , 1998 .

[6]  William H. Press,et al.  Numerical recipes in C. The art of scientific computing , 1987 .

[7]  Tyrone L. Vincent,et al.  Analysis of Head Pose Accuracy in Augmented Reality , 2000, IEEE Trans. Vis. Comput. Graph..

[8]  Mark A. Livingston,et al.  Managing latency in complex augmented reality systems , 1997, SI3D.

[9]  Hong Z. Tan,et al.  Discrimination of Real and Virtual High-Definition Textured Surfaces , 2006, 2006 14th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.

[10]  Gábor Székely,et al.  High Precision Augmented Reality Haptics , 2006 .

[11]  Luc Soler,et al.  A Complete Augmented Reality Guidance System for Liver Punctures: First Clinical Evaluation , 2005, MICCAI.

[12]  Katherine J. Kuchenbecker,et al.  Event-based haptics and acceleration matching: portraying and assessing the realism of contact , 2005, First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics Conference.

[13]  Kaj Madsen,et al.  Methods for Non-Linear Least Squares Problems , 1999 .

[14]  Philippe C. Cattin,et al.  Camera-marker alignment framework and comparison with hand-eye calibration for augmented reality applications , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[15]  Axel Pinz,et al.  Building a hybrid tracking system: integration of optical and magnetic tracking , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[16]  Manolis I. A. Lourakis,et al.  The design and implementation of a generic sparse bundle adjustment software package based on the Le , 2004 .

[17]  Ulrich Neumann,et al.  Dynamic registration correction in augmented-reality systems , 1995, Proceedings Virtual Reality Annual International Symposium '95.

[18]  John C. Platt,et al.  Elastically deformable models , 1987, SIGGRAPH.

[19]  F. A. Seiler,et al.  Numerical Recipes in C: The Art of Scientific Computing , 1989 .

[20]  Gregory D. Hager,et al.  Fast and Globally Convergent Pose Estimation from Video Images , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Marie-Odile Berger,et al.  Handling uncertain sensor data in vision-based camera tracking , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[22]  Roger Y. Tsai,et al.  A new technique for fully autonomous and efficient 3D robotics hand/eye calibration , 1988, IEEE Trans. Robotics Autom..

[23]  James M. Van Verth,et al.  Adding force feedback to graphics systems: issues and solutions , 1996, SIGGRAPH.

[24]  Gábor Székely,et al.  Assessing the Fidelity of Haptically Rendered Deformable Objects , 2006, 2006 14th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.

[25]  Nassir Navab,et al.  Automatic Patient Registration for Port Placement in Minimally Invasixe Endoscopic Surgery , 2005, MICCAI.

[26]  Ronald Azuma,et al.  Hybrid inertial and vision tracking for augmented reality registration , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[27]  William H. Press,et al.  The Art of Scientific Computing Second Edition , 1998 .