Providing guidance for maintenance operations using automatic markerless Augmented Reality system

This paper proposes a new real-time Augmented Reality based tool to help in disassembly for maintenance operations. This tool provides workers with augmented instructions to perform maintenance tasks more efficiently. Our prototype is a complete framework characterized by its capability to automatically generate all the necessary data from input based on untextured 3D triangle meshes, without requiring additional user intervention. An automatic offline stage extracts the basic geometric features. These are used during the online stage to compute the camera pose from a monocular image. Thus, we can handle the usual textureless 3D models used in industrial applications. A self-supplied and robust markerless tracking system that combines an edge tracker, a point based tracker and a 3D particle filter has also been designed to continuously update the camera pose. Our framework incorporates an automatic path-planning module. During the offline stage, the assembly/disassembly sequence is automatically deduced from the 3D model geometry. This information is used to generate the disassembly instructions for workers.

[1]  Mark Billinghurst,et al.  Augmented assembly using a mobile phone , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[2]  Dieter Schmalstieg,et al.  ARToolKitPlus for Pose Trackin on Mobile Devices , 2007 .

[3]  Luigi di Stefano,et al.  Augmented Reality for Aircraft Maintenance Training and Operations Support , 2011, IEEE Computer Graphics and Applications.

[4]  Andrew Calway,et al.  Real-Time Camera Tracking Using a Particle Filter , 2005, BMVC.

[5]  Diego Borro,et al.  Path‐planning techniques for the simulation of disassembly tasks , 2007 .

[6]  Michael T. Heath,et al.  Feature Detection for Surface Meshes , 2002 .

[7]  Diego Borro,et al.  Towards real time 3D tracking and reconstruction on a GPU using Monte Carlo simulations , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[8]  David W. Murray,et al.  Full-3D Edge Tracking with a Particle Filter , 2006, BMVC.

[9]  Gerrit Voss,et al.  Augmented reality for enhancement of endoscopic interventions , 2003, IEEE Virtual Reality, 2003. Proceedings..

[10]  Kenji Shimada,et al.  Polygon crawling: feature edge extraction from a general polygonal surface for mesh generation , 2009, Engineering with Computers.

[11]  Robert Laganière,et al.  The detection of junction features in images , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[12]  Andrea Salgian,et al.  A Perceptual Grouping Hierarchy for Appearance-Based 3D Object Recognition , 1999, Comput. Vis. Image Underst..

[13]  Jee-Hwan Ryu,et al.  Development of Educational System for Automotive Engineering based on Augmented Reality , 2009 .

[14]  Vincent Lepetit,et al.  Polyhedral object detection and pose estimation for augmented reality applications , 2002, Proceedings of Computer Animation 2002 (CA 2002).

[15]  Didier Stricker,et al.  Real-time 3D Camera Tracking for Industrial Augmented Reality Applications , 2005, WSCG.

[16]  Hiroyuki Yamamoto,et al.  A Fast Initialization Method for Edge-based Registration Using an Inclination Constraint , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[17]  Diego Borro,et al.  Junction assisted 3D pose retrieval of untextured 3D models in monocular images , 2013, Comput. Vis. Image Underst..

[18]  C. Steger.  OCCLUSION , CLUTTER , AND ILLUMINATION INVARIANT OBJECT RECOGNITION , 2002 .

[19]  Tom Drummond,et al.  Semi-automatic Annotations in Unknown Environments , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[20]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[21]  Jean-Claude Latombe,et al.  Motion Planning: A Journey of Robots, Molecules, Digital Actors, and Other Artifacts , 1999, Int. J. Robotics Res..

[22]  Didier Stricker,et al.  Adaptable Model-Based Tracking Using Analysis-by-Synthesis Techniques , 2007, CAIP.

[23]  Diego Borro,et al.  Providing guidance for maintenance operations using automatic markerless augmented reality system , 2011, ISMAR.

[24]  Vincent Lepetit,et al.  Combining edge and texture information for real-time accurate 3D camera tracking , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[25]  Pascal Fua,et al.  Vision Based 3D Tracking and Pose Estimation for Mixed Reality , 2007 .

[26]  Dieter Schmalstieg,et al.  Augmented Reality for Industrial Building Acceptance , 2008, 2008 IEEE Virtual Reality Conference.

[27]  Andrew Y. C. Nee,et al.  Augmented reality for assembly guidance using a virtual interactive tool , 2008 .

[28]  Werner Hartmann,et al.  Authoring of a mixed reality assembly instructor for hierarchical structures , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[29]  Didier Stricker,et al.  Lessons learned on the way to industrial augmented reality applications, a retrospective on ARVIKA , 2003, Comput. Graph..

[30]  Pierre Gurdjos,et al.  MARCH: mobile augmented reality for cultural heritage , 2009, ACM Multimedia.

[31]  Woodrow Barfield,et al.  Evaluating the effectiveness of augmented reality displays for a manual assembly task , 1999, Virtual Reality.

[32]  Alfredo Liverani,et al.  A CAD-augmented Reality Integrated Environment for Assembly Sequence Check and Interactive Validation , 2004, Concurr. Eng. Res. Appl..

[33]  Marion Langer,et al.  Automatic contour model creation out of polygonal CAD models for markerless Augmented Reality , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[34]  Carlos Rocha,et al.  Local shape features for object recognition , 2008, 2008 19th International Conference on Pattern Recognition.

[35]  Tim Hauke Heibel,et al.  A mobile markerless AR system for maintenance and repair , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[36]  Randall H. Wilson,et al.  On geometric assembly planning , 1992 .

[37]  Kazem Abhary,et al.  Assembly sequence planning and optimisation using genetic algorithms: Part I. Automatic generation of feasible assembly sequences , 2003, Appl. Soft Comput..

[38]  Hirokazu Kato,et al.  Marker tracking and HMD calibration for a video-based augmented reality conferencing system , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[39]  Vincent Lepetit,et al.  Dominant orientation templates for real-time detection of texture-less objects , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[40]  Tom Drummond,et al.  Fusing points and lines for high performance tracking , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[41]  Diego Borro,et al.  Parallel RRT-based path planning for selective disassembly planning , 2008 .

[42]  Frank Biocca,et al.  Comparative effectiveness of augmented reality in object assembly , 2003, CHI '03.

[43]  V. Lepetit,et al.  EPnP: An Accurate O(n) Solution to the PnP Problem , 2009, International Journal of Computer Vision.

[44]  Maria del Carmen Juan Lizandra,et al.  Learning Words Using Augmented Reality , 2010, 2010 10th IEEE International Conference on Advanced Learning Technologies.

[45]  Rajeev Sharma,et al.  Interactive evaluation of assembly sequences using augmented reality , 1999, IEEE Trans. Robotics Autom..

[46]  Arthur C. Sanderson,et al.  Representations of mechanical assembly sequences , 1991, IEEE Trans. Robotics Autom..

[47]  Steven K. Feiner,et al.  Evaluating the benefits of augmented reality for task localization in maintenance of an armored personnel carrier turret , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.