Mechanical assembly assistance using marker-less augmented reality system

Purpose This paper aims to propose a real-time augmented reality (AR)-based assembly assistance system using a coarse-to-fine marker-less tracking strategy. The system automatically adapts to tracking requirement when the topological structure of the assembly changes after each assembly step. Design/methodology/approach The prototype system’s process can be divided into two stages: the offline preparation stage and online execution stage. In the offline preparation stage, planning results (assembly sequence, parts position, rotation, etc.) and image features [gradient and oriented FAST and rotated BRIEF (ORB)features] are extracted automatically from the assembly planning process. In the online execution stage, too, image features are extracted and matched with those generated offline to compute the camera pose, and planning results stored in XML files are parsed to generate the assembly instructions for manipulators. In the prototype system, the working range of template matching algorithm, LINE-MOD, is first extended by using depth information; then, a fast and robust marker-less tracker that combines the modified LINE-MOD algorithm and ORB tracker is designed to update the camera pose continuously. Furthermore, to track the camera pose stably, a tracking strategy according to the characteristic of assembly is presented herein. Findings The tracking accuracy and time of the proposed marker-less tracking approach were evaluated, and the results showed that the tracking method could run at 30 fps and the position and pose tracking accuracy was slightly superior to ARToolKit. Originality/value The main contributions of this work are as follows: First, the authors present a coarse-to-fine marker-less tracking method that uses modified state-of-the-art template matching algorithm, LINE-MOD, to find the coarse camera pose. Then, a feature point tracker ORB is activated to calculate the accurate camera pose. The whole tracking pipeline needs, on average, 24.35 ms for each frame, which can satisfy the real-time requirement for AR assembly. On basis of this algorithm, the authors present a generic tracking strategy according to the characteristics of the assembly and develop a generic AR-based assembly assistance platform. Second, the authors present a feature point mismatch-eliminating rule based on the orientation vector. By obtaining stable matching feature points, the proposed system can achieve accurate tracking results. The evaluation of the camera position and pose tracking accuracy result show that the study’s method is slightly superior to ARToolKit markers.

[1]  In-So Kweon,et al.  Robust model-based 3D object recognition by combining feature matching with tracking , 2003, 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422).

[2]  Shiqi Li,et al.  A Robust O(n) Solution to the Perspective-n-Point Problem , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Alexei A. Goon,et al.  A SURVEY OF TRACKING TECHNOLOGY FOR VIRTUAL ENVIRONMENTS , 1999 .

[4]  Jiri Matas,et al.  P-N learning: Bootstrapping binary classifiers by structural constraints , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[5]  Ralf Dörner,et al.  Accuracy in optical tracking with fiducial markers: an accuracy function for ARToolKit , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[6]  Suya You,et al.  Fusion of vision and gyro tracking for robust augmented reality registration , 2001, Proceedings IEEE Virtual Reality 2001.

[7]  Silvio Savarese,et al.  A coarse-to-fine model for 3D pose estimation and sub-category recognition , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Vincent Lepetit,et al.  Model Based Training, Detection and Pose Estimation of Texture-Less 3D Objects in Heavily Cluttered Scenes , 2012, ACCV.

[9]  Soh-Khim Ong,et al.  Integrated product design and assembly planning in an augmented reality environment , 2013 .

[10]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[11]  Éric Marchand,et al.  Real-time 3D model-based tracking: combining edge and texture information , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[12]  Vincent Lepetit,et al.  A Novel Representation of Parts for Accurate 3D Object Detection and Tracking in Monocular Images , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[13]  Gary R. Bradski,et al.  ORB: An efficient alternative to SIFT or SURF , 2011, 2011 International Conference on Computer Vision.

[14]  Daniel Cremers,et al.  LSD-SLAM: Large-Scale Direct Monocular SLAM , 2014, ECCV.

[15]  Daniel Cremers,et al.  Large-scale direct SLAM for omnidirectional cameras , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[16]  Javier Díaz,et al.  Real-Time Model-Based Rigid Object Pose Estimation and Tracking Combining Dense and Sparse Visual Cues , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Roland Siegwart,et al.  BRISK: Binary Robust invariant scalable keypoints , 2011, 2011 International Conference on Computer Vision.

[18]  David G. Lowe,et al.  What and Where: 3D Object Recognition with Accurate Pose , 2006, Toward Category-Level Object Recognition.

[19]  Carl K. Chang,et al.  Rigid Object Tracking Algorithms for Low-Cost AR Devices , 2014 .

[20]  Andrew Y. C. Nee,et al.  Assembly planning and evaluation in an augmented reality environment , 2013 .

[21]  Tobias Höllerer,et al.  Multithreaded Hybrid Feature Tracking for Markerless Augmented Reality , 2009, IEEE Transactions on Visualization and Computer Graphics.

[22]  A. Y. C. Nee,et al.  Multi-modal augmented-reality assembly guidance based on bare-hand interface , 2016, Adv. Eng. Informatics.

[23]  Éric Marchand,et al.  Deformable random dot markers , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[24]  Soh-Khim Ong,et al.  Real-virtual components interaction for assembly simulation and planning , 2016 .

[25]  Ian D. Reid,et al.  PWP3D: Real-time Segmentation and Tracking of 3D Objects , 2009, BMVC.

[26]  Vincent Lepetit,et al.  Gradient Response Maps for Real-Time Detection of Textureless Objects , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Federico Tombari,et al.  BOLD Features to Detect Texture-less Objects , 2013, 2013 IEEE International Conference on Computer Vision.

[28]  Vincent Lepetit,et al.  Stable real-time 3D tracking using online and offline information , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Gregory D. Hager,et al.  Fast and Globally Convergent Pose Estimation from Video Images , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[30]  Andrew Y. C. Nee,et al.  An authorable context-aware augmented reality system to assist the maintenance technicians , 2012 .

[31]  K. G. Swift,et al.  Decision support for sequence generation in an assembly oriented design environment , 2004 .

[32]  Benjamin Lok,et al.  A pipeline for rapidly incorporating real objects into a mixed environment , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[33]  Antonija Mitrovic,et al.  Intelligent Augmented Reality Training for Motherboard Assembly , 2015, International Journal of Artificial Intelligence in Education.

[34]  Jörg Stückler,et al.  Large-scale direct SLAM with stereo cameras , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[35]  Bin Wang,et al.  Global optimal searching for textureless 3D object tracking , 2015, The Visual Computer.

[36]  Rafael Radkowski.  A Point Cloud-Based Method for Object Alignment Verification for Augmented Reality Applications , 2015 .

[37]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[38]  Xinyu Li,et al.  Markless tracking based on natural feature for Augmented Reality , 2010, 2010 International Conference on Educational and Information Technology.

[39]  Cordelia Schmid,et al.  An Affine Invariant Interest Point Detector , 2002, ECCV.

[40]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[41]  Sinisa Todorovic,et al.  From contours to 3D object detection and pose estimation , 2011, 2011 International Conference on Computer Vision.

[42]  Diego Borro,et al.  Providing guidance for maintenance operations using automatic markerless augmented reality system , 2011, ISMAR.

[43]  Vincent Lepetit,et al.  Randomized trees for real-time keypoint recognition , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[44]  Vincent Lepetit,et al.  Model Based Training, Detection and Pose Estimation of Texture-Less 3D Objects in Heavily Cluttered Scenes , 2012, ACCV.

[45]  Dima Damen,et al.  Real-time Learning and Detection of 3D Texture-less Objects: A Scalable Approach , 2012, BMVC 2012.