Programming manipulation tasks by demonstration in visuo-haptic augmented reality

This work investigates the use of visuo-haptic augmented reality for programming object manipulation tasks by demonstration. A desktop augmented reality system is presented in which the user is not co-located with the environment and a 3DOF haptic device, providing force feedback, is adopted for user interaction. The interaction paradigm allows the user to select and manipulate virtual objects superimposed upon a visual representation of the real workspace. The approach also supports physics-based animation of rigid bodies. In the proposed method automatic object recognition and registration are performed from 3D range data acquired by a laser scanner mounted on a robot arm. We next investigate the potential of the augmented reality system for programming manipulation tasks by demonstration. Experiments show that a precedence graph, encoding the sequential structure of the task, can be successfully extracted from multiple user demonstrations.

[1]  Andrew Y. C. Nee,et al.  Interactive robot trajectory planning and simulation using Augmented Reality , 2012 .

[2]  Seokhee Jeon,et al.  Haptic Augmented Reality: Taxonomy and an Example of Stiffness Modulation , 2009, PRESENCE: Teleoperators and Virtual Environments.

[3]  Gregory D. Hager,et al.  Augmented Reality Combining Haptics and Vision , 2003 .

[4]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Matthias Harders,et al.  Extending haptic augmented reality: Modulating stiffness during two-point squeezing , 2012, 2012 IEEE Haptics Symposium (HAPTICS).

[6]  Gregory D. Hager,et al.  VisHap: augmented reality combining haptics and vision , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[7]  Kwanghee Ko,et al.  Introduction of Physics Simulation in Augmented Reality , 2008, 2008 International Symposium on Ubiquitous Virtual Reality.

[8]  Christian Sandor,et al.  Visuo-Haptic Systems: Half-Mirrors Considered Harmful , 2007, Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (WHC'07).

[9]  D. Burschka,et al.  Dynamic Augmented Reality for Sensory Substitution in Robot-Assisted Surgical Systems , 2006, 2006 International Conference of the IEEE Engineering in Medicine and Biology Society.

[10]  Christopher M. Brown,et al.  Haptics in augmented reality , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[11]  A. Kheddar,et al.  A haptic prototype for the simulations of aeronautics mounting/unmounting operations , 2001, Proceedings 10th IEEE International Workshop on Robot and Human Interactive Communication. ROMAN 2001 (Cat. No.01TH8591).

[12]  Damon Shing-Min Liu,et al.  A Physics-Based Augmented Reality Jenga Stacking Game , 2011, 2011 Workshop on Digital Media and Digital Content Management.

[13]  Nico Blodow,et al.  Fast Point Feature Histograms (FPFH) for 3D registration , 2009, 2009 IEEE International Conference on Robotics and Automation.

[14]  M. Bergamasco,et al.  Hand - a haptic system for analysis and driving of hand movements in augmented reality environment , 2003, The 12th IEEE International Workshop on Robot and Human Interactive Communication, 2003. Proceedings. ROMAN 2003..

[15]  Darwin G. Caldwell,et al.  Integrating Haptics with Augmented Reality in a Femoral Palpation and Needle Insertion Training Simulation , 2011, IEEE Transactions on Haptics.

[16]  Gábor Székely,et al.  Calibration, Registration, and Synchronization for High Precision Augmented Reality Haptics , 2009, IEEE Transactions on Visualization and Computer Graphics.

[17]  Soh-Khim Ong,et al.  Robot Programming Using Augmented Reality , 2009, 2009 International Conference on CyberWorlds.

[18]  Andreas Pichler,et al.  User centered framework for intuitive robot programming , 2010, 2010 IEEE International Workshop on Robotic and Sensors Environments.

[19]  Mary C. Whitton,et al.  Incorporating dynamic real objects into immersive virtual environments , 2003, I3D '03.

[20]  Gábor Székely,et al.  High Precision Augmented Reality Haptics , 2006 .

[21]  Éric Marchand,et al.  Real-time markerless tracking for augmented reality: the virtual visual servoing framework , 2006, IEEE Transactions on Visualization and Computer Graphics.

[22]  Dangxiao Wang,et al.  Collocation Accuracy of Visuo-Haptic System: Metrics and Calibration. , 2011, IEEE transactions on haptics.

[23]  Osamu Oshiro,et al.  Visual and haptic augmentation technologies for surgical navigation , 2010, 2010 International Symposium on Micro-NanoMechatronics and Human Science.

[24]  Domenico Prattichizzo,et al.  Shape and weight rendering for haptic Augmented Reality , 2010, 19th International Symposium in Robot and Human Interactive Communication.

[25]  Siam Charoenseang,et al.  Augmented reality for skill transfer in assembly task , 2005, ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005..

[26]  Fabio Bruno,et al.  Visuo-Haptic Mixed Reality with Unobstructed Tool-Hand Integration , 2013, IEEE Transactions on Visualization and Computer Graphics.

[27]  Takuji Narumi,et al.  Poster: A wearable augmented reality system with haptic feedback and its performance in virtual assembly tasks , 2013, 2013 IEEE Symposium on 3D User Interfaces (3DUI).

[28]  Stefano Caselli,et al.  Object categorization and grasping by parts from range scan data , 2012, 2012 IEEE International Conference on Robotics and Automation.

[29]  Andrew Y. C. Nee,et al.  Robot programming using augmented reality: An interactive method for planning collision-free paths , 2009 .

[30]  Purang Abolmaesumi,et al.  An Augmented Reality Haptic Training Simulator for Spinal Needle Procedures , 2013, IEEE Transactions on Biomedical Engineering.

[31]  Thomas Hulin,et al.  Evaluating exemplary training accelerators for Programming-by-Demonstration , 2010, 19th International Symposium in Robot and Human Interactive Communication.

[32]  Hee Chan Kim,et al.  A haptic augmented reality surgeon console for a laparoscopic surgery robot system , 2013, 2013 13th International Conference on Control, Automation and Systems (ICCAS 2013).

[33]  Stefano Caselli,et al.  Object manipulation in visuo-haptic augmented reality with physics-based animation , 2010, 19th International Symposium in Robot and Human Interactive Communication.

[34]  Rüdiger Dillmann,et al.  Towards Cognitive Robots: Building Hierarchical Task Representations of Manipulations from Human Demonstration , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.