A Flexible Mixed Reality Simulation Framework for Software Development in Robotics

In recent years robots have dramatically improved in functionality, but as a result their designs have also become more complicated. The increase in the complexity of the tasks and the design of the robot result in a challenging and time-consuming robot development cycle. This paper identifies requirements for improving the robot development process, focusing on facilitating a safe and reliable transition process from simulation to real world tests. We propose that, by applying the concept of Mixed Reality (MR) to robot simulation, it is possible to create a realistic, synthetic environment that enables robot software developers to experiment freely and safely. The new simulation approach, named MR simulation, offers the flexibility of creating simulation environments composed of real and virtual objects that seamlessly interact with one another. This paper contributes a generic, conceptual MR framework which is demonstrated to provide a common, standardised simulation environment for experimentation involving real and virtual objects. An MR robot simulator is implemented based on the framework, integrating standardised data interfaces in its design to increase interoperability. Evaluation results demonstrate that the MR robot simulator is reusable in three different applications. It improved simulation reliability, reduced development cost, and increased efficiency.

[1]  Burkhard Wünsche,et al.  Analysing Mixed Reality Simulation for Industrial Applications: A Case Study in the Development of a Robotic Screw Remover System , 2010, SIMPAR.

[2]  Dieter Schmalstieg,et al.  Agents that talk and hit back: animated agents in augmented reality , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[3]  Bruce A. MacDonald,et al.  Covariance Visualisations for Simultaneous Localisation and Mapping , 2009 .

[4]  Dieter Schmalstieg,et al.  The Studierstube Augmented Reality Project , 2002, Presence: Teleoperators & Virtual Environments.

[5]  Zhen Li,et al.  HARDWARE-IN-THE-LOOP SIMULATION , 2004 .

[6]  James J. Kuffner,et al.  OpenRAVE: A Planning Architecture for Autonomous Robotics , 2008 .

[7]  Burkhard Wünsche,et al.  Designing a Mixed Reality Framework for Enriching Interactions in Robot Simulation , 2010, GRAPP.

[8]  Bruce A. MacDonald,et al.  An Augmented Reality Debugging System for Mobile Robot Software Engineers , 2010 .

[9]  Andrew Howard,et al.  Design and use paradigms for Gazebo, an open-source multi-robot simulator , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[10]  Mark Billinghurst,et al.  OSGART - A Pragmatic Approach to MR , 2006 .

[11]  John Pretlove,et al.  Augmented reality for programming industrial robots , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[12]  Stefano Caselli,et al.  Robot grasp synthesis from virtual demonstration and topology-preserving environment reconstruction , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[13]  Bruce H. Thomas,et al.  An object-oriented software architecture for 3D mixed reality applications , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[14]  M. Bacic,et al.  On hardware-in-the-loop simulation , 2005, Proceedings of the 44th IEEE Conference on Decision and Control.

[15]  Gregory M. P. O'Hare,et al.  Using Mixed Reality Agents as Social Interfaces for Robots , 2007, RO-MAN 2007 - The 16th IEEE International Symposium on Robot and Human Interactive Communication.

[16]  Paul Milgram,et al.  Telerobotic control using augmented reality , 1995, Proceedings 4th IEEE International Workshop on Robot and Human Communication.

[17]  J. Banks,et al.  Handbook of Simulation , 1998 .

[18]  Paul Milgram,et al.  A Taxonomy of Real and Virtual World Display Integration , 1999 .

[19]  P. Ridao,et al.  Neptune: a hil simulator for multiple UUVs , 2004, Oceans '04 MTS/IEEE Techno-Ocean '04 (IEEE Cat. No.04CH37600).

[20]  Stefano Carpin,et al.  Quantitative Assessments of USARSim Accuracy , 2006 .

[21]  Jörg Kaiser,et al.  Visualization of robot's awareness and perception , 2010, IWDE '10.

[22]  Yvonne Rogers,et al.  A Conceptual Framework for Mixed Reality Environments: Designing Novel Learning Activities for Young Children , 2002, Presence: Teleoperators & Virtual Environments.

[23]  Ali Haydar GÖKTO,et al.  An Augmented Reality System for Multi-UAV Missions , 2005 .

[24]  Michael A. Goodrich,et al.  Ecological Interfaces for Improving Mobile Robot Teleoperation , 2007, IEEE Transactions on Robotics.

[25]  Zoltán Papp,et al.  High Fidelity Real-Time Simulator With Mixed Real and Virtual Sensors , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[26]  D.M. Lane,et al.  An Augmented Reality Architecture for the Creation of Hardware-in-the-Loop & Hybrid Simulation Test Scenarios for Unmanned Underwater Vehicles , 2007, OCEANS 2007.

[27]  M.R. Emami,et al.  An Architecture for Robotic Hardware-in-the-Loop Simulation , 2006, 2006 International Conference on Mechatronics and Automation.

[28]  Bruce A. MacDonald,et al.  An intuitive interface for a cognitive programming by demonstration system , 2008, 2008 IEEE International Conference on Robotics and Automation.

[29]  Andrew Y. C. Nee,et al.  Methodologies for immersive robot programming in an augmented reality environment , 2006, GRAPHITE '06.

[30]  Fumitoshi Matsuno,et al.  Time Follower's Vision: a teleoperation interface with past images , 2005, IEEE Computer Graphics and Applications.

[31]  Atsushi Konno,et al.  Hybrid simulation of a dual-arm space robot colliding with a floating object , 2008, 2008 IEEE International Conference on Robotics and Automation.

[32]  Tetsuo Kotoku,et al.  A simulation environment for OpenRTM-aist , 2009, 2009 IEEE/SICE International Symposium on System Integration (SII).

[33]  Vesna Brujic-Okretic,et al.  Remote vehicle manoeuvring using augmented reality , 2003 .

[34]  Tamaki Ura,et al.  AUV test using real/virtual synthetic world , 1996, Proceedings of Symposium on Autonomous Underwater Vehicle Technology.

[35]  David W. Payton,et al.  World embedded interfaces for human-robot interaction , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[36]  Emilio Frazzoli,et al.  Aggressive Maneuvering of Small Autonomous Helicopters: A Human-Centered Approach , 2001, Int. J. Robotics Res..

[37]  Stefano Carpin,et al.  USARSim: a robot simulator for research and education , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[38]  Satoshi Kagami,et al.  Viewing and reviewing how humanoids sensed, planned and behaved with Mixed Reality technology , 2007, 2007 7th IEEE-RAS International Conference on Humanoid Robots.

[39]  Satoshi Kagami,et al.  Augmented Reality for Robot Development and Experimentation , 2005 .

[40]  Takashi Suehiro,et al.  A Software Platform for Component Based RT-System Development: OpenRTM-Aist , 2008, SIMPAR.

[41]  Eric N. Johnson,et al.  FLIGHT SIMULATION FOR THE DEVELOPMENT OF AN EXPERIMENTAL UAV , 2002 .

[42]  Richard T. Vaughan,et al.  The Player/Stage Project: Tools for Multi-Robot and Distributed Sensor Systems , 2003 .

[43]  Steve Benford,et al.  Understanding and constructing shared spaces with mixed-reality boundaries , 1998, TCHI.

[44]  Satoshi Kagami,et al.  Mixed reality environment for autonomous robot development , 2008, 2008 IEEE International Conference on Robotics and Automation.

[45]  Burkhard Wünsche,et al.  Mixed reality simulation for mobile robots , 2009, 2009 IEEE International Conference on Robotics and Automation.

[46]  Richard Vaughan,et al.  Massively multi-robot simulation in stage , 2008, Swarm Intelligence.

[47]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[48]  Jerry Banks,et al.  Handbook of simulation - principles, methodology, advances, applications, and practice , 1998, A Wiley-Interscience publication.

[49]  Abhinandan Jain,et al.  A reusable, real-time spacecraft dynamics simulator , 1997, 16th DASC. AIAA/IEEE Digital Avionics Systems Conference. Reflections to the Future. Proceedings.