Haptic Feedback in Teleoperation in Micro- and Nanoworlds

Robotic systems have been developed to handle very small objects, but their use remains complex and necessitates long-duration training. Simulators, such as molecular simulators, can provide access to large amounts of raw data, but only highly trained users can interpret the results of such systems. Haptic feedback in teleoperation, which provides force feedback to an operator, appears to be a promising solution for interaction with such systems, as it allows intuitiveness and flexibility. However, several issues arise while implementing teleoperation schemes at the micro- and nanoscale, owing to complex force fields that must be transmitted to users and scaling differences between the haptic device and the manipulated objects. Major advances in such technology have been made in recent years. In this chapter, we review the main systems in this area and highlight how some fundamental issues in teleoperation for microand nanoscale applications have been addressed. We consider three types of teleoperation, including (a) direct (manipulation of real objects), (b) virtual (use of simulators), and (c) augmented (combining real robotic systems and simulators). Remaining issues that must be addressed for further advances in teleoperation for micro- and nanoworlds are also discussed, including (a) comprehension of phenomena that dictate very small object (<500 micrometers) behavior and (b) design of intuitive 3-D manipulation systems. Design guidelines to realize an intuitive haptic feedback teleoperation system at the micro- and nanoscale level are proposed.

[1]  Hiroshi Tanaka,et al.  Concept and prototype of protein-ligand docking simulator with force feedback technology , 2002, Bioinform..

[2]  Frederick P. Brooks,et al.  Using a manipulator for force display in molecular docking , 1988, Proceedings. 1988 IEEE International Conference on Robotics and Automation.

[3]  Mehdi Ammi,et al.  Robotic Assisted Micromanipulation System using Virtual Fixtures and Metaphors , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[4]  Cagatay Basdogan,et al.  A Virtual Reality Toolkit for Path Planning and Manipulation at Nano-scale , 2006, 2006 14th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.

[5]  Stéphane Régnier,et al.  Touching the microworld with force-feedback optical tweezers. , 2009, Optics express.

[6]  Hideki Hashimoto,et al.  Macro to nano tele-manipulation through nanoelectromechanical systems , 1998, IECON '98. Proceedings of the 24th Annual Conference of the IEEE Industrial Electronics Society (Cat. No.98CH36200).

[7]  Mandayam A. Srinivasan,et al.  Human-operated 3D micro-manipulator with haptic feedback , 2012, 2012 IEEE Haptics Symposium (HAPTICS).

[8]  C. Bustamante,et al.  Ten years of tension: single-molecule DNA mechanics , 2003, Nature.

[9]  Jean-Loup Florens,et al.  Haptic localization and shape recognition of Nano Objects , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[10]  Sergej Fatikow,et al.  Electrothermal microgrippers for pick-and-place operations , 2008 .

[11]  Dominic Ruh,et al.  Fast parallel interferometric 3D tracking of numerous optically trapped particles and their hydrodynamic interaction. , 2011, Optics express.

[12]  Aude Bolopion,et al.  Haptic Teleoperation for 3-D Microassembly of Spherical Objects , 2012, IEEE/ASME Transactions on Mechatronics.

[13]  Byungkyu Kim,et al.  Cellular force measurement for force reflected biomanipulation , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[14]  M. Sitti,et al.  Augmented reality user interface for an atomic force microscope-based nanorobotic system , 2006, IEEE Transactions on Nanotechnology.

[15]  James R. Lewis,et al.  Sample sizes for usability tests: mostly math, not magic , 2006, INTR.

[16]  Anders Ynnerman,et al.  Designing and Evaluating a Haptic System for Biomolecular Education , 2007, 2007 IEEE Virtual Reality Conference.

[17]  Ganapathy Sivakumar,et al.  Haptic controlled three-axis MEMS gripper system. , 2010, The Review of scientific instruments.

[18]  Graham M. Gibson,et al.  Stereoscopic particle tracking for 3D touch, vision and closed-loop control in optical tweezers , 2011 .

[19]  Aude Bolopion,et al.  Comparing position and force control for interactive molecular simulators with haptic feedback. , 2010, Journal of molecular graphics & modelling.

[20]  S. Chu,et al.  Observation of a single-beam gradient force optical trap for dielectric particles. , 1986, Optics letters.

[21]  Jing Wang,et al.  Visual Haptic-Based Biomolecular Docking , 2008, 2008 International Conference on Cyberworlds.

[22]  Aude Bolopion,et al.  A Review of Haptic Feedback Teleoperation Systems for Micromanipulation and Microassembly , 2013, IEEE Transactions on Automation Science and Engineering.

[23]  Allison M. Okamura,et al.  Feeling is Believing: Using a Force‐Feedback Joystick to Teach Dynamic Systems , 2000 .

[24]  D. Maugis Contact, Adhesion and Rupture of Elastic Solids , 2000 .

[25]  Noah S. Podolefsky,et al.  Use of Analogy in Learning Physics: The Role of Representations. , 2006 .

[26]  Cagatay Basdogan,et al.  Haptic Manipulation of Microspheres Using Optical Tweezers Under the Guidance of Artificial Force Fields , 2007, PRESENCE: Teleoperators and Virtual Environments.

[27]  Yong-Gu Lee,et al.  Stable manipulating of nanowires by line optical tweezers with haptic feedback , 2007, SPIE NanoScience + Engineering.

[28]  Sergej Fatikow,et al.  Haptic interface for a nanohandling robot , 2004 .

[29]  Antoine Ferreira,et al.  Prototyping bio-nanorobots using molecular dynamics simulation and virtual reality , 2008, Microelectron. J..

[30]  Gentiane Venture,et al.  Force-feedback micromanipulation with unconditionally stable coupling , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[31]  Ergonomic requirements for office work with visual display terminals ( VDTs ) — Part 11 : Guidance on usability , 1998 .

[32]  Xinyu Liu,et al.  Vision-based cellular force measurement using an elastic microfabricated device , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[33]  Y. Tashiro,et al.  A Study on Motion Visualization System Using Motion Capture Data , 2007 .

[34]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[35]  Stéphane Régnier,et al.  Invited article: a review of haptic optical tweezers for an interactive microworld exploration. , 2013, The Review of scientific instruments.

[36]  Metin Sitti,et al.  Task-based and stable telenanomanipulation in a nanoscale virtual environment , 2006, IEEE Transactions on Automation Science and Engineering.

[37]  Hideki Hashimoto,et al.  Tele-nanorobotics using atomic force microscope , 1998, Proceedings. 1998 IEEE/RSJ International Conference on Intelligent Robots and Systems. Innovations in Theory, Practice and Applications (Cat. No.98CH36190).

[38]  Noah S. Podolefsky,et al.  When learning about the real world is better done virtually: A study of substituting computer simulations for laboratory equipment , 2005 .

[39]  S. Salcudean,et al.  Toward a tele-nanorobotic manipulation system with atomic scale force feedback and motion resolution , 1990, IEEE Proceedings on Micro Electro Mechanical Systems, An Investigation of Micro Structures, Sensors, Actuators, Machines and Robots..

[40]  Makoto Sato,et al.  An Educational Environment for Chemical Contents with Haptic Interaction , 2007, 2007 International Conference on Cyberworlds (CW'07).

[41]  Yong-Gu Lee,et al.  Smoothing haptic interaction using molecular force calculations , 2004, Comput. Aided Des..

[42]  M. Sitti,et al.  Teleoperated 3-D Force Feedback From the Nanoscale With an Atomic Force Microscope , 2010, IEEE Transactions on Nanotechnology.

[43]  Frederick P. Brooks,et al.  Project GROPEHaptic displays for scientific visualization , 1990, SIGGRAPH.

[44]  Metin Sitti,et al.  Teleoperated touch feedback from the surfaces at the nanoscale: modeling and experiments , 2003 .

[45]  Moussa Boukhnifer,et al.  Wave-based passive control for transparent micro-teleoperation system , 2006, Robotics Auton. Syst..

[46]  Saeid Nahavandi,et al.  A haptic training environment for the heart myoblast cell injection procedure , 2010, 2010 11th International Conference on Control Automation Robotics & Vision.

[47]  K. Mølhave,et al.  Multimodal Electrothermal Silicon Microgrippers for Nanotube Manipulation , 2009, IEEE Transactions on Nanotechnology.

[48]  Ben Horan,et al.  Haptic guidance for microrobotic intracellular injection , 2010, 2010 3rd IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics.

[49]  W. G. Matthews,et al.  Investigation and modification of molecular structures with the nanoManipulator. , 1999, Journal of molecular graphics & modelling.

[50]  Ning Xi,et al.  Modeling and Control of Active End Effector for the AFM Based Nano Robotic Manipulators , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[51]  Claude Cadoz,et al.  Interactive learning of nanophysics phenomena , 2005 .

[52]  Antoine Ferreira,et al.  Virtual reality and haptics for nanorobotics , 2006, IEEE Robotics & Automation Magazine.

[53]  Falvo,et al.  Nanomanipulation Experiments Exploring Frictional and Mechanical Properties of Carbon Nanotubes , 1998, Microscopy and Microanalysis.

[54]  K. Neuman,et al.  Optical trapping. , 2004, The Review of scientific instruments.

[55]  Moussa Boukhnifer,et al.  $H_{\infty }$ Loop Shaping Bilateral Controller for a Two-Fingered Tele-Micromanipulation System , 2007, IEEE Transactions on Control Systems Technology.

[56]  Cagdas D. Onal,et al.  A Scaled Bilateral Control System for Experimental One-dimensional Teleoperated Nanomanipulation , 2009, Int. J. Robotics Res..

[57]  S. Safavi-Naeini,et al.  A Scalable $1\times N$ Optical MEMS Switch Architecture Utilizing a Microassembled Rotating Micromirror , 2007, IEEE Journal of Selected Topics in Quantum Electronics.

[58]  Annie Luciani,et al.  Educational tool for nanophysics using multisensory rendering , 2005, First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics Conference.

[59]  Norihiro Abe,et al.  Providing Simulation of Medical Manipulation with Haptic Feedback , 2007, 17th International Conference on Artificial Reality and Telexistence (ICAT 2007).

[60]  Akio Yamamoto,et al.  The concept of “Haptic Tweezer”, a non-contact object handling system using levitation techniques and haptics , 2007 .

[61]  Aude Bolopion,et al.  2D micro teleoperation with force feedback , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[62]  Aude Bolopion,et al.  Variable gain haptic coupling for molecular simulation , 2011, 2011 IEEE World Haptics Conference.

[63]  Gentiane Venture,et al.  Force-feedback coupling for micro-handling applications , 2006 .

[64]  Brahim Tamadazte,et al.  A Direct Visual Servoing Scheme for Automatic Nanopositioning , 2012, IEEE/ASME Transactions on Mechatronics.

[65]  Thomas R. Tretter,et al.  Haptic Augmentation of Science Instruction: Does Touch Matter? , 2006 .

[66]  Kenneth M Merz,et al.  Haptic applications for molecular structure manipulation. , 2007, Journal of molecular graphics & modelling.

[67]  Jungsik Kim,et al.  Haptic Feedback Based on Physically Based Modeling for Cellular Manipulation Systems , 2008, EuroHaptics.

[68]  Hui Xie,et al.  Three-dimensional automated micromanipulation using a nanotip gripper with multi-feedback , 2009 .

[69]  Deok-Ho Kim,et al.  Dexterous teleoperation for micro parts handling based on haptic/visual interface , 2001, MHS2001. Proceedings of 2001 International Symposium on Micromechatronics and Human Science (Cat. No.01TH8583).

[70]  Stéphane Redon,et al.  Practical modeling of molecular systems with symmetries , 2010, J. Comput. Chem..

[71]  Ryad Benosman,et al.  Asynchronous Event-Based Visual Shape Tracking for Stable Haptic Feedback in Microrobotics , 2012, IEEE Transactions on Robotics.

[72]  Fumihito Arai,et al.  Multi-beam bilateral teleoperation of holographic optical tweezers. , 2012, Optics express.

[73]  Jakob Nielsen,et al.  Chapter 4 – The Usability Engineering Lifecycle , 1993 .

[74]  Anatole Lécuyer,et al.  Path-planning and manipulation of nanotubes using visual and haptic guidance , 2009, 2009 IEEE International Conference on Virtual Environments, Human-Computer Interfaces and Measurements Systems.

[75]  Erk Subasi,et al.  A New Haptic Interaction and Visualization Approach for Rigid Molecular Docking in Virtual Environments , 2008, PRESENCE: Teleoperators and Virtual Environments.

[76]  Antoine Ferreira,et al.  Molecular mechanics simulation of bio-nanorobotic components using force feedback , 2005, 2005 IEEE International Conference on Robotics and Biomimetics - ROBIO.

[77]  Mehdi Ammi,et al.  Evaluation of 3D Pseudo-Haptic Rendering using Vision for Cell Micromanipulation , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[78]  Kuo-Kang Liu,et al.  Optical tweezers for single cells , 2008, Journal of The Royal Society Interface.

[79]  Sung-Yong Cho,et al.  A new micro biological cell injection system , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[80]  M. Gail Jones,et al.  Learning at the nanoscale: The impact of students' use of remote microscopy on concepts of viruses, scale, and microscopy , 2003 .

[81]  N. Xi,et al.  Development of A Haptic User Interface for Surface Sensing and Nanomanipulation Based on Atomic Force Microscope , 2006, 2006 1st IEEE International Conference on Nano/Micro Engineered and Molecular Systems.

[82]  Hui Xie,et al.  Development of a Flexible Robotic System for Multiscale Applications of Micro/Nanoscale Manipulation and Assembly , 2011, IEEE/ASME Transactions on Mechatronics.

[83]  Philippe Lutz,et al.  Towards micro-assembly of hybrid MOEMS components on a reconfigurable silicon free-space micro-optical bench , 2010 .

[84]  H. Ladjal,et al.  Reality-Based Real-Time Cell Indentation Simulator , 2012, IEEE/ASME Transactions on Mechatronics.

[85]  Stéphane Régnier,et al.  Stable six degrees of freedom haptic feedback for flexible ligand-protein docking , 2009, Comput. Aided Des..

[86]  Vincent Hayward,et al.  Haptic interface transparency achieved through viscous coupling , 2012, Int. J. Robotics Res..

[87]  Hyunjae Kang,et al.  Development of a piezoelectric polymer-based sensorized microgripper for microassembly and micromanipulation , 2003, Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453).

[88]  Frederick P. Brooks,et al.  Pearls found on the way to the ideal interface for scanned probe microscopes , 1997, Proceedings. Visualization '97 (Cat. No. 97CB36155).

[89]  Charles Baur,et al.  Delta Haptic Device as a nanomanipulator , 2001, Optics East.

[90]  Aude Bolopion,et al.  Analysis of stability and transparency for nanoscale force feedback in bilateral coupling , 2008 .

[91]  Sergej Fatikow,et al.  AFM-based micro force sensor and haptic interface for a nanohandling robot , 2002, IEEE/RSJ International Conference on Intelligent Robots and Systems.

[92]  M. Gail Jones,et al.  The impact of haptic augmentation on middle school students’ conceptions of the animal cell , 2006, Virtual Reality.

[93]  Johannes S Kanger,et al.  UvA-DARE ( Digital Academic Repository ) Micro magnetic tweezers for nanomanipulation inside live cells , 2005 .

[94]  Fumihito Arai,et al.  Indirect manipulation and bilateral control of the microbe by the laser manipulated microtools , 2000, Proceedings. 2000 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2000) (Cat. No.00CH37113).

[95]  Stephen J. Morrissey,et al.  Reviews of Human Factors and Ergonomics , 2008 .

[96]  Hamid Ladjal,et al.  Microrobotic simulator for assisted biological cell injection , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[97]  Megumi Nakao,et al.  Nanomanipulation and nanotechnology for future diagnostics. , 2008, Studies in health technology and informatics.

[98]  Jaydev P. Desai,et al.  Force feedback interface for cell injection , 2005, First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics Conference.

[99]  Pál Ormos,et al.  Optical microassembly platform for constructing reconfigurable microenvironments for biomedical studies. , 2009, Optics express.

[100]  Yunhui Liu,et al.  Modeling of nanomanipulation with an integrated teleoperated system , 2005, 2005 IEEE International Conference on Robotics and Biomimetics - ROBIO.

[101]  Jean-Loup Florens,et al.  Multi-sensorial interaction with a nano-scale phenomenon : the force curve , 2010, ArXiv.

[102]  Anatole Lécuyer,et al.  Haptics and graphic analogies for the understanding of atomic force microscopy , 2013, Int. J. Hum. Comput. Stud..

[103]  Yuan-Shin Lee,et al.  Energy-Field Optimization and Haptic-Based Molecular Docking and Assembly Search System for Computer-Aided Molecular Design (CAMD) , 2006, 2006 14th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.

[104]  Anatole Lécuyer,et al.  Improving Perception and Understanding of Nanoscale Phenomena Using Haptics and Visual Analogy , 2008, EuroHaptics.

[105]  Graham M. Gibson,et al.  Assembly and force measurement with SPM-like probes in holographic optical tweezers , 2009 .

[106]  V. Altuzar,et al.  Atmospheric pollution profiles in Mexico City in two different seasons , 2003 .

[107]  Jean Scholtz,et al.  Common metrics for human-robot interaction , 2006, HRI '06.

[108]  Charlie Gosse,et al.  Magnetic tweezers: micromanipulation and force measurement at the molecular level. , 2002, Biophysical journal.

[109]  C. P. A C O R E T,et al.  Asynchronous event-based high speed vision for microparticle tracking , 2011 .

[110]  Y. Hatamura,et al.  Direct coupling system between nanometer world and human world , 1990, IEEE Proceedings on Micro Electro Mechanical Systems, An Investigation of Micro Structures, Sensors, Actuators, Machines and Robots..

[111]  Russell M. Taylor,et al.  Controlled manipulation of molecular samples with the nanoManipulator , 1999, 1999 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (Cat. No.99TH8399).

[112]  Aude Bolopion,et al.  Vision-Based Haptic Feedback for Remote Micromanipulation in–SEM Environment , 2012 .

[113]  Sergej Fatikow,et al.  Remote microscale teleoperation through virtual reality and haptic feedback , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[114]  Michel F. Sanner,et al.  Role of haptics in teaching structural molecular biology , 2003, 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2003. HAPTICS 2003. Proceedings..

[115]  Jaydev P. Desai,et al.  Evaluating the Effect of Force Feedback in Cell Injection , 2007, IEEE Transactions on Automation Science and Engineering.