Haptic Signal Reconstruction in eHealth Internet of Things

With the haptic technology continuously enlarging the eHealth Industry Internet of Things (IIoT) ecosystem, haptic perception service which requires effective haptic signal reconstruction for immersive experience has become an indispensable function. However, the majority of existing haptic signal reconstruction methods are generally inefficient because of undergoing extremely complex operations or inefficient feature representations. To resolve this dilemma, this article proposes a long short-term memory-based force reconstruction network (LSTM-FRN) by designing a novel sparse attention module for low-latency reconstruction and a novel metric learning-based constraint for high-precision reconstruction, yielding an excellent tradeoff between the computational complexity and feature representation. To train our network, we construct a large-scale data set of synchronous needle motion signals and haptic signals in acupuncture needle insertion. Finally, we build an interactive needle insertion training system (HapAR-NITS) by integrating augmented reality (AR), the LSTM-FRN-based haptic reconstruction as well as a skill assessment subsystem. Comprehensive experiments demonstrate that the proposed multiple technologies enable our HapAR-NITS to achieve satisfying immersive experience and manipulation effects.

[1]  Z. Khurshid,et al.  Significance of Haptic and Virtual Reality Simulation (VRS) in the Dental Education: A Review of Literature , 2021, Applied Sciences.

[2]  Zhengtao Yu,et al.  Intelligent Intraoperative Haptic-AR Navigation for COVID-19 Lung Biopsy Using Deep Hybrid Model , 2021, IEEE Transactions on Industrial Informatics.

[3]  Jianxin Chen,et al.  Cross-Modal Stream Scheduling for eHealth , 2021, IEEE Journal on Selected Areas in Communications.

[4]  Kosuke Sato,et al.  HaptoMapping: Visuo-Haptic AR System usingProjection-based Control of Wearable Haptic Devices , 2020, SIGGRAPH ASIA Emerging Technologies.

[5]  Dan Wu,et al.  Cross-Modal Collaborative Communications , 2020, IEEE Wireless Communications.

[6]  Alexandre Krupa,et al.  Real-time Teleoperation of Flexible Beveled-tip Needle Insertion using Haptic Force Feedback and 3D Ultrasound Guidance , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[7]  Gilberto Osorio-Gómez,et al.  Haptic Augmented Reality (HapticAR) for assembly guidance , 2019, International Journal on Interactive Design and Manufacturing (IJIDeM).

[8]  Marilena Vendittelli,et al.  Enhancing Force Feedback in Teleoperated Needle Insertion Through On-Line Identification of the Needle-Tissue Interaction Parameters , 2018, 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob).

[9]  Antonio Frisoli,et al.  Wearable Haptics and Immersive Virtual Reality Rehabilitation Training in Children With Neuromotor Impairments , 2018, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[10]  Shuxiang Guo,et al.  The virtual reality simulator-based catheter training system with haptic feedback , 2017, 2017 IEEE International Conference on Mechatronics and Automation (ICMA).

[11]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[12]  Gourishetti Ravali,et al.  Haptic Feedback in Needle Insertion Modeling and Simulation , 2017, IEEE Reviews in Biomedical Engineering.

[13]  Roy George,et al.  The need for virtual reality simulators in dental education: A review , 2017, The Saudi dental journal.

[14]  Eyal Ofek,et al.  NormalTouch and TextureTouch: High-fidelity 3D Haptic Shape Rendering on Handheld Virtual Reality Controllers , 2016, UIST.

[15]  Roland Werthschützky,et al.  Pseudo-Haptic Feedback in Teleoperation , 2016, IEEE Transactions on Haptics.

[16]  Nima Enayati,et al.  Haptics in Robot-Assisted Surgery: Challenges and Benefits , 2016, IEEE Reviews in Biomedical Engineering.

[17]  Daniel J. Wigdor,et al.  Snake Charmer: Physically Enabling Virtual Objects , 2016, TEI.

[18]  Eun-Jung Kim,et al.  Quantification of the parameters of twisting–rotating acupuncture manipulation using a needle force measurement system , 2015, Integrative medicine research.

[19]  Sriram Subramanian,et al.  UltraHaptics: multi-point mid-air haptic feedback for touch surfaces , 2013, UIST.

[20]  Ali Israr,et al.  AIREAL: interactive tactile experiences in free air , 2013, ACM Trans. Graph..

[21]  Seung-Hun Cho,et al.  Does the effect of acupuncture depend on needling sensation and manipulation? , 2013, Complementary therapies in medicine.

[22]  Elena De Momi,et al.  Coaxial Needle Insertion Assistant With Enhanced Force Feedback , 2013, IEEE Transactions on Biomedical Engineering.

[23]  Hossein Mousavi Hondori,et al.  Haptic Augmented Reality to monitor human arm's stiffness in rehabilitation , 2012, 2012 IEEE-EMBS Conference on Biomedical Engineering and Sciences.

[24]  Rajnikant V. Patel,et al.  A Novel Force Modeling Scheme for Needle Insertion Using Multiple Kalman Filters , 2012, IEEE Transactions on Instrumentation and Measurement.

[25]  Martin Hachet,et al.  Toucheo: multitouch and stereo combined in a seamless workspace , 2011, UIST.

[26]  Stephane Cotin,et al.  Constraint-Based Haptic Rendering of Multirate Compliant Mechanisms , 2011, IEEE Transactions on Haptics.

[27]  K Yan,et al.  A real-time prostate cancer detection technique using needle insertion force and patient-specific criteria during percutaneous intervention. , 2009, Medical physics.

[28]  A. Okamura Haptic feedback in robot-assisted minimally invasive surgery , 2009, Current opinion in urology.

[29]  Bernard Bayle,et al.  In Vivo Model Estimation and Haptic Characterization of Needle Insertions , 2007, Int. J. Robotics Res..

[30]  Septimiu E. Salcudean,et al.  Interactive simulation of needle insertion models , 2005, IEEE Transactions on Biomedical Engineering.

[31]  Steven K. Feiner,et al.  Cross-dimensional gestural interaction techniques for hybrid immersive environments , 2005, IEEE Proceedings. VR 2005. Virtual Reality, 2005..

[32]  Robert W. Lindeman,et al.  Towards full-body haptic feedback: the design and deployment of a spatialized vibrotactile feedback system , 2004, VRST '04.

[33]  Matt Adcock,et al.  Haptic collaboration with augmented reality , 2004, SIGGRAPH '04.

[34]  Panadda Marayong,et al.  The effect of visual and haptic feedback on computer-assisted needle insertion , 2004, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[35]  S. Hochreiter,et al.  Long Short-Term Memory , 1997, Neural Computation.

[36]  Grigore C. Burdea,et al.  A Portable Dextrous Master with Force Feedback , 1992, Presence: Teleoperators & Virtual Environments.

[37]  Weiliang Xu,et al.  Multimodal Haptic Display for Virtual Reality: A Survey , 2020, IEEE Transactions on Industrial Electronics.

[38]  Edward Price,et al.  Applied Microsoft Power BI : Bring your data to life! , 2020 .

[39]  Tarun Kanti Podder,et al.  Needle Insertion Force Estimation Model using Procedure-specific and Patient-specific Criteria , 2006, 2006 International Conference of the IEEE Engineering in Medicine and Biology Society.

[40]  Thomas H. Massie,et al.  The PHANToM Haptic Interface: A Device for Probing Virtual Objects , 1994 .