Dream-Experiment: A MR User Interface with Natural Multi-channel Interaction for Virtual Experiments

This paper studies a set of MR technologies for middle school experimental teaching environments and develops a multi-channel MR user interface called Dream-Experiment. The goal of Dream-Experiment is to improve the traditional MR user interface, so that users can get a real, natural 3D interactive experience like real experiments, but without danger and pollution. In terms of visual presentation, we design multi-camera collaborative registration to realize robust 6-DoF MR interactive space, and also define a complete rendering pipeline to provide improved processing of virtual-real objects' occlusion including translucent devices. In the virtual-real interaction, we provide six interaction modes that support visual interaction, tangible interaction, virtual-real gestures with touching, voice, thermal feeling, and olfactory feeling. After users' testing, we find that Dream-Experiment has better interactive efficiency and user experience than traditional MR environments.

[1]  Vlatko Cingoski,et al.  A computer aided education system based on augmented reality by immersion to 3-D magnetic field , 2016, 2016 IEEE Conference on Electromagnetic Field Computation (CEFC).

[2]  Mitsunori Makino,et al.  An AR-based support system for learning chemical reaction formula in science of junior high school , 2018, 2018 International Conference on Electronics, Information, and Communication (ICEIC).

[3]  Joseph J. LaViola,et al.  Determining Design Requirements for AR Physics Education Applications , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[4]  Pheng-Ann Heng,et al.  Augmented Reality-Based Personalized Virtual Operative Anatomy for Neurosurgical Guidance and Training , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[5]  David Wells,et al.  Augmenting the learning experience in primary and secondary school education: a systematic review of recent trends in augmented reality game-based learning , 2019, Virtual Reality.

[6]  Vincent Lepetit,et al.  A semi-automatic method for resolving occlusion in augmented reality , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[7]  Yen-Lin Chen,et al.  Edge Snapping-Based Depth Enhancement for Dynamic Occlusion Handling in Augmented Reality , 2016, 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[8]  Rafael Kuffner dos Anjos,et al.  Dynamic Occlusion Handling for Real-Time AR Applications , 2019, VRCAI.

[9]  Ajune Wanis Ismail,et al.  3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review , 2019, IEEE Access.

[10]  Hongbo Fu,et al.  Two‐Finger Gestures for 6DOF Manipulation of 3D Objects , 2012, Comput. Graph. Forum.

[11]  Aljoscha Smolic,et al.  Egocentric Gesture Recognition for Head-Mounted AR Devices , 2018, 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct).

[12]  M. M. Shah,et al.  Occlusion in augmented reality , 2012, 2012 8th International Conference on Information Science and Digital Content Technology (ICIDT2012).

[13]  Dan Xia,et al.  Handling occlusions in augmented reality based on 3D reconstruction method , 2015, Neurocomputing.

[14]  Tobias Isenberg,et al.  Mouse, Tactile, and Tangible Input for 3D Manipulation , 2016, CHI.

[15]  Xu Wang,et al.  A case study of Augmented Reality simulation system application in a chemistry course , 2014, Comput. Hum. Behav..

[16]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Chady Abou Jaoude,et al.  A New Robust Approach for Real-Time Hand Detection and Gesture Recognition , 2017, 2017 International Conference on Computer and Applications (ICCA).

[18]  Fabrizio Lamberti,et al.  Investigating Tangible User Interaction in Mixed-Reality Robotic Games , 2019, 2019 IEEE 9th International Conference on Consumer Electronics (ICCE-Berlin).

[19]  Cheng Wang,et al.  Real-Time Occlusion Handling in Augmented Reality Based on an Object Tracking Approach , 2010, Sensors.

[20]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.

[21]  Dieter Schmalstieg,et al.  Interactive Focus and Context Visualization for Augmented Reality , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[22]  Frédéric Valentin,et al.  MapSense: Multi-Sensory Interactive Maps for Children Living with Visual Impairments , 2016, CHI.

[23]  Jia Chen,et al.  Occlusion handling using moving volume and ray casting techniques for augmented reality systems , 2018, Multimedia Tools and Applications.

[24]  Hyuk-Jae Lee,et al.  Ambidextrous Virtual Keyboard Design with Finger Gesture Recognition , 2018, 2018 IEEE International Symposium on Circuits and Systems (ISCAS).

[25]  Woontack Woo,et al.  An empirical evaluation of virtual hand techniques for 3D object manipulation in a tangible augmented reality environment , 2010, 2010 IEEE Symposium on 3D User Interfaces (3DUI).

[26]  Stefan Müller,et al.  Occlusion Matting: Realistic Occlusion Handling for Augmented Reality Applications , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[27]  Tobias Höllerer,et al.  Real-time hand interaction for augmented reality on mobile phones , 2013, IUI '13.

[28]  Tuck Wah Ng,et al.  Augmented reality experimentation on oxygen gas generation from hydrogen peroxide and bleach reaction , 2018, Biochemistry and molecular biology education : a bimonthly publication of the International Union of Biochemistry and Molecular Biology.

[29]  Danli Wang,et al.  AR-maze: a tangible programming tool for children based on AR technology , 2018, IDC.

[30]  Erhan Ekmekcioglu,et al.  Virtual Transcendence Experiences: Exploring Technical and Design Challenges in Multi-Sensory Environments , 2018, MMVE@MMSys.

[31]  Claudio Kirner,et al.  Using Augmented Reality Artifacts in Education and Cognitive Rehabilitation , 2012 .

[32]  Zhigeng Pan,et al.  A Virtual-real Occlusion Method Based on GPU Acceleration for MR , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[33]  Sandra G. Hart,et al.  Nasa-Task Load Index (NASA-TLX); 20 Years Later , 2006 .

[34]  Huidong Bai,et al.  Natural Gesture Based Interaction for Handheld Augmented Reality , 2013 .

[35]  Hui-Chun Chu,et al.  An Interactive 5E Learning Cycle-Based Augmented Reality System to Improve Students' Learning Achievement in a Microcosmic Chemistry Molecule Course , 2016, 2016 5th IIAI International Congress on Advanced Applied Informatics (IIAI-AAI).

[36]  Xiaoying Tang,et al.  YUE Virtual Park: Multi-sensory experience creative cultural products based on AR technology , 2019, EVA.

[37]  Eugénio C. Ferreira,et al.  Virtual laboratories in (bio)chemical engineering education , 2010 .

[38]  S.-W. Huang,et al.  Applying augmented reality to enhance learning: a study of different teaching materials , 2017, J. Comput. Assist. Learn..

[39]  Antonio Peñalver Benavent,et al.  Evaluating the usability and acceptance of an AR app in learning Chemistry for Secondary Education , 2018, Interacción.

[40]  Anthony Steed,et al.  Accurate real-time occlusion for mixed reality , 2017, VRST.

[41]  Johanna Beyer,et al.  The Hologram in My Hand: How Effective is Interactive Exploration of 3D Visualizations in Immersive Tangible Augmented Reality? , 2018, IEEE Transactions on Visualization and Computer Graphics.

[42]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[43]  Christiane Eichenberg,et al.  Virtual Reality in Psychological, Medical and Pedagogical Applications. , 2012 .

[44]  Eyal Ofek,et al.  Haptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality Experiences , 2016, CHI.

[45]  Hiroshi Okumura,et al.  Development of an evaluation system for upper limb function using AR technology , 2018, GECCO.

[46]  Chandan Dasgupta,et al.  Investigating the Nature of Students' Reasoning in Connecting Molecular Structures of Stereoisomers with their Physical Properties Using an AR App , 2018, 2018 IEEE Tenth International Conference on Technology for Education (T4E).

[47]  Michael Gervautz,et al.  Occlusion in collaborative augmented environments , 1999, Comput. Graph..

[48]  Chung-Ho Su,et al.  A Sustainability Innovation Experiential Learning Model for Virtual Reality Chemistry Laboratory: An Empirical Study with PLS-SEM and IPMA , 2019, Sustainability.

[49]  Rafael Radkowski,et al.  Interactive Hand Gesture-based Assembly for Augmented Reality Applications , 2012, ACHI 2012.

[50]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Xuan Wang,et al.  CoDine: an interactive multi-sensory system for remote dining , 2011, UbiComp '11.