Continuous micro finger writing recognition with a commodity smartwatch: demo abstract

Input is a significant problem for wearable devices, particularly for head-mounted virtual and augmented reality systems. Contemporary AR/VR systems use in-air gestures or handheld controllers for interactivity. However, mid-air handwriting provides a natural, subtle, and easy-to-use way to input commands and text. In this demo, we propose and investigate ViFin, a new technique for input commands and text entry which tracks continuous micro finger-level writing with a commodity smartwatch through vibrations. Inspired by the recurrent neural aligner and transfer learning, ViFin recognizes continuous finger writing and works across different users and achieves an accuracy of 90% and 91% for recognizing numbers and letters, respectively. Finally, a real-time writing system with two specific applications using AR smartglasses are implemented.

[1]  Yang Xu,et al.  WiFinger: talk to your smart devices with finger-grained gesture , 2016, UbiComp.

[2]  Marco Gruteser,et al.  HandSense: capacitive coupling-based dynamic, micro finger gesture recognition , 2019, SenSys.

[3]  Anupam Agrawal,et al.  Vision based hand gesture recognition for human computer interaction: a survey , 2012, Artificial Intelligence Review.

[4]  Cheng Zhang,et al.  FingerTrak , 2020, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..