MagHacker: eavesdropping on stylus pen writing via magnetic sensing from commodity mobile devices

Stylus pens have been widely used with today's mobile devices to provide a convenient handwriting input method, but also bring a unique security vulnerability that may unveil the user's handwriting contents to a nearby eavesdropper. In this paper, we present MagHacker, a new sensing system that realizes such eavesdropping attack over commodity mobile devices, which monitor and analyze the magnetic field being produced by the stylus pen's internal magnet. MagHacker divides the continuous magnetometer readings into small segments that represent individual letters, and then translates these readings into writing trajectories for letter recognition. Experiment results over realistic handwritings from multiple human beings demonstrate that MagHacker can accurately eavesdrop more than 80% of handwriting with stylus pens, from a distance of 10cm. Only slight degradation in such accuracy is produced when the eavesdropping distance or the handwriting speed increases. MagHacker is highly energy efficient, and can well adapt to different stylus pen models and environmental contexts.

[1]  M. Kraichman.  Handbook of electromagnetic propagation in conducting media , 1970 .

[2]  Richard Kronland-Martinet,et al.  Reading and Understanding Continuous Wavelet Transforms , 1989 .

[3]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[4]  Shigeki Sagayama,et al.  Pen pressure features for writer-independent on-line handwriting recognition based on substroke HMM , 2002, Object recognition supported by user interaction for service robots.

[5]  Xunnian Yang,et al.  Efficient circular arc interpolation based on active tolerance control , 2002, Comput. Aided Des..

[6]  Lianwen Jin,et al.  A Novel Vision based Finger-writing Character Recognition System , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[7]  Lianwen Jin,et al.  A Novel Vision-Based Finger-Writing Character Recognition System , 2007, J. Circuits Syst. Comput..

[8]  H. Seki,et al.  Wearable handwriting input device using magnetic field , 2007, SICE Annual Conference 2007.

[9]  D. Bub,et al.  Features for Identification of Uppercase and Lowercase Letters , 2008, Psychological science.

[10]  John Krumm,et al.  Exploring end user preferences for location obfuscation, location-based services, and the value of location , 2010, UbiComp.

[11]  Chien-Cheng Lee,et al.  Fingertip-Writing Alphanumeric Character Recognition for Vision-Based Human Computer Interaction , 2010, 2010 International Conference on Broadband, Wireless Computing, Communication and Applications.

[12]  Sean White,et al.  uTrack: 3D input using two magnetic sensors , 2013, UIST.

[13]  Wang Jeen-Shing,et al.  Online Handwriting Recognition Using an Accelerometer-Based Pen Device , 2013, CSE 2013.

[14]  Eric C. Larson,et al.  DopLink: using the doppler effect for multi-device interaction , 2013, UbiComp.

[15]  Yunhao Liu,et al.  Shake and walk: Acoustic direction finding and fine-grained indoor localization using smartphones , 2014, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[16]  Sang Ho Yoon,et al.  TMotion: Embedded 3D Mobile Input using Magnetic Sensing Technique , 2015, UIST.

[17]  Jin Zhang,et al.  MIO: Enhancing Wireless Communications Security Through Physical Layer Multiple Inter-Symbol Obfuscation , 2015, IEEE Transactions on Information Forensics and Security.

[18]  Debi Prosad Dogra,et al.  Segmentation and recognition of text written in 3D using Leap motion interface , 2015, 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR).

[19]  Sangki Yun,et al.  Turning a Mobile Device into a Mouse in the Air , 2015, MobiSys.

[20]  Satoshi Naoi,et al.  Beyond human recognition: A CNN-based framework for handwritten character recognition , 2015, 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR).

[21]  Buntarou Shizuki,et al.  Input Techniques to the Surface around a Smartphone using a Magnet Attached on a Stylus , 2016, CHI Extended Abstracts.

[22]  Biing-Hwang Juang,et al.  Air-Writing Recognition—Part II: Detection and Recognition of Writing Activity in Continuous Stream of Motion Data , 2016, IEEE Transactions on Human-Machine Systems.

[23]  Liang Tong,et al.  A hierarchical edge cloud architecture for mobile computing , 2016, IEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer Communications.

[24]  Klara Nahrstedt,et al.  WritingHacker: audio based eavesdropping of handwriting via mobile devices , 2016, UbiComp.

[25]  Wen-Yo Lee,et al.  A Vision-Based Fingertip-Writing Character Recognition System , 2016 .

[26]  Shashidhar Patil,et al.  Handwriting Recognition in Free Space Using WIMU-Based Hand Motion Analysis , 2016, J. Sensors.

[27]  Xiaohui Liang,et al.  When CSI Meets Public WiFi: Inferring Your Mobile Phone Password via WiFi Signals , 2016, CCS.

[28]  Shwetak N. Patel,et al.  Finexus: Tracking Precise Motions of Multiple Fingertips Using Magnetic Sensing , 2016, CHI.

[29]  Gregory D. Abowd,et al.  SynchroWatch: One-Handed Synchronous Smartwatch Gestures Using Correlation and Magnetic Sensing , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[30]  Victor Carbune,et al.  Multi-Language Online Handwriting Recognition , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Wei Gao,et al.  Interconnecting heterogeneous devices in the personal mobile cloud , 2017, IEEE INFOCOM 2017 - IEEE Conference on Computer Communications.

[32]  Feng Xiao,et al.  PatternListener: Cracking Android Pattern Lock Using Acoustic Signals , 2018, CCS.

[33]  Yuan Feng,et al.  MotionHacker: Motion sensor based eavesdropping on handwriting via smartwatch , 2018, IEEE INFOCOM 2018 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS).

[34]  Panlong Yang,et al.  WordRecorder: Accurate Acoustic-based Handwriting Recognition Using Deep Learning , 2018, IEEE INFOCOM 2018 - IEEE Conference on Computer Communications.

[35]  Panlong Yang,et al.  WritePad: Consecutive Number Writing on Your Hand With Smart Acoustic Sensing , 2018, IEEE Access.

[36]  Sunanda Dixit,et al.  HMM Segmentation Approach for Offline Cursive Handwritten Words , 2018 .

[37]  Hao Jiang,et al.  Motion Eavesdropper: Smartwatch-based Handwriting Recognition Using Deep Learning , 2019, ICMI.

[38]  Yingying Chen,et al.  An Indirect Eavesdropping Attack of Keystrokes on Touch Screen through Acoustic Sensing , 2021, IEEE Transactions on Mobile Computing.