Eavesdrop with PoKeMon: Position free keystroke monitoring using acoustic data

Abstract Keyboard input is an important way to enter massive personal information into computing systems and the Internet in our social life. The large amount of acoustic data from the keystroke tapping makes it possible for eavesdroppers to snoop the sensitive and private input remotely. Keystroke monitoring using the acoustic signals, which extracts the user information directly from the keystroke acoustics, is a critical security and privacy issue and has recently attracted much research attention. The existing works use smartphones to collect and infer the keystrokes with various data analysis algorithms. However, most of them rely on the assumption that the phone-keyboard relative position is known in advance, which is often unrealistic for real-world scenarios such as adversary eavesdropping. To address this problem and make keystroke monitoring free to use, in this paper, we propose a Po sition free Ke ystroke Mon itoring scheme using a single smartphone. By exploiting the reliable linguistic accountings and the layout of a certain keyboard, we develop a geometrical approach to estimate the relative positions and angles of the smartphone to the keyboard. Our proposed scheme can self-calibrate the position estimation with the continuous acoustic signals. After that, effective keystroke snooping is established based on the Mel-Frequency Cepstral Coefficient and adaptive k -means clustering. We conduct experiments with Samsung S4 smartphones and simulations. The results show that the positioning scheme achieves a high accuracy by 93%, and the keystroke snooping accuracy is improved by 32.6% compared to the existing works.