HEAR: Human Action Recognition via Neural Networks on Homomorphically Encrypted Data

Remote monitoring to support “aging in place” is an active area of research. Advanced computer vision technology based on deep learning can provide near real-time home monitoring to detect falling and symptoms related to seizure, and stroke. Affordable webcams, together with cloud computing services (to run machine learning algorithms), can potentially bring significant social and health benefits. However, it has not been deployed in practice because of privacy and security concerns. People may feel uncomfortable sending their videos of daily activities (with potentially sensitive private information) to a computing service provider (e.g., on a commercial cloud). In this paper, we propose a novel strategy to resolve this dilemma by applying fully homomorphic encryption (FHE) to an alternative representation of human actions (i.e., skeleton joints), which guarantees information confidentiality while retaining high-performance action detection at a low cost. We design an FHE-friendly neural network for action recognition and present a secure neural network evaluation strategy to achieve near real-time action detection. Our framework for private inference achieves an 87.99% recognition accuracy (86.21% sensitivity and 99.14% specificity in detecting falls) with a latency of 3.1 seconds on real-world datasets. Our evaluation shows that our elaborated and fine-tuned method reduces the inference latency by 23.81%∼74.67% over a straightforward implementation.