An Efficient Convolution Core Architecture for Privacy-Preserving Deep Learning

Cloud service for deep learning (DL) has been widely used except for applications involving medical, financial, or other sensitive data due to privacy and security requirements. By employing homomorphic encryption, trained deep convolutional neural networks (CNNs) can be converted to CryptoNets, which is suitable for privacy-preserving DL cloud service. However, the high computation complexity of CryptoNets leads to tremendous implementation challenge. In this paper, to the best of our knowledge, efficient hardware acceleration of CryptoNets is discussed for the first time in open literature. In more detail, without compromise of security and inference accuracy, encryption parameters and modular multiplication algorithm are carefully selected to reduce the computation complexity of polynomial multiplication. Besides, based on the negative wrapped convolution and fast finite impulse filter schemes, an efficient algorithm for convolutions in CryptoNets is developed. Moreover, a dedicated low complexity convolution core architecture for CryptoNets is proposed and implemented with a 90nm CMOS technology. Compared to a well optimized CPU implementation of CryptoNets, this architecture is 11.9× faster while consuming a power of only 537mW.