Fighting COVID-19 in the Dark: Methodology for Improved Inference Using Homomorphically Encrypted DNN

Privacy-preserving deep neural network (DNN) inference is a necessity in different regulated industries such as healthcare, finance, and retail. Recently, homomorphic encryption (HE) has been used as a method to enable analytics while addressing privacy concerns. HE enables secure predictions over encrypted data. However, there are several challenges related to the use of HE, including DNN size limitations and the lack of support for some operation types. Most notably, the commonly used ReLU activation is not supported under some HE schemes. We propose a structured methodology to replace ReLU with a quadratic polynomial activation. To address the accuracy degradation issue, we use a pre-trained model that trains another HE-friendly model, using techniques such as "trainable activation" functions and knowledge distillation. We demonstrate our methodology on the AlexNet architecture, using the chest X-Ray and CT datasets for COVID-19 detection. Our experiments show that by using our approach, the gap between the F1 score and accuracy of the models trained with ReLU and the HE-friendly model is narrowed down to within a mere 1.1 - 5.3 percent degradation.

[1]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.

[2]  Antonio J. Peña,et al.  Enabling Homomorphically Encrypted Inference for Large DNN Models , 2021, IEEE Transactions on Computers.

[3]  Melek Önen,et al.  FHE-Compatible Batch Normalization for Privacy Preserving Deep Learning , 2018, DPM/CBT@ESORICS.

[4]  Jianping Gou,et al.  Knowledge Distillation: A Survey , 2020, International Journal of Computer Vision.

[5]  Francesco Piazza,et al.  Neural networks with digital LUT activation functions , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[6]  Serkan Kiranyaz,et al.  Exploring the effect of image enhancement techniques on COVID-19 detection using chest X-ray images , 2020, Computers in Biology and Medicine.

[7]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[8]  Danilo Comminiello,et al.  Learning activation functions from data using cubic spline interpolation , 2020, Neural Advances in Processing Nonlinear Dynamic Signals.

[9]  Michael Naehrig,et al.  CryptoNets: applying neural networks to encrypted data with high throughput and accuracy , 2016, ICML 2016.

[10]  Stephen M. Moore,et al.  The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository , 2013, Journal of Digital Imaging.

[11]  Hassan Takabi,et al.  CryptoDL: Deep Neural Networks over Encrypted Data , 2017, ArXiv.

[12]  Sharath Pankanti,et al.  Towards Deep Neural Network Training on Encrypted Data , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[13]  Arnold Schönhage,et al.  A Lower Bound for the Length of Addition Chains , 1975, Theor. Comput. Sci..

[14]  Margarita Vald,et al.  On the Privacy of Protocols based on CPA-Secure Homomorphic Encryption , 2021, IACR Cryptol. ePrint Arch..

[15]  Alexander Wong,et al.  COVID-Net: a tailored deep convolutional neural network design for detection of COVID-19 cases from chest X-ray images , 2020, Scientific reports.

[16]  Mamun Bin Ibne Reaz,et al.  Can AI Help in Screening Viral and COVID-19 Pneumonia? , 2020, IEEE Access.

[17]  Payman Mohassel,et al.  SecureML: A System for Scalable Privacy-Preserving Machine Learning , 2017, 2017 IEEE Symposium on Security and Privacy (SP).

[18]  Alexander Wong,et al.  COVIDNet-CT: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases From Chest CT Images , 2020, Frontiers in Medicine.

[19]  Craig Gentry,et al.  Fully homomorphic encryption using ideal lattices , 2009, STOC '09.

[20]  Ming Xian,et al.  PPolyNets: Achieving High Prediction Accuracy and Efficiency With Parametric Polynomial Activations , 2018, IEEE Access.

[21]  Craig Gentry,et al.  (Leveled) fully homomorphic encryption without bootstrapping , 2012, ITCS '12.

[22]  Craig Gentry,et al.  Computing arbitrary functions of encrypted data , 2010, CACM.

[23]  Jung Hee Cheon,et al.  Homomorphic Encryption for Arithmetic of Approximate Numbers , 2017, ASIACRYPT.

[24]  Seyed Iman Mirzadeh,et al.  Improved Knowledge Distillation via Teacher Assistant , 2020, AAAI.

[25]  Geoffrey C. Fox,et al.  Glyph: Fast and Accurately Training Deep Neural Networks on Encrypted Data , 2019, NeurIPS.

[26]  T. Giordano,et al.  The Health Insurance Portability and Accountability Act of 1996 (HIPAA) privacy rule: implications for clinical research. , 2006, Annual review of medicine.

[27]  Jung Hee Cheon,et al.  Numerical Methods for Comparison on Homomorphically Encrypted Numbers , 2019, IACR Cryptol. ePrint Arch..

[28]  Ming Zhang,et al.  Neuron-adaptive higher order neural-network models for automated financial data modeling , 2002, IEEE Trans. Neural Networks.

[29]  Pramod Kumar Meher.  An optimized lookup-table for the evaluation of sigmoid function for artificial neural networks , 2010, 2010 18th IEEE/IFIP International Conference on VLSI and System-on-Chip.

[30]  Joseph Paul Cohen,et al.  COVID-19 Image Data Collection: Prospective Predictions Are the Future , 2020, ArXiv.

[31]  Jong-Seon No,et al.  Privacy-Preserving Machine Learning With Fully Homomorphic Encryption for Deep Neural Network , 2021, IEEE Access.

[32]  Jeff Druce,et al.  Privacy preserving Neural Network Inference on Encrypted Data with GPUs , 2019, ArXiv.