CrypTFlow: Secure TensorFlow Inference

We present CrypTFlow, a first of its kind system that converts TensorFlow inference code into Secure Multi-party Computation (MPC) protocols at the push of a button. To do this, we build three components. Our first component, Athos, is an end-to-end compiler from TensorFlow to a variety of semihonest MPC protocols. The second component, Porthos, is an improved semi-honest 3-party protocol that provides significant speedups for TensorFlow like applications. Finally, to provide malicious secure MPC protocols, our third component, Aramis, is a novel technique that uses hardware with integrity guarantees to convert any semi-honest MPC protocol into an MPC protocol that provides malicious security. The malicious security of the protocols output by Aramis relies on integrity of the hardware and semi-honest security of MPC. Moreover, our system matches the inference accuracy of plaintext TensorFlow.We experimentally demonstrate the power of our system by showing the secure inference of real-world neural networks such as ResNet50 and DenseNet121 over the ImageNet dataset with running times of about 30 seconds for semi-honest security and under two minutes for malicious security. Prior work in the area of secure inference has been limited to semi-honest security of small networks over tiny datasets such as MNIST or CIFAR. Even on MNIST/CIFAR, CrypTFlow outperforms prior work.

[1]  Hao Chen,et al.  CHET: an optimizing compiler for fully-homomorphic neural-network inferencing , 2019, PLDI.

[2]  Raluca Ada Popa,et al.  Delphi: A Cryptographic Inference System for Neural Networks , 2020, IACR Cryptol. ePrint Arch..

[3]  Silvio Micali,et al.  A Completeness Theorem for Protocols with Honest Majority , 1987, STOC 1987.

[4]  Marcel Keller,et al.  Secure Evaluation of Quantized Neural Networks , 2019, IACR Cryptol. ePrint Arch..

[5]  Ajith Suresh,et al.  Trident: Efficient 4PC Framework for Privacy Preserving Machine Learning , 2019, IACR Cryptol. ePrint Arch..

[6]  Stratis Ioannidis,et al.  Privacy-Preserving Ridge Regression on Hundreds of Millions of Records , 2013, 2013 IEEE Symposium on Security and Privacy.

[7]  Yuval Ishai,et al.  LevioSA: Lightweight Secure Arithmetic Computation , 2019, CCS.

[8]  Yehuda Lindell,et al.  High-Throughput Semi-Honest Secure Three-Party Computation with an Honest Majority , 2016, IACR Cryptol. ePrint Arch..

[9]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[10]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[11]  Michael Hicks,et al.  Wysteria: A Programming Language for Generic, Mixed-Mode Multiparty Computations , 2014, 2014 IEEE Symposium on Security and Privacy.

[12]  Michael Zohner,et al.  ABY - A Framework for Efficient Mixed-Protocol Secure Two-Party Computation , 2015, NDSS.

[13]  Patrick Traynor,et al.  A Hybrid Approach to Secure Function Evaluation using SGX , 2019, AsiaCCS.

[14]  Yehuda Lindell,et al.  Generalizing the SPDZ Compiler For Other Protocols , 2018, IACR Cryptol. ePrint Arch..

[15]  Payman Mohassel,et al.  Practical Privacy-Preserving K-means Clustering , 2020, IACR Cryptol. ePrint Arch..

[16]  Joan Feigenbaum,et al.  Using Intel Software Guard Extensions for Efficient Two-Party Secure Function Evaluation , 2016, Financial Cryptography Workshops.

[17]  Yixing Lao,et al.  nGraph-HE: a graph compiler for deep learning on homomorphically encrypted data , 2018, IACR Cryptol. ePrint Arch..

[18]  Christian Weinert,et al.  Secure and Private Function Evaluation with Intel SGX , 2019, CCSW@CCS.

[19]  Andrew Chi-Chih Yao,et al.  How to generate and exchange secrets , 1986, 27th Annual Symposium on Foundations of Computer Science (sfcs 1986).

[20]  Yao Lu,et al.  Oblivious Neural Network Predictions via MiniONN Transformations , 2017, IACR Cryptol. ePrint Arch..

[21]  Payman Mohassel,et al.  SecureML: A System for Scalable Privacy-Preserving Machine Learning , 2017, 2017 IEEE Symposium on Security and Privacy (SP).

[22]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Stefan Katzenbeisser,et al.  HyCC: Compilation of Hybrid Protocols for Practical Secure Computation , 2018, CCS.

[24]  Vitaly Shmatikov,et al.  Exploiting Unintended Feature Leakage in Collaborative Learning , 2018, 2019 IEEE Symposium on Security and Privacy (SP).

[25]  Farinaz Koushanfar,et al.  Chameleon: A Hybrid Secure Computation Framework for Machine Learning Applications , 2018, IACR Cryptol. ePrint Arch..

[26]  Marcel Keller,et al.  MP-SPDZ: A Versatile Framework for Multi-Party Computation , 2020, IACR Cryptol. ePrint Arch..

[27]  Dan Bogdanov,et al.  Sharemind: A Framework for Fast Privacy-Preserving Computations , 2008, ESORICS.

[28]  Yuval Ishai,et al.  Outsourcing Private Machine Learning via Lightweight Secure Arithmetic Computation , 2018, ArXiv.

[29]  Jonathan Katz,et al.  Global-Scale Secure Multiparty Computation , 2017, CCS.

[30]  Aseem Rastogi,et al.  EzPC: Programmable and Efficient Secure Two-Party Computation for Machine Learning , 2019, 2019 IEEE European Symposium on Security and Privacy (EuroS&P).

[31]  Michael Naehrig,et al.  CryptoNets: applying neural networks to encrypted data with high throughput and accuracy , 2016, ICML 2016.

[32]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[33]  Keith B. Frikken.  Secure multiparty computation , 2010 .

[34]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Paolo D'Arco,et al.  Secure Two-Party Computation: A Visual Way , 2013, ICITS.

[36]  Bo Chen,et al.  Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[37]  Uli Schell.  Keras/Tensorflow , 2022, Maschinelles Lernen mit R.

[38]  Christos Gkantsidis,et al.  VC3: Trustworthy Data Analytics in the Cloud Using SGX , 2015, 2015 IEEE Symposium on Security and Privacy.

[39]  Morten Dahl,et al.  Private Machine Learning in TensorFlow using Secure Computation , 2018, ArXiv.

[40]  Peter Rindal,et al.  ABY3: A Mixed Protocol Framework for Machine Learning , 2018, IACR Cryptol. ePrint Arch..

[41]  Dan Boneh,et al.  Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware , 2018, ICLR.

[42]  Farinaz Koushanfar,et al.  XONN: XNOR-based Oblivious Deep Neural Network Inference , 2019, IACR Cryptol. ePrint Arch..

[43]  Daniel Rueckert,et al.  A generic framework for privacy preserving deep learning , 2018, ArXiv.

[44]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[45]  Sebastian Nowozin,et al.  Oblivious Multi-Party Machine Learning on Trusted Processors , 2016, USENIX Security Symposium.

[46]  Patrick Traynor,et al.  Frigate: A Validated, Extensible, and Efficient Compiler and Interpreter for Secure Computation , 2016, 2016 IEEE European Symposium on Security and Privacy (EuroS&P).

[47]  Ion Stoica,et al.  Opaque: An Oblivious and Encrypted Distributed Analytics Platform , 2017, NSDI.

[48]  Jonathan Katz,et al.  Authenticated Garbling and Efficient Maliciously Secure Two-Party Computation , 2017, CCS.

[49]  Jonathan Katz,et al.  Secure Multi-Party Computation of Boolean Circuits with Applications to Privacy in On-Line Marketplaces , 2012, CT-RSA.

[50]  Vivek Seshadri,et al.  Compiling KB-sized machine learning models to tiny IoT devices , 2019, PLDI.

[51]  Donald Beaver,et al.  Efficient Multiparty Protocols Using Circuit Randomization , 1991, CRYPTO.

[52]  Benny Pinkas,et al.  Fairplay - Secure Two-Party Computation System , 2004, USENIX Security Symposium.

[53]  Michael I. Schwartzbach,et al.  A domain-specific programming language for secure multiparty computation , 2007, PLAS '07.

[54]  Deian Stefan,et al.  Information-Flow Control for Programming on Encrypted Data , 2012, 2012 IEEE 25th Computer Security Foundations Symposium.

[55]  Forrest N. Iandola,et al.  SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size , 2016, ArXiv.

[56]  Ran Canetti,et al.  Security and Composition of Multiparty Cryptographic Protocols , 2000, Journal of Cryptology.

[57]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[58]  Stratis Ioannidis,et al.  Privacy-preserving matrix factorization , 2013, CCS.

[59]  Shafi Goldwasser,et al.  Machine Learning Classification over Encrypted Data , 2015, NDSS.

[60]  Amir Salman Avestimehr,et al.  CodedPrivateML: A Fast and Privacy-Preserving Framework for Distributed Machine Learning , 2019, IEEE Journal on Selected Areas in Information Theory.

[61]  Ashish Choudhury,et al.  ASTRA: High Throughput 3PC over Rings with Application to Secure Prediction , 2019, IACR Cryptol. ePrint Arch..

[62]  Sameer Wagh,et al.  SecureNN: 3-Party Secure Computation for Neural Network Training , 2019, Proc. Priv. Enhancing Technol..

[63]  Jonathan Katz,et al.  Optimizing Authenticated Garbling for Faster Secure Two-Party Computation , 2018, IACR Cryptol. ePrint Arch..

[64]  Helmut Veith,et al.  Secure two-party computations in ANSI C , 2012, CCS.

[65]  Vitaly Shmatikov,et al.  Chiron: Privacy-preserving Machine Learning as a Service , 2018, ArXiv.

[66]  Alfred V. Aho,et al.  Compilers: Principles, Techniques, and Tools (2nd Edition) , 2006 .

[67]  Vladimir Kolesnikov,et al.  Scalable Private Set Union from Symmetric-Key Techniques , 2019, IACR Cryptol. ePrint Arch..

[68]  Matt J. Kusner,et al.  QUOTIENT: Two-Party Secure Neural Network Training and Prediction , 2019, CCS.

[69]  Ahmad-Reza Sadeghi,et al.  Secure Multiparty Computation from SGX , 2017, Financial Cryptography.

[70]  Fan Zhang,et al.  Sealed-Glass Proofs: Using Transparent Enclaves to Prove and Sell Knowledge , 2017, 2017 IEEE European Symposium on Security and Privacy (EuroS&P).

[71]  Anantha Chandrakasan,et al.  Gazelle: A Low Latency Framework for Secure Neural Network Inference , 2018, IACR Cryptol. ePrint Arch..

[72]  Lars Ailo Bongo,et al.  Reproduction study using public data of: Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs , 2018, PloS one.

[73]  Michael Zohner,et al.  Ad-Hoc Secure Two-Party Computation on Mobile Devices using Hardware Tokens , 2014, USENIX Security Symposium.

[74]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[75]  Dan Boneh,et al.  Prio: Private, Robust, and Scalable Computation of Aggregate Statistics , 2017, NSDI.

[76]  Markus Nagel,et al.  Data-Free Quantization Through Weight Equalization and Bias Correction , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[77]  Ion Stoica,et al.  Helen: Maliciously Secure Coopetitive Learning for Linear Models , 2019, 2019 IEEE Symposium on Security and Privacy (SP).

[78]  Rosario Cammarota,et al.  nGraph-HE2: A High-Throughput Framework for Neural Network Inference on Encrypted Data , 2019, IACR Cryptol. ePrint Arch..

[79]  Ahmad-Reza Sadeghi,et al.  Automated Synthesis of Optimized Circuits for Secure Computation , 2015, CCS.

[80]  Kartik Nayak,et al.  ObliVM: A Programming Framework for Secure Computation , 2015, 2015 IEEE Symposium on Security and Privacy.

[81]  Moti Yung,et al.  On Deploying Secure Computing Commercially: Private Intersection-Sum Protocols and their Business Applications , 2019, IACR Cryptol. ePrint Arch..

[82]  Maria Zhdanova,et al.  Time to Rethink: Trust Brokerage Using Trusted Execution Environments , 2015, TRUST.