Photonic Differential Privacy with Direct Feedback Alignment

Optical Processing Units (OPUs) – low-power photonic chips dedicated to large scale random projections – have been used in previous work to train deep neural networks using Direct Feedback Alignment (DFA), an effective alternative to backpropagation. Here, we demonstrate how to leverage the intrinsic noise of optical random projections to build a differentially private DFA mechanism, making OPUs a solution of choice to provide a private-by-design training. We provide a theoretical analysis of our adaptive privacy mechanism, carefully measuring how the noise of optical random projections propagates in the process and gives rise to provable Differential Privacy. Finally, we conduct experiments demonstrating the ability of our learning procedure to achieve solid end-task performance.

[1]  A. Boes,et al.  11 TOPS photonic convolutional accelerator for optical neural networks , 2021, Nature.

[2]  David Bol,et al.  Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks , 2019 .

[3]  Vitaly Shmatikov,et al.  Differential Privacy Has Disparate Impact on Model Accuracy , 2019, NeurIPS.

[4]  Sara Hooker,et al.  Moving beyond “algorithmic bias is a data problem” , 2021, Patterns.

[5]  Ilya Mironov,et al.  Rényi Differential Privacy , 2017, 2017 IEEE 30th Computer Security Foundations Symposium (CSF).

[6]  The dynamics of learning with feedback alignment , 2020, ArXiv.

[7]  Mikhail V. Konnik,et al.  High-level numerical simulations of noise in CCD and CMOS photosensors: review and tutorial , 2014, ArXiv.

[8]  Arild Nøkland,et al.  Direct Feedback Alignment Provides Learning in Deep Neural Networks , 2016, NIPS.

[9]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[10]  Jonathan Dong,et al.  Kernel Computations from Large-Scale Random Features Obtained by Optical Processing Units , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[11]  Cynthia Dwork,et al.  Calibrating Noise to Sensitivity in Private Data Analysis , 2006, TCC.

[12]  G. Crooks.  On Measures of Entropy and Information , 2015 .

[13]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[14]  Yu-Xiang Wang,et al.  Subsampled Rényi Differential Privacy and Analytical Moments Accountant , 2018, AISTATS.

[15]  L. Pardo.  Statistical Inference Based on Divergence Measures , 2005 .

[16]  Florent Krzakala,et al.  Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment , 2020, ArXiv.

[17]  F. Krzakala,et al.  Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures , 2020, NeurIPS.

[18]  Colin J. Akerman,et al.  Random synaptic feedback weights support error backpropagation for deep learning , 2016, Nature Communications.

[19]  Bhavani M. Thuraisingham,et al.  Privacy Preserving Synthetic Data Release Using Deep Learning , 2018, ECML/PKDD.

[20]  Daniel Kifer,et al.  Differentially Private Deep Learning with Direct Feedback Alignment , 2020, ArXiv.

[21]  H. Vincent Poor,et al.  MVG Mechanism: Differential Privacy under Matrix-Valued Query , 2018, CCS.

[22]  Shanhui Fan,et al.  Training of Photonic Neural Networks through In Situ Backpropagation , 2018, 2019 Conference on Lasers and Electro-Optics (CLEO).

[23]  Úlfar Erlingsson,et al.  Scalable Private Learning with PATE , 2018, ICLR.

[24]  Emiliano De Cristofaro,et al.  Differentially Private Mixture of Generative Neural Networks , 2017, 2017 IEEE International Conference on Data Mining (ICDM).

[25]  Dawn Xiaodong Song,et al.  Towards Practical Differential Privacy for SQL Queries , 2017, Proc. VLDB Endow..

[26]  Yu-Xiang Wang,et al.  Improving the Gaussian Mechanism for Differential Privacy: Analytical Calibration and Optimal Denoising , 2018, ICML.

[27]  Ian Goodfellow,et al.  Deep Learning with Differential Privacy , 2016, CCS.

[28]  Cynthia Dwork,et al.  Differential Privacy: A Survey of Results , 2008, TAMC.

[29]  Gordon Wetzstein,et al.  Inference in artificial intelligence with deep optics and photonics , 2020, Nature.

[30]  Oliver Kosut,et al.  Three Variants of Differential Privacy: Lossless Conversion and Applications , 2021, IEEE Journal on Selected Areas in Information Theory.

[31]  A. I. Lvovsky,et al.  End-to-end optical backpropagation for training neural networks , 2019, ArXiv.

[32]  Hyunghoon Cho,et al.  Emerging technologies towards enhancing privacy in genomic data sharing , 2019, Genome Biology.