Quality Guarantees for Autoencoders via Unsupervised Adversarial Attacks

Autoencoders are an essential concept in unsupervised learning. Currently, the quality of autoencoders is assessed either internally (e.g. based on mean square error) or externally (e.g. by classification performance). Yet, there is no possibility to prove that autoencoders generalize beyond the finite training data, and hence, they are not reliable for safety-critical applications that require formal guarantees also for unseen data. To address this issue, we propose the first framework to bound the worst-case error of an autoencoder within a safety-critical region of an infinite value domain, as well as the definition of unsupervised adversarial examples that cause such worst-case errors. Technically, our framework reduces the infinite search space for a uniform error bound to checking satisfiability of logical formulas in Linear Real Arithmetic. This allows us to leverage highly-optimized SMT solvers, a strategy that is very successful in the context of deductive software verification. We demonstrate our ability to find unsupervised adversarial examples as well as formal quality guarantees both on synthetic and real-world data.

[1]  Pedro M. Domingos,et al.  Adversarial classification , 2004, KDD.

[2]  Zohar Manna,et al.  The calculus of computation - decision procedures with applications to verification , 2007 .

[3]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[4]  Zhaolei Zhang,et al.  A Deep Non-linear Feature Mapping for Large-Margin kNN Classification , 2009, 2009 Ninth IEEE International Conference on Data Mining.

[5]  Clark W. Barrett,et al.  The SMT-LIB Standard Version 2.0 , 2010 .

[6]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[7]  Marc'Aurelio Ranzato,et al.  Building high-level features using large scale unsupervised learning , 2011, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[8]  Alessandro Sperduti,et al.  Pre-training of Recurrent Neural Networks via Linear Autoencoders , 2014, NIPS.

[9]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[10]  Takehisa Yairi,et al.  Anomaly Detection Using Autoencoders with Nonlinear Dimensionality Reduction , 2014, MLSDA'14.

[11]  Jonathon Shlens,et al.  Explaining and Harnessing Adversarial Examples , 2014, ICLR.

[12]  Lovedeep Gondara,et al.  Medical Image Denoising Using Convolutional Denoising Autoencoders , 2016, 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW).

[13]  Antonio Criminisi,et al.  Measuring Neural Net Robustness with Constraints , 2016, NIPS.

[14]  Rüdiger Ehlers,et al.  Formal Verification of Piece-Wise Linear Feed-Forward Neural Networks , 2017, ATVA.

[15]  Mykel J. Kochenderfer,et al.  Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks , 2017, CAV.

[16]  Paul J. Kennedy,et al.  Relational autoencoder for feature extraction , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[17]  Swarat Chaudhuri,et al.  AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation , 2018, 2018 IEEE Symposium on Security and Privacy (SP).

[18]  Ji-Rong Wen,et al.  Unsupervised Adversarial Attacks on Deep Feature-based Retrieval with GAN , 2019, ArXiv.

[19]  Timon Gehr,et al.  Boosting Robustness Certification of Neural Networks , 2018, ICLR.

[20]  Prasant Mohapatra,et al.  Strong Black-box Adversarial Attacks on Unsupervised Machine Learning Models , 2019, ArXiv.

[21]  Sharon Gannot,et al.  Deep Clustering Based On A Mixture Of Autoencoders , 2018, 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP).