Detecting Out-of-Distribution Inputs to Deep Generative Models Using a Test for Typicality

Recent work has shown that deep generative models can assign higher likelihood to out-of-distribution data sets than to their training data. We posit that this phenomenon is caused by a mismatch between the model's typical set and its areas of high probability density. In-distribution inputs should reside in the former but not necessarily in the latter, as previous work has presumed. To determine whether or not inputs reside in the typical set, we propose a statistically principled, easy-to-implement test using the empirical distribution of model likelihoods. The test is model agnostic and widely applicable, only requiring that the likelihood can be computed or closely approximated. We report experiments showing that our procedure can successfully detect the out-of-distribution sets in several of the challenging cases reported by Nalisnick et al. (2019).

[1]  Václav Smídl,et al.  Are generative deep models for novelty detection truly better? , 2018, ArXiv.

[2]  C. Huber-Carol.  Goodness-of-Fit Tests and Model Validity , 2012 .

[3]  Yee Whye Teh,et al.  Do Deep Generative Models Know What They Don't Know? , 2018, ICLR.

[4]  James J. Little,et al.  Does Your Model Know the Digit 6 Is Not a Cat? A Less Biased Evaluation of "Outlier" Detectors , 2018, ArXiv.

[5]  E. Tabak,et al.  A Family of Nonparametric Density Estimation Algorithms , 2013 .

[6]  D. Darling,et al.  A Test of Goodness of Fit , 1954 .

[7]  Michael Brady,et al.  Novelty detection for the identification of masses in mammograms , 1995 .

[8]  A. Martin-Löf.  On the composition of elementary errors , 1994 .

[9]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[10]  Shakir Mohamed,et al.  Distribution Matching in Variational Inference , 2018, ArXiv.

[11]  David Haussler,et al.  Exploiting Generative Models in Discriminative Classifiers , 1998, NIPS.

[12]  Alexander A. Alemi,et al.  WAIC, but Why? Generative Ensembles for Robust Anomaly Detection , 2018 .

[13]  Qiang Liu,et al.  A Kernelized Stein Discrepancy for Goodness-of-fit Tests , 2016, ICML.

[14]  F. Massey.  The Kolmogorov-Smirnov Test for Goodness of Fit , 1951 .

[15]  Daan Wierstra,et al.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models , 2014, ICML.

[16]  M. Stephens.  EDF Statistics for Goodness of Fit and Some Comparisons , 1974 .

[17]  Anders Høst-Madsen,et al.  Data Discovery and Anomaly Detection Using Atypicality for Real-Valued Data , 2019, Entropy.

[18]  Arthur Gretton,et al.  A Kernel Test of Goodness of Fit , 2016, ICML.

[19]  Robert Tibshirani,et al.  An Introduction to the Bootstrap , 1994 .

[20]  E. Giné,et al.  On the Bootstrap of $U$ and $V$ Statistics , 1992 .

[21]  Prafulla Dhariwal,et al.  Glow: Generative Flow with Invertible 1x1 Convolutions , 2018, NeurIPS.

[22]  Thomas G. Dietterich,et al.  Deep Anomaly Detection with Outlier Exposure , 2018, ICLR.

[23]  Samy Bengio,et al.  Density estimation using Real NVP , 2016, ICLR.

[24]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[25]  Alex Graves,et al.  Conditional Image Generation with PixelCNN Decoders , 2016, NIPS.

[26]  Michael Betancourt,et al.  A Conceptual Introduction to Hamiltonian Monte Carlo , 2017, 1701.02434.

[27]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[28]  Vic Barnett,et al.  Outliers in Statistical Data , 1980 .

[29]  Peter E. Latham,et al.  Zipf’s Law Arises Naturally When There Are Underlying, Unobserved Variables , 2016, PLoS Comput. Biol..

[30]  Roman Vershynin,et al.  High-Dimensional Probability , 2018 .

[31]  F. E. Grubbs.  Procedures for Detecting Outlying Observations in Samples , 1969 .

[32]  Bernhard Schölkopf,et al.  A Kernel Two-Sample Test , 2012, J. Mach. Learn. Res..

[33]  Lucas C. Parra,et al.  Statistical Independence and Novelty Detection with Information Preserving Nonlinear Maps , 1996, Neural Computation.

[34]  David Hinkley,et al.  Bootstrap Methods: Another Look at the Jackknife , 2008 .

[35]  Christopher M. Bishop,et al.  Novelty detection and neural network validation , 1994 .

[36]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[37]  David A. Clifton,et al.  Extending the Generalised Pareto Distribution for Novelty Detection in High-Dimensional Spaces , 2013, J. Signal Process. Syst..

[38]  David A. Clifton,et al.  A review of novelty detection , 2014, Signal Process..

[39]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .