Spectral Regularization for Support Estimation

In this paper we consider the problem of learning from data the support of a probability distribution when the distribution does not have a density (with respect to some reference measure). We propose a new class of regularized spectral estimators based on a new notion of reproducing kernel Hilbert space, which we call "completely regular ". Completely regular kernels allow to capture the relevant geometric and topological properties of an arbitrary probability space. In particular, they are the key ingredient to prove the universal consistency of the spectral estimators and in this respect they are the analogue of universal kernels for supervised problems. Numerical experiments show that spectral estimators compare favorably to state of the art machine learning algorithms for density support estimation.

[1]  Bruno Pelletier,et al.  Asymptotic Normality in Density Support Estimation , 2009 .

[2]  Ulrike von Luxburg,et al.  A tutorial on spectral clustering , 2007, Stat. Comput..

[3]  S. Smale,et al.  Geometry on Probability Spaces , 2009 .

[4]  J. Williamson.  Harmonic Analysis on Semigroups , 1967 .

[5]  P. Anselone,et al.  Collectively Compact Operator Approximation Theory and Applications to Integral Equations , 1971 .

[6]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[7]  R. Stephenson.  A and V , 1962, The British journal of ophthalmology.

[8]  E. B. Andersen,et al.  Information Science and Statistics , 1986 .

[9]  A. Tsybakov.  On nonparametric estimation of density level sets , 1997 .

[10]  Michael Frazier,et al.  Studies in Advanced Mathematics , 2004 .

[11]  Heiko Hoffmann,et al.  Kernel PCA for novelty detection , 2007, Pattern Recognit..

[12]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[13]  Don R. Hush,et al.  A Classification Framework for Anomaly Detection , 2005, J. Mach. Learn. Res..

[14]  Andreas Christmann,et al.  Support vector machines , 2008, Data Mining and Knowledge Discovery Handbook.

[15]  Bernhard Schölkopf,et al.  Maximal margin classification for metric spaces , 2005, J. Comput. Syst. Sci..

[16]  Bernhard Schölkopf,et al.  Kernel Methods for Implicit Surface Modeling , 2004, NIPS.

[17]  Stephen Smale,et al.  A Topological View of Unsupervised Learning from Noisy Data , 2011, SIAM J. Comput..

[18]  Mikhail Belkin,et al.  Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples , 2006, J. Mach. Learn. Res..

[19]  C. Carmeli,et al.  VECTOR VALUED REPRODUCING KERNEL HILBERT SPACES OF INTEGRABLE FUNCTIONS AND MERCER THEOREM , 2006 .

[20]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[21]  G. Folland.  A course in abstract harmonic analysis , 1995 .

[22]  I. S. Gradshteyn,et al.  Table of Integrals, Series, and Products , 1976 .

[23]  Mikhail Belkin,et al.  On Learning with Integral Operators , 2010, J. Mach. Learn. Res..

[24]  Bernhard Schölkopf,et al.  Estimating the Support of a High-Dimensional Distribution , 2001, Neural Computation.

[25]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[26]  H. Engl,et al.  Regularization of Inverse Problems , 1996 .

[27]  R. Rifkin,et al.  Notes on Regularized Least Squares , 2007 .