A Proximal Algorithm for Estimating the Regularized Wavelet-Based Density-Difference

Density-Difference (DD) estimation is an important unsupervised learning procedure that proceeds many regression methods. The present work details a novel method for estimating the Difference of Densities (DoD) between two distributions. This new method directly calculates the DD, in the form of a wavelet expansion, without the need for explicitly reconstructing individual distributions. Furthermore, the method applies a regularization technique that utilizes both l2 and l1 norm penalties to robustly estimate the coefficients of the wavelet expansion. Optimizing the regularized objective is accomplished via a Proximal Gradient Descent (PGD) approach. Thus, we term our method Regularized Wavelet-based Density-Difference (RWDD) with PGD. On extensive simulated datasets, from complex multimodal to skewed distributions, our method demonstrated superior performance in comparison to other contemporary techniques.

[1]  M. Zenga,et al.  Bank loan recovery rates: Measuring and nonparametric density estimation , 2010 .

[2]  Masashi Sugiyama,et al.  Change-Point Detection in Time-Series Data by Direct Density-Ratio Estimation , 2009, SDM.

[3]  John R. Hershey,et al.  Approximating the Kullback Leibler Divergence Between Gaussian Mixture Models , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[4]  Meng Wang,et al.  Semi-supervised kernel density estimation for video annotation , 2009, Comput. Vis. Image Underst..

[5]  Ingrid Daubechies,et al.  Ten Lectures on Wavelets , 1992 .

[6]  Anand Rangarajan,et al.  Maximum Likelihood Wavelet Density Estimation With Applications to Image and Shape Matching , 2008, IEEE Transactions on Image Processing.

[7]  M. Vannucci.  Nonparametric Density Estimation using Wavelets , 1995 .

[8]  Daniel T. Larose,et al.  Discovering Knowledge in Data: An Introduction to Data Mining , 2005 .

[9]  Hyunjoong Kim,et al.  Functional Analysis I , 2017 .

[10]  M. C. Jones,et al.  Comparison of Smoothing Parameterizations in Bivariate Kernel Density Estimation , 1993 .

[11]  Miss A.O. Penney.  (b) , 1974, The New Yale Book of Quotations.

[12]  David J. Hand,et al.  Intelligent Data Analysis: An Introduction , 2005 .

[13]  Jun Yan,et al.  Kernel Density Estimation of traffic accidents in a network space , 2008, Comput. Environ. Urban Syst..

[14]  Mark Moyou,et al.  Change Detection for Streaming Data Using Wavelet-Based Least Squares Density–Difference , 2018, Contributions to Statistics.

[15]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[16]  P. J. Green,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[17]  Nouna Kettaneh,et al.  Statistical Modeling by Wavelets , 1999, Technometrics.

[18]  Sung-Hyuk Cha.  Comprehensive Survey on Distance/Similarity Measures between Probability Density Functions , 2007 .

[19]  Stéphane Mallat,et al.  A Theory for Multiresolution Signal Decomposition: The Wavelet Representation , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  Anthony O. Smith,et al.  A Majorization-Minimization Algorithm for Estimating the Regularized Wavelet-Based Density-Difference , 2018, 2018 International Conference on Computational Science and Computational Intelligence (CSCI).

[21]  A. Izenman.  Review Papers: Recent Developments in Nonparametric Density Estimation , 1991 .

[22]  Motoaki Kawanabe,et al.  Direct Importance Estimation with Model Selection and Its Application to Covariate Shift Adaptation , 2007, NIPS.

[23]  Bernard W. Silverman,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[24]  KawaharaYoshinobu,et al.  Sequential change-point detection based on direct density-ratio estimation , 2012 .

[25]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[26]  James G. Scott,et al.  Proximal Algorithms in Statistics and Machine Learning , 2015, ArXiv.

[27]  Trevor Hastie,et al.  Statistical Learning with Sparsity: The Lasso and Generalizations , 2015 .

[28]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[29]  Takafumi Kanamori,et al.  Density-Difference Estimation , 2012, Neural Computation.

[30]  장윤희,et al.  Y. , 2003, Industrial and Labor Relations Terms.

[31]  Masashi Sugiyama,et al.  Sequential change‐point detection based on direct density‐ratio estimation , 2012, Stat. Anal. Data Min..