Additive models in high dimensions

Additive decompositions are established tools in nonparametric statistics and effectively address the curse of dimensionality. For the analysis of the approximation properties of additive decompositions, we introduce a novel framework which includes the number of variables as an ingredient in the definition of the smoothness of the underlying functions. This approach is motivated by the effect of concentration of measure in high dimensional spaces. Using the resulting smoothness conditions, convergence of the additive decompositions is established. Several examples confirm the error rates predicted by our error bounds. Explicit expressions for optimal additive decompositions (in an $L_2$ sense) are given which can be seen as a generalisation of multivariate Taylor polynomials where the monomials are replaced by higher order interactions. The results can be applied to the numerical approximation of functions with hundreds of variables.

[1]  Karim K. Hirji,et al.  Discovering data mining: from concept to implementation , 1999, SKDD.

[2]  Piotr Indyk,et al.  Approximate nearest neighbors: towards removing the curse of dimensionality , 1998, STOC '98.

[3]  M. Hegland.  Computational challenges in data mining , 2000 .

[4]  G. Wahba.  Spline models for observational data , 1990 .

[5]  Richard Bellman,et al.  Adaptive Control Processes: A Guided Tour , 1961, The Mathematical Gazette.

[6]  Henryk Wozniakowski,et al.  When Are Quasi-Monte Carlo Algorithms Efficient for High Dimensional Integrals? , 1998, J. Complex..

[7]  J. Friedman.  Multivariate adaptive regression splines , 1990 .

[8]  E. Novak,et al.  Foundations of Computational Mathematics: When are integration and discrepancy tractable? , 2001 .

[9]  M. Gromov.  Metric Structures for Riemannian and Non-Riemannian Spaces , 1999 .

[10]  F. J. Hickernell.  Quadrature Error Bounds with Applications to Lattice Rules , 1997 .

[11]  Jonathan Goldstein,et al.  When Is ''Nearest Neighbor'' Meaningful? , 1999, ICDT.

[12]  R. Landau,et al.  Numerical Differentiation , 2019, Numerical Methods.

[13]  Padhraic Smyth,et al.  From Data Mining to Knowledge Discovery: An Overview , 1996, Advances in Knowledge Discovery and Data Mining.

[14]  B. Efron,et al.  The Jackknife Estimate of Variance , 1981 .

[15]  Markus Hegland,et al.  For numerical differentiation, dimensionality can be a blessing! , 1999, Math. Comput..

[16]  Michael J. A. Berry,et al.  Data mining techniques - for marketing, sales, and customer support , 1997, Wiley computer publishing.

[17]  R. Tibshirani,et al.  Generalized additive models for medical research , 1986, Statistical methods in medical research.

[18]  Vladimir Pestov,et al.  On the geometry of similarity search: Dimensionality curse and concentration of measure , 1999, Inf. Process. Lett..

[19]  Jerome H. Friedman.  Multivariate adaptive regression splines (with discussion) , 1991 .

[20]  M. Talagrand.  Concentration of measure and isoperimetric inequalities in product spaces , 1994, math/9406212.

[21]  Vladimir Pestov,et al.  A geometric framework for modelling similarity search , 1999, Proceedings. Tenth International Workshop on Database and Expert Systems Applications. DEXA 99.