Useful Tools for Statistics and Machine Learning

As much as we would like to have analytical solutions to important problems, it is a fact that many of them are simply too difficult to admit closed-form solutions. Common examples of this phenomenon are finding exact distributions of estimators and statistics, computing the value of an exact optimum procedure, such as a maximum likelihood estimate, and numerous combinatorial algorithms of importance in computer science and applied probability. Unprecedented advances in computing powers and availability have inspired creative new methods and algorithms for solving old problems; often, these new methods are better than what we had in our toolbox before. This chapter provides a glimpse into a few selected computing tools and algorithms that have had a significant impact on the practice of probability and statistics, specifically, the bootstrap, the EM algorithm, and the use of kernels for smoothing and modern statistical classification. The treatment is supposed to be introductory, with references to more advanced parts of the literature.

[1]  Jianqing Fan,et al.  Frontiers in Statistics , 2006 .

[2]  László Györfi,et al.  A Probabilistic Theory of Pattern Recognition , 1996, Stochastic Modelling and Applied Probability.

[3]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[4]  K. Chan,et al.  Monte Carlo EM Estimation for Time Series Models Involving Counts , 1995 .

[5]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[6]  E. Giné,et al.  Necessary Conditions for the Bootstrap of the Mean , 1989 .

[7]  Peter Hall,et al.  A Short Prehistory of the Bootstrap , 2003 .

[8]  M. Chavance.  [Jackknife and bootstrap]. , 1992, Revue d'epidemiologie et de sante publique.

[9]  H. Künsch.  The Jackknife and the Bootstrap for General Stationary Observations , 1989 .

[10]  G. McLachlan,et al.  The EM Algorithm and Extensions: Second Edition , 2008 .

[11]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[12]  P. Hall.  The Bootstrap and Edgeworth Expansion , 1992 .

[13]  D. Freedman,et al.  Some Asymptotic Theory for the Bootstrap , 1981 .

[14]  New York Dover,et al.  ON THE CONVERGENCE PROPERTIES OF THE EM ALGORITHM , 1983 .

[15]  P. Bickel,et al.  Mathematical Statistics: Basic Ideas and Selected Topics , 1977 .

[16]  B. Efron,et al.  Second thoughts on the bootstrap , 2003 .

[17]  M. Aizerman,et al.  Theoretical Foundations of the Potential Function Method in Pattern Recognition Learning , 1964 .

[18]  George Casella,et al.  Implementations of the Monte Carlo EM Algorithm , 2001 .

[19]  M. Rosenblatt.  Remarks on Some Nonparametric Estimates of a Density Function , 1956 .

[20]  Baver Okutmustur.  Reproducing kernel Hilbert spaces , 2005 .

[21]  E. Lehmann.  Elements of large-sample theory , 1998 .

[22]  P. Hall,et al.  On blocking rules for the bootstrap with dependent data , 1995 .

[23]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[24]  E. Cheney.  Analysis for Applied Mathematics , 2001 .

[25]  W. Rudin.  Real and complex analysis , 1968 .

[26]  G. C. Wei,et al.  A Monte Carlo Implementation of the EM Algorithm and the Poor Man's Data Augmentation Algorithms , 1990 .

[27]  P. Hall.  On the Number of Bootstrap Simulations Required to Construct a Confidence Interval , 1986 .

[28]  T. N. Sriram.  Asymptotics in Statistics–Some Basic Concepts , 2002 .

[29]  A. Berlinet,et al.  Reproducing kernel Hilbert spaces in probability and statistics , 2004 .

[30]  Joseph P. Romano,et al.  The stationary bootstrap , 1994 .

[31]  H. White,et al.  Automatic Block-Length Selection for the Dependent Bootstrap , 2004 .

[32]  S. N. Lahiri.  Bootstrap Methods: A Review , 2006 .

[33]  K. Singh,et al.  On the Asymptotic Accuracy of Efron's Bootstrap , 1981 .

[34]  Y. L. Tong.  The multivariate normal distribution , 1989 .

[35]  Kenneth Lange,et al.  Numerical analysis for statisticians , 1999 .

[36]  G. McLachlan,et al.  The EM algorithm and extensions , 1996 .

[37]  Peter Hall,et al.  On efficient bootstrap simulation , 1989 .

[38]  A. Dasgupta.  Asymptotic Theory of Statistics and Probability , 2008 .

[39]  S. Lahiri.  Theoretical comparisons of block bootstrap methods , 1999 .

[40]  J. Mercer.  Functions of Positive and Negative Type, and their Connection with the Theory of Integral Equations , 1909 .

[41]  P. Hall.  Rate of convergence in bootstrap approximations , 1988 .

[42]  K. Athreya.  BOOTSTRAP OF THE MEAN IN THE INFINITE VARIANCE CASE , 1987 .

[43]  E. Carlstein.  The Use of Subseries Values for Estimating the Variance of a General Statistic from a Stationary Sequence , 1986 .

[44]  S. Lahiri.  Resampling Methods for Dependent Data , 2003 .

[45]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[46]  E. L. Lehmann,et al.  Theory of point estimation , 1950 .

[47]  Yuan Yao,et al.  Mercer's Theorem, Feature Maps, and Smoothing , 2006, COLT.

[48]  Ward Cheney,et al.  A course in approximation theory , 1999 .

[49]  J. Shao,et al.  The jackknife and bootstrap , 1996 .

[50]  P. Hall.  Asymptotic Properties of the Bootstrap for Heavy-Tailed Distributions , 1990 .