Pattern Recognition and Density Estimation under the General i.i.d. Assumption

Statistical learning theory considers three main problems, pattern recognition, regression and density estimation. This paper studies solvability of these problems (mainly concentrating on pattern recognition and density estimation) in the "high-dimensional" case, where the patterns in the training and test sets are never repeated. We show that, assuming an i.i.d. data source but without any further assumptions, the problems of pattern recognition and regression can often be solved (and there are practically useful algorithms to solve them). On the other hand, the problem of density estimation, as we formalize it, cannot be solved under the general i.i.d. assumption, and additional assumptions are required.

[1]  Alexander Gammerman,et al.  Transduction with Confidence and Credibility , 1999, IJCAI.

[2]  L. Levin,et al.  THE COMPLEXITY OF FINITE OBJECTS AND THE DEVELOPMENT OF THE CONCEPTS OF INFORMATION AND RANDOMNESS BY MEANS OF THE THEORY OF ALGORITHMS , 1970 .

[3]  Alexander Gammerman,et al.  Learning by Transduction , 1998, UAI.

[4]  Alexander Gammerman,et al.  Machine-Learning Applications of Algorithmic Randomness , 1999, ICML.

[5]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[6]  Vladimir Vovk,et al.  Ridge Regression Confidence Machine , 2001, International Conference on Machine Learning.

[7]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[8]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[9]  A. Gammerman,et al.  Bayesian diagnostic probabilities without assuming independence of symptoms. , 1991, Methods of information in medicine.

[10]  V. Vapnik,et al.  Bounds on Error Expectation for Support Vector Machines , 2000, Neural Computation.

[11]  Nello Cristianini,et al.  An introduction to Support Vector Machines , 2000 .

[12]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 2019, Texts in Computer Science.

[13]  Alexander Gammerman,et al.  Transductive Confidence Machines for Pattern Recognition , 2002, ECML.

[14]  Alexander Gammerman,et al.  Computationally Efficient Transductive Machines , 2000, ALT.

[15]  A. McCray,et al.  Yearbook of Medical Informatics , 2013, Yearbook of Medical Informatics.

[16]  V. Vovk,et al.  On the Empirical Validity of the Bayesian Method , 1993 .