We propose a convex optimization approach to solving the nonparametric regression estimation problem when the underlying regression function is Lipschitz continuous. This approach is based on the minimization of the sum of empirical squared errors, subject to the constraints implied by Lipschitz continuity. The resulting optimization problem has a convex objective function and linear constraints, and as a result, is efficiently solvable. The estimated function computed by this technique, is proven to convergeto the underlying regression function uniformly and almost surely, when the sample size grows to infinity, thus providing a very strong form of consistency. Wealso propose a convex optimization approach to the maximum likelihood estimation of unknown parameters in statistical models, where the parameters depend continuously on some observable input variables. For a number of classical distributional forms, the objective function in the underlying optimization problem is convex and the constraints are linear. These problems are, therefore, also efficiently solvable.
[1]
Luc Devroye,et al.
The uniform convergence of nearest neighbor regression function estimators and their application in optimization
,
1978,
IEEE Trans. Inf. Theory.
[2]
David Haussler,et al.
Decision Theoretic Generalizations of the PAC Model for Neural Net and Other Learning Applications
,
1992,
Inf. Comput..
[3]
Peter L. Bartlett,et al.
Efficient agnostic learning of neural networks with bounded fan-in
,
1996,
IEEE Trans. Inf. Theory.
[4]
Vladimir N. Vapnik,et al.
The Nature of Statistical Learning Theory
,
2000,
Statistics for Engineering and Information Science.