On Regularization Schemes for Data-Driven Optimization

This paper presents several regularization schemes for data-driven optimization problems. Data-driven optimization is a challenging task because the model or the distribution describing the inherent stochastic disturbance is unknown but instead only a set of data sampled from it can be available. Most existing literature tackles the data-driven optimization problem by using the approach of distributionally robust optimization which is however an infinite-dimensional optimization problem and thus is required to be transformed into a tractable finite-dimensional optimization problem. As a different line, the regularization schemes presented in this paper contribute to solving the data-driven minimization problem by approximating the unknown expectation with some data-dependent surrogates in a high confidence and by minimizing the surrogates in a tractable way. The tradeoff between the confidence level and the optimization error is analyzed.

[1]  Di Wu,et al.  A Bayesian Risk Approach to Data-driven Stochastic Optimization: Formulations and Asymptotics , 2016, SIAM J. Optim..

[2]  A. Keziou.  Dual representation of Φ-divergences and applications , 2003 .

[3]  A. Owen.  Empirical likelihood ratio confidence intervals for a single functional , 1988 .

[4]  A. Guillin,et al.  On the rate of convergence in Wasserstein distance of the empirical measure , 2013, 1312.2128.

[5]  M. KarthyekRajhaaA.,et al.  Robust Wasserstein profile inference and applications to machine learning , 2019, J. Appl. Probab..

[6]  Xi Chen,et al.  Wasserstein Distributional Robustness and Regularization in Statistical Learning , 2017, ArXiv.

[7]  Henry Lam,et al.  Robust Sensitivity Analysis for Stochastic Systems , 2013, Math. Oper. Res..

[8]  Daniel Kuhn,et al.  Distributionally Robust Logistic Regression , 2015, NIPS.

[9]  Rui Gao.  Distributional Robustness and Regularization in Statistical Learning , 2017 .

[10]  John Duchi,et al.  Statistics of Robust Optimization: A Generalized Empirical Likelihood Approach , 2016, Math. Oper. Res..

[11]  Yongpei Guan,et al.  Data-driven risk-averse stochastic optimization with Wasserstein metric , 2018, Oper. Res. Lett..

[12]  Daniel Kuhn,et al.  A distributionally robust perspective on uncertainty quantification and chance constrained programming , 2015, Mathematical Programming.

[13]  Massimiliano Pontil,et al.  Empirical Bernstein Bounds and Sample-Variance Penalization , 2009, COLT.

[14]  Zhaolin Hu,et al.  Kullback-Leibler divergence constrained distributionally robust optimization , 2012 .

[15]  Huifu Xu,et al.  Convergence Analysis for Distributionally Robust Optimization and Equilibrium Problems , 2016, Math. Oper. Res..

[16]  R. Rockafellar,et al.  Optimization of conditional value-at risk , 2000 .

[17]  Daniel Kuhn,et al.  Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations , 2015, Mathematical Programming.

[18]  John C. Duchi,et al.  Variance-based Regularization with Convex Objectives , 2016, NIPS.

[19]  J. Lynch,et al.  A weak convergence approach to the theory of large deviations , 1997 .

[20]  Alexander Shapiro,et al.  Lectures on Stochastic Programming - Modeling and Theory, Second Edition , 2014, MOS-SIAM Series on Optimization.

[21]  Yinyu Ye,et al.  Distributionally Robust Optimization Under Moment Uncertainty with Application to Data-Driven Problems , 2010, Oper. Res..

[22]  Imre Csiszár,et al.  Systematic stress tests with entropic plausibility constraints , 2013 .

[23]  Melvyn Sim,et al.  Distributionally Robust Optimization and Its Tractable Approximations , 2010, Oper. Res..

[24]  Keith A. Baggerly,et al.  Empirical likelihood as a goodness-of-fit measure , 1998 .

[25]  Michel Broniatowski,et al.  Parametric estimation and tests through divergences and the duality technique , 2008, J. Multivar. Anal..

[26]  Martin J. Wainwright,et al.  Estimating Divergence Functionals and the Likelihood Ratio by Convex Risk Minimization , 2008, IEEE Transactions on Information Theory.

[27]  Anja De Waegenaere,et al.  Robust Solutions of Optimization Problems Affected by Uncertain Probabilities , 2011, Manag. Sci..

[28]  Daniel Kuhn,et al.  Regularization via Mass Transportation , 2017, J. Mach. Learn. Res..

[29]  Güzin Bayraksan,et al.  Data-Driven Stochastic Programming Using Phi-Divergences , 2015 .

[30]  A. Kleywegt,et al.  Distributionally Robust Stochastic Optimization with Wasserstein Distance , 2016, Math. Oper. Res..