Compensating data shortages in manufacturing with monotonicity knowledge

We present a regression method for enhancing the predictive power of a model by exploiting expert knowledge in the form of shape constraints, or more specifically, monotonicity constraints. Incorporating such information is particularly useful when the available data sets are small or do not cover the entire input space, as is often the case in manufacturing applications. We set up the regression subject to the considered monotonicity constraints as a semi-infinite optimization problem, and we propose an adaptive solution algorithm. The method is conceptually simple, applicable in multiple dimensions and can be extended to more general shape constraints. It is tested and validated on two real-world manufacturing processes, namely laser glass bending and press hardening of sheet metal. It is found that the resulting models both comply well with the expert's monotonicity knowledge and predict the training data accurately. The suggested approach leads to lower root-mean-squared errors than comparative methods from the literature for the sparse data sets considered in this work.

[1]  Elijah Polak,et al.  Optimization: Algorithms and Consistent Approximations , 1997 .

[2]  H. Wan,et al.  Least Squares Isotonic Regression in Two Dimensions , 2003 .

[3]  V. Chernozhukov,et al.  Massachusetts Institute of Technology Department of Economics Working Paper Series Improving Point and Interval Estimates of Monotone Functions by Rearrangement Improving Point and Interval Estimates of Monotone Functions by Rearrangement , 2022 .

[4]  Arnold Neumaier,et al.  Introduction to Numerical Analysis , 2001 .

[5]  H. D. Brunk,et al.  Statistical inference under order restrictions : the theory and application of isotonic regression , 1973 .

[6]  Joseph Mendola,et al.  From the Good , 2014 .

[7]  David B. Dunson,et al.  Bayesian monotone regression using Gaussian process projection , 2013, 1306.4041.

[8]  Nikolaos V. Sahinidis,et al.  The ALAMO approach to machine learning , 2017, Comput. Chem. Eng..

[9]  Christopher K. I. Williams,et al.  Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning) , 2005 .

[10]  E. Mammen,et al.  A General Projection Framework for Constrained Smoothing , 2001 .

[11]  Tor Arne Johansen,et al.  Identification of non-linear systems using empirical data and prior knowledge - an optimization approach , 1996, Autom..

[12]  Olvi L. Mangasarian,et al.  Nonlinear Knowledge in Kernel Approximation , 2007, IEEE Transactions on Neural Networks.

[13]  Michael Bortz,et al.  Gray-Box Modeling for the Optimization of Chemical Processes , 2018, Chemie Ingenieur Technik.

[14]  Olvi L. Mangasarian,et al.  Nonlinear Knowledge-Based Classification , 2008, IEEE Transactions on Neural Networks.

[15]  Raoul Heese,et al.  Some Aspects of Combining Data and Models in Process Engineering , 2020 .

[16]  Chih-Chuan Chen,et al.  Incorporating monotonic domain knowledge in support vector learning for data mining regression problems , 2019, Neural Computing and Applications.

[17]  H. Dette,et al.  A simple nonparametric estimator of a strictly monotone regression function , 2006 .

[18]  Reimund Neugebauer,et al.  Press hardening - An innovative and challenging technology , 2012 .

[19]  Holger Dette,et al.  Strictly monotone and smooth nonparametric regression for two or more variables , 2005 .

[20]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[21]  Lukas Morand,et al.  The Good, the Bad and the Ugly: Augmenting a black-box model with expert knowledge , 2019, ICANN.

[22]  William Wright,et al.  Visual Classification: Expert Knowledge Guides Machine Learning , 2010, IEEE Computer Graphics and Applications.

[23]  Zhang Liu,et al.  Interior-point methods for large-scale cone programming , 2011 .

[24]  Rainer Hettich,et al.  Numerische Methoden der Approximation und semi-infiniten Optimierung , 1982 .

[25]  Geurt Jongbloed,et al.  Nonparametric Estimation under Shape Constraints , 2014 .

[26]  Nataliya I. Kalashnykova,et al.  Bilevel Programming Problems , 2015 .

[27]  Gérard Bloch,et al.  Incorporating prior knowledge in support vector regression , 2007, Machine Learning.

[28]  Klaus Neumann,et al.  RELIABLE INTEGRATION OF CONTINUOUS CONSTRAINTS INTO EXTREME LEARNING MACHINES , 2013 .

[29]  William F. Eddy,et al.  An Algorithm for Isotonic Regression on Ordered Rectangular Grids , 1996 .

[30]  R. Estrada,et al.  Introduction to the Theory of Distributions , 1994 .

[31]  Jürgen Neugebauer,et al.  Applications for curved glass in buildings , 2014 .

[32]  Enno Mammen,et al.  Estimating a Smooth Monotone Regression Function , 1991 .

[33]  J. Mixter Fast , 2012 .

[34]  Jochen Schmid Approximation, characterization, and continuity of multivariate monotonic regression functions , 2020, ArXiv.

[35]  Thomas G. Dietterich,et al.  Learning from Sparse Data by Exploiting Monotonicity Constraints , 2005, UAI.

[36]  Birgit Kirsch,et al.  Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems , 2019 .

[37]  Nikolaos V. Sahinidis,et al.  Automated learning of chemical reaction networks , 2019, Comput. Chem. Eng..

[38]  Carl Sandrock,et al.  A simplicial homology algorithm for Lipschitz optimisation , 2018, Journal of Global Optimization.

[39]  Wojciech Kotlowski,et al.  Rule learning with monotonicity constraints , 2009, ICML '09.

[40]  Tobias Rist,et al.  Feasibility of bent glasses with small bending radii , 2018, ce/papers.

[41]  Oliver Stein,et al.  Bi-Level Strategies in Semi-Infinite Programming , 2003 .

[42]  Jan Schwientek,et al.  A transformation-based discretization method for solving general semi-infinite optimization problems , 2020 .

[43]  Quentin F. Stout,et al.  Isotonic Regression via Partitioning , 2013, Algorithmica.

[44]  Aki Vehtari,et al.  Gaussian processes with monotonicity information , 2010, AISTATS.

[45]  P. Hall,et al.  NONPARAMETRIC KERNEL REGRESSION SUBJECT TO MONOTONICITY CONSTRAINTS , 2001 .

[46]  Donald Goldfarb,et al.  A numerically stable dual method for solving strictly convex quadratic programs , 1983, Math. Program..

[47]  Quentin F. Stout,et al.  Isotonic Regression for Multiple Independent Variables , 2015, Algorithmica.

[48]  Stefan Wrobel,et al.  A review of machine learning for the optimization of production processes , 2019, The International Journal of Advanced Manufacturing Technology.

[49]  R. Landel,et al.  The Temperature Dependence of Relaxation Mechanisms in Amorphous Polymers and Other Glass-Forming Liquids , 1955 .

[50]  Gérard Bloch,et al.  Incorporating prior knowledge in support vector machines for classification: A review , 2008, Neurocomputing.

[51]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[52]  P. Alam ‘A’ , 2021, Composites Engineering: An A–Z Guide.

[53]  Nikolaos V. Sahinidis,et al.  A combined first-principles and data-driven approach to model building , 2015, Comput. Chem. Eng..

[54]  Marco A. López,et al.  Semi-infinite programming , 2007, Eur. J. Oper. Res..

[55]  Oliver Stein,et al.  How to solve a semi-infinite optimization problem , 2012, Eur. J. Oper. Res..

[56]  Hari Mukerjee,et al.  Monotone Nonparametric Regression , 1988 .

[57]  J. E. Falk,et al.  Infinitely constrained optimization problems , 1976 .