Deterministic Global Optimization with Artificial Neural Networks Embedded

Artificial neural networks are used in various applications for data-driven black-box modeling and subsequent optimization. Herein, we present an efficient method for deterministic global optimization of optimization problems with artificial neural networks embedded. The proposed method is based on relaxations of algorithms using McCormick relaxations in a reduced space (Mitsos et al. in SIAM J Optim 20(2):573–601, 2009) employing the convex and concave envelopes of the nonlinear activation function. The optimization problem is solved using our in-house deterministic global solver. The performance of the proposed method is shown in four optimization examples: an illustrative function, a fermentation process, a compressor plant and a chemical process. The results show that computational solution time is favorable compared to a state-of-the-art global general-purpose optimization solver.

[1]  Ramon E. Moore,et al.  Methods and Applications of Interval Analysis (SIAM Studies in Applied and Numerical Mathematics) (Siam Studies in Applied Mathematics, 2.) , 1979 .

[2]  R. Horst,et al.  Global Optimization: Deterministic Approaches , 1992 .

[3]  N. Sahinidis,et al.  Global optimization of nonconvex NLPs and MINLPs with applications in process design , 1995 .

[4]  Christos T. Maravelias,et al.  Surrogate-Based Process Synthesis , 2010 .

[5]  Dimitri P. Bertsekas,et al.  Convex Optimization Algorithms , 2015 .

[6]  D. Wong,et al.  CHEMICAL PROCESS SYSTEM ENGINEERING , 2016 .

[7]  Nikolaos V. Sahinidis,et al.  Convexification and Global Optimization in Continuous and Mixed-Integer Nonlinear Programming , 2002 .

[8]  Alexander Mitsos,et al.  Tighter McCormick relaxations through subgradient propagation , 2017, Journal of Global Optimization.

[9]  A. Mitsos,et al.  MAiNGO – McCormick-based Algorithm for mixed-integer Nonlinear Global Optimization , 2018 .

[10]  Alexander Mitsos,et al.  Convergence rate of McCormick relaxations , 2012, J. Glob. Optim..

[11]  Paul I. Barton,et al.  McCormick-Based Relaxations of Algorithms , 2009, SIAM J. Optim..

[12]  Christodoulos A. Floudas,et al.  ANTIGONE: Algorithms for coNTinuous / Integer Global Optimization of Nonlinear Equations , 2014, Journal of Global Optimization.

[13]  Sanjeev S. Tambe,et al.  Artificial neural‐network‐assisted stochastic process optimization strategies , 2001 .

[14]  Paul I. Barton,et al.  Differentiable McCormick relaxations , 2016, Journal of Global Optimization.

[15]  Donald R. Jones,et al.  A Taxonomy of Global Optimization Methods Based on Response Surfaces , 2001, J. Glob. Optim..

[16]  Magali R. G. Meireles,et al.  A comprehensive review for industrial applicability of artificial neural networks , 2003, IEEE Trans. Ind. Electron..

[17]  W. Luyben.  Design and Control of the Cumene Process , 2010 .

[18]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[19]  Kaveh Ghorbanian,et al.  An artificial neural network approach to compressor performance prediction , 2009 .

[20]  J. A. Mulder,et al.  Neural Network Output Optimization Using Interval Analysis , 2009, IEEE Transactions on Neural Networks.

[21]  S. Agatonovic-Kustrin,et al.  Basic concepts of artificial neural network (ANN) modeling and its application in pharmaceutical research. , 2000, Journal of pharmaceutical and biomedical analysis.

[22]  Alexander Mitsos,et al.  Deterministic global optimization of process flowsheets in a reduced space using McCormick relaxations , 2017, Journal of Global Optimization.

[23]  Nikolaos V. Sahinidis,et al.  A polyhedral branch-and-cut approach to global optimization , 2005, Math. Program..

[24]  Efstratios N. Pistikopoulos,et al.  A Reduced Space Branch and Bound Algorithm for Global optimization , 1997, J. Glob. Optim..

[25]  Edward M. B. Smith,et al.  Global optimisation of nonconvex MINLPs , 1997 .

[26]  Gilles Trystram,et al.  Interest of neural networks for the optimization of the crossflow filtration process , 1995 .

[27]  Hosahalli S. Ramaswamy,et al.  Modeling and optimization of variable retort temperature (VRT) thermal processing using coupled neural networks and genetic algorithms , 2002 .

[28]  Matthew E. Wilhelm,et al.  Corrections to: Differentiable McCormick relaxations , 2018, Journal of Global Optimization.

[29]  Benoît Chachuat,et al.  Set-Theoretic Approaches in Analysis, Estimation and Control of Nonlinear Systems , 2015 .

[30]  Andy J. Keane,et al.  Recent advances in surrogate-based optimization , 2009 .

[31]  Nikolaos V. Sahinidis,et al.  The ALAMO approach to machine learning , 2017, Comput. Chem. Eng..

[32]  Jure Zupan,et al.  Neural networks in chemistry , 1993 .

[33]  Arne Stolbjerg Drud,et al.  CONOPT - A Large-Scale GRG Code , 1994, INFORMS J. Comput..

[34]  Dieter Kraft,et al.  Algorithm 733: TOMP–Fortran modules for optimal control calculations , 1994, TOMS.

[35]  Cláudio Augusto Oller do Nascimento,et al.  Neural network based approach for optimisation applied to an industrial nylon-6,6 polymerisation process , 1998 .

[36]  Timo Berthold,et al.  Three enhancements for optimization-based bound tightening , 2017, J. Glob. Optim..

[37]  Fabiano A.N. Fernandes,et al.  Optimization of Fischer‐Tropsch Synthesis Using Neural Networks , 2006 .

[38]  Nikolaos V. Sahinidis,et al.  Global optimization of mixed-integer nonlinear programs: A theoretical and computational study , 2004, Math. Program..

[39]  Fabio Schoen,et al.  Global Optimization: Theory, Algorithms, and Applications , 2013 .

[40]  Claudia Gutiérrez-Antonio,et al.  Multiobjective Stochastic Optimization of Dividing-wall Distillation Columns Using a Surrogate Model Based on Neural Networks , 2016 .

[41]  Christos T. Maravelias,et al.  Surrogate‐based superstructure optimization framework , 2011 .

[42]  Tao Chen,et al.  Meta-modelling in chemical process system engineering , 2017 .

[43]  Johanna Kleinekorte,et al.  Techno-economic Optimization of a Green-Field Post-Combustion CO2 Capture Process Using Superstructure and Rate-Based Models , 2016 .

[44]  Rekha S. Singhal,et al.  Comparison of artificial neural network (ANN) and response surface methodology (RSM) in fermentation media optimization: Case study of fermentative production of scleroglucan , 2008 .

[45]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[46]  Matthew D. Stuber,et al.  Generalized McCormick relaxations , 2011, J. Glob. Optim..

[47]  Jari Lewandowski,et al.  Use of Neural Networks in the Simulation and Optimization of Pressure Swing Adsorption Processes , 1998 .

[48]  Matthew D. Stuber,et al.  Convex and concave relaxations of implicit functions , 2015, Optim. Methods Softw..

[49]  J. E. Falk,et al.  An Algorithm for Separable Nonconvex Programming Problems , 1969 .

[50]  Garth P. McCormick,et al.  Computability of global solutions to factorable nonconvex programs: Part I — Convex underestimating problems , 1976, Math. Program..

[51]  Paul I. Barton,et al.  Reverse propagation of McCormick relaxations , 2015, Journal of Global Optimization.

[52]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[53]  Nilay Shah,et al.  An efficient model construction strategy to simulate microalgal lutein photo‐production dynamic process , 2017, Biotechnology and bioengineering.

[54]  Artur M. Schweidtmann,et al.  A Multiobjective Optimization Including Results of Life Cycle Assessment in Developing Biorenewables-Based Processes. , 2017, ChemSusChem.

[55]  Carlos A. Henao.  A superstructure modeling framework for process synthesis using surrogate models , 2012 .

[56]  David C. Miller,et al.  Learning surrogate models for simulation‐based optimization , 2014 .

[57]  Selen Cremaschi,et al.  Process synthesis of biodiesel production plant using artificial neural networks as the surrogate models , 2012, Comput. Chem. Eng..

[58]  Alexander Mitsos,et al.  Convergence analysis of multivariate McCormick relaxations , 2016, J. Glob. Optim..

[59]  Alexander Mitsos,et al.  Erratum to: Multivariate McCormick relaxations , 2017, J. Glob. Optim..

[60]  Nikolaos V. Sahinidis,et al.  A combined first-principles and data-driven approach to model building , 2015, Comput. Chem. Eng..

[61]  Frederico W. Tavares,et al.  Machine learning model and optimization of a PSA unit for methane-nitrogen separation , 2017, Comput. Chem. Eng..

[62]  Alexander Mitsos,et al.  Multivariate McCormick relaxations , 2014, J. Glob. Optim..

[63]  Sanjeev S Tambe,et al.  Genetic Programming Assisted Stochastic Optimization Strategies for Optimization of Glucose to Gluconic Acid Fermentation , 2002, Biotechnology progress.

[64]  Clark A. Mount-Campbell,et al.  Process optimization via neural network metamodeling , 2002 .

[65]  Nikolaos V. Sahinidis,et al.  A branch-and-reduce approach to global optimization , 1996, J. Glob. Optim..

[66]  Roberto Guardani,et al.  Neural network based approach for optimization of industrial chemical processes , 2000 .

[67]  Artur M. Schweidtmann,et al.  Efficient multiobjective optimization employing Gaussian processes, spectral sampling and a genetic algorithm , 2018, Journal of Global Optimization.

[68]  A. Mitsos,et al.  Infeasible Path Global Flowsheet Optimization Using McCormick Relaxations , 2017 .

[69]  Khim Hoong Chu,et al.  Optimization of a fermentation medium using neural networks and genetic algorithms , 2003, Biotechnology Letters.

[70]  G. McCormick.  Nonlinear Programming: Theory, Algorithms and Applications , 1983 .

[71]  Selen Cremaschi,et al.  CFD-Based Optimization of a Flooded Bed Algae Bioreactor , 2013 .

[72]  Anna Witek-Krowiak,et al.  Application of response surface methodology and artificial neural network methods in modelling and optimization of biosorption process. , 2014, Bioresource technology.

[73]  Jorge Otávio Trierweiler,et al.  The Importance of Nominal Operating Point Selection in Self-Optimizing Control , 2016 .

[74]  Mohamed Azlan Hussain,et al.  Review of the applications of neural networks in chemical process control - simulation and online implementation , 1999, Artif. Intell. Eng..

[75]  Wolfgang R. Huster,et al.  Deterministic global optimization of the design of a geothermal organic rankine cycle , 2017 .

[76]  Nando de Freitas,et al.  Taking the Human Out of the Loop: A Review of Bayesian Optimization , 2016, Proceedings of the IEEE.

[77]  Dimitri P. Bertsekas,et al.  Convex Analysis and Optimization , 2003 .

[78]  Paul I. Barton,et al.  Global optimization of bounded factorable functions with discontinuities , 2013, J. Glob. Optim..