A systematic comparison of machine learning methods for modeling of dynamic processes applied to combustion emission rate modeling

Abstract Ten established, data-driven dynamic algorithms are surveyed and a practical guide for understanding these methods generated. Existing Python programming packages for implementing each algorithm are acknowledged, and the model equations necessary for prediction are presented. A case study on a coal-fired power plant’s NOx emission rates is performed, directly comparing each modeling method’s performance on a mutual system. Each model is evaluated by its root mean squared error (RMSE) on out-of-sample future horizon predictions. Optimal hyperparameters are identified using either an exhaustive search or genetic algorithm. The top five model structures of each method are used to recursively predict future NOx emission rates over a 60-step time horizon. The RMSE at each future timestep is determined, and the recursive output prediction trends compared against measurements in time. The GRU neural network is identified as the best candidate for representing the system, demonstrating accurate and stable predictions across the future horizon by all considered models, while satisfactory performance was observed in several of the ARX/NARX formulations. These efforts have contributed 1) a concise resource of multiple proven dynamic machine learning methods, 2) a practical guide explaining the use of these methods, effectively lowering the “barrier-to-entry” of deploying such models in control systems, 3) a comparison study evaluating each method’s performance on a mutual system, 4) demonstration of accurate multi-timestep emissions modeling suitable for systems-level control, and 5) generalizable results demonstrating the suitability of each method for prediction over a multi-step future horizon to other complex dynamic systems.

[1]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[2]  Thomas Martinetz,et al.  The Intrinsic Recurrent Support Vector Machine , 2007, ESANN.

[3]  Fang Wang,et al.  Prediction of NOX emission for coal-fired boilers based on deep belief network , 2018, Control Engineering Practice.

[4]  Yao Wang,et al.  Generalized Recurrent Neural Network accommodating Dynamic Causal Modeling for functional MRI analysis , 2018, NeuroImage.

[5]  Gang Chen,et al.  Modeling and reduction of NOX emissions for a 700 MW coal-fired boiler with the advanced machine learning method , 2016 .

[6]  K. Cen,et al.  Combining Support Vector Regression and Ant Colony Optimization to Reduce NOx Emissions in Coal-Fired Utility Boilers , 2008 .

[7]  D. W. Pershing,et al.  Influence of coal composition on the fate of volatileand char nitrogen during combustion , 1982 .

[8]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[9]  T. Fletcher,et al.  Evaluation of NOx-Reduction Measures for Iron-Ore Rotary Kilns , 2020, Energy & Fuels.

[10]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[11]  Jin Zhao,et al.  Discriminant deep belief network for high-resolution SAR image classification , 2017, Pattern Recognit..

[12]  Heesung Yoon,et al.  Comparative Evaluation of ANN- and SVM-Time Series Models for Predicting Freshwater-Saltwater Interface Fluctuations , 2017 .

[13]  Xingpeng Jiang,et al.  Inference of microbial interactions from time series data using vector autoregression model , 2013, 2013 IEEE International Conference on Bioinformatics and Biomedicine.

[14]  Emine Ayaz,et al.  Elman's recurrent neural network applications to condition monitoring in nuclear power plant and rotating machinery , 2003 .

[15]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[16]  Andrea Pigorini,et al.  Multivariate autoregressive models with exogenous inputs for intracerebral responses to direct electrical stimulation of the human brain , 2012, Front. Hum. Neurosci..

[17]  Y. Zayko SOLUTION OF NP-COMPLETE PROBLEMS ON THE LANDAUER'S COMPUTER , 2010 .

[18]  Nurlaila Ismail,et al.  Model Predictive Control using ARX model for steam distillation essential oil extraction system , 2010, 2010 International Conference on Intelligent and Advanced Systems.

[19]  Rao Tatavarti,et al.  Forecasting Air Quality Index Using an Ensemble of Artificial Neural Networks and Regression Models , 2019, J. Intell. Syst..

[20]  Luis A. Aguirre,et al.  Modeling Nonlinear Dynamics and Chaos: A Review , 2009 .

[21]  Ron Cass,et al.  Adaptive Process Optimization using Functional-Link Networks and Evolutionary Optimization , 1996 .

[22]  Primož Potočnik,et al.  Multi-step-ahead prediction of NOx emissions for a coal-based boiler , 2013 .

[23]  Hao Li,et al.  Approximation techniques for dynamic real-time optimization (DRTO) of distributed MPC systems , 2018, Comput. Chem. Eng..

[24]  Kang Li,et al.  Modelling and Prediction of NOx Emission in a Coal-Fired Power Generation Plant , 2004 .

[25]  Chris Tofallis,et al.  A better measure of relative prediction accuracy for model selection and model estimation , 2014, J. Oper. Res. Soc..

[26]  Erik Marchi,et al.  A novel approach for automatic acoustic novelty detection using a denoising autoencoder with bidirectional LSTM neural networks , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[27]  A. E. Eiben,et al.  Introduction to Evolutionary Computing , 2003, Natural Computing Series.

[28]  Jung-Lung Hsu,et al.  Microstructural white matter changes in normal aging: A diffusion tensor imaging study with higher-order polynomial regression models , 2010, NeuroImage.

[29]  W. Fuller,et al.  Distribution of the Estimators for Autoregressive Time Series with a Unit Root , 1979 .

[30]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[31]  Fi-John Chang,et al.  A nonlinear spatio-temporal lumping of radar rainfall for modeling multi-step-ahead inflow forecasts by data-driven techniques , 2016 .

[32]  Jingge Song,et al.  A globally enhanced general regression neural network for on-line multiple emissions prediction of utility boiler , 2017, Knowl. Based Syst..

[33]  Morten Christian Melaaen,et al.  Kiln Process Impact of Alternative Solid Fuel Combustion in the Cement Kiln Main Burner – Mathematical Modelling and Full-Scale Experiment , 2014 .

[34]  Kody M. Powell,et al.  On-line classification of coal combustion quality using nonlinear SVM for improved neural network NOx emission rate prediction , 2020, Comput. Chem. Eng..

[35]  Abdul Razak Hamdan,et al.  A solution representation of genetic algorithm for neural network weights and structure , 2016, Inf. Process. Lett..

[36]  Kody M. Powell,et al.  Dynamic modeling and optimization of a coal-fired utility boiler to forecast and minimize NOx and CO emissions simultaneously , 2019, Comput. Chem. Eng..

[37]  A. W. Coats,et al.  Kinetic Parameters from Thermogravimetric Data , 1964, Nature.

[38]  Mohammad Pooyan,et al.  Genetic algorithm-optimised structure of convolutional neural network for face recognition applications , 2016, IET Comput. Vis..

[39]  Adrià Giménez,et al.  Speaker-Adapted Confidence Measures for ASR Using Deep Bidirectional Recurrent Neural Networks , 2018, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[40]  Yaoqi Zhou,et al.  Improving protein disorder prediction by deep bidirectional long short‐term memory recurrent neural networks , 2016, Bioinform..

[41]  Jian Hou,et al.  Recent advances on SVM based fault diagnosis and process monitoring in complicated industrial processes , 2016, Neurocomputing.

[42]  Kody M. Powell,et al.  Using Real-Time Electricity Prices to Leverage Electrical Energy Storage and Flexible Loads in a Smart Grid Environment Utilizing Machine Learning Techniques , 2019 .

[43]  Kody M. Powell,et al.  Sustainable NOx emission reduction at a coal-fired power station through the use of online neural network modeling and particle swarm optimization , 2019 .

[44]  Panagiotis D. Christofides,et al.  Real-Time Adaptive Machine-Learning-Based Predictive Control of Nonlinear Processes , 2020, Industrial & Engineering Chemistry Research.

[45]  Jing Jing Li Li,et al.  Study on Prediction Model of HIV Incidence Based on GRU Neural Network Optimized by MHPSO , 2020, Ieee Access.

[46]  Robert J. Marks,et al.  Neural Smithing: Supervised Learning in Feedforward Artificial Neural Networks , 1999 .

[47]  G. Richards,et al.  Ultra-Low NOX Integrated System for Coal Fired Power Plants , 2004 .

[48]  T. Chai,et al.  Root mean square error (RMSE) or mean absolute error (MAE)? – Arguments against avoiding RMSE in the literature , 2014 .

[49]  Alessia Amelio,et al.  Time Series Vector Autoregression Prediction of the Ecological Footprint based on Energy Parameters , 2019, ArXiv.

[50]  R. Sioshansi,et al.  A vector autoregression weather model for electricity supply and demand modeling , 2018 .

[51]  Sen Zhang,et al.  Prediction model of permeability index for blast furnace based on the improved multi-layer extreme learning machine and wavelet transform , 2017, J. Frankl. Inst..

[52]  James A. Miller,et al.  Modeling nitrogen chemistry in combustion , 2018, Progress in Energy and Combustion Science.

[53]  Kody M. Powell,et al.  Dynamic Optimization of a Hybrid Solar Thermal and Fossil Fuel System , 2014 .

[54]  Bing Zhang,et al.  Applications of Recurrent Neural Network Language Model in Offline Handwriting Recognition and Word Spotting , 2014, 2014 14th International Conference on Frontiers in Handwriting Recognition.

[55]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[56]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[57]  Aijun Liu,et al.  Abnormal Breast Detection in Mammogram Images by Feed-forward Neural Network Trained by Jaya Algorithm , 2017, Fundam. Informaticae.

[58]  Bana M. Abuzayed,et al.  Long range dependence in an emerging stock market’s sectors: volatility modelling and VaR forecasting , 2018 .

[59]  Christian Igel,et al.  Training restricted Boltzmann machines: An introduction , 2014, Pattern Recognit..

[60]  George Eastman House,et al.  Sparse Bayesian Learning and the Relevan e Ve tor Ma hine , 2001 .

[61]  Jörg Tiedemann,et al.  Character-based Joint Segmentation and POS Tagging for Chinese using Bidirectional RNN-CRF , 2017, IJCNLP.

[62]  Zheng Yao,et al.  A new approach for function approximation in boiler combustion optimization based on modified structural AOSVR , 2009, Expert Syst. Appl..

[63]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[64]  Ying-Wong Cheung,et al.  Lag Order and Critical Values of the Augmented Dickey-Fuller Test , 1995 .

[65]  Geoffrey E. Hinton Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[66]  Ali Abbas,et al.  The hybrid MPC-MINLP algorithm for optimal operation of coal-fired power plants with solvent based post-combustion CO2 capture , 2017 .

[67]  Hao Zhou,et al.  Modeling and optimization of the NOx emission characteristics of a tangentially fired boiler with artificial neural networks , 2004 .

[68]  Georgios A. Keliris,et al.  Introduction to Research Topic – Binocular Rivalry: A Gateway to Studying Consciousness , 2012, Front. Hum. Neurosci..

[69]  Yang Liu,et al.  Optimizing combustion of coal fired boilers for reducing NOx emission using Gaussian Process , 2018, Energy.

[70]  Skipper Seabold,et al.  Statsmodels: Econometric and Statistical Modeling with Python , 2010, SciPy.

[71]  Kody M. Powell,et al.  Real-time optimization of multi-cell industrial evaporative cooling towers using machine learning and particle swarm optimization , 2020 .

[72]  Gang Chen,et al.  Dynamic modeling of NOX emission in a 660 MW coal-fired boiler with long short-term memory , 2019, Energy.

[73]  Jingge Song,et al.  Improved artificial bee colony-based optimization of boiler combustion considering NOX emissions, heat rate and fly ash recycling for on-line applications , 2016 .

[74]  Zsolt Dobó,et al.  Investigation of co-milling Utah bituminous coal with prepared woody biomass materials in a Raymond Bowl Mill , 2018 .

[75]  Kody M. Powell,et al.  Analysis of a thermal generator’s participation in the Western Energy Imbalance Market and the resulting effects on overall performance and emissions , 2019, The Electricity Journal.

[76]  Boqiang Lin,et al.  Growth of industrial CO2 emissions in Shanghai city: Evidence from a dynamic vector autoregression analysis , 2018 .

[77]  Olivier Juan,et al.  Using a genetic algorithm and CFD to identify low NOx configurations in an industrial boiler , 2015 .

[78]  Jianhua Yan,et al.  Models for the combustion of single solid fuel particles in fluidized beds: A review , 2017 .

[79]  Meng Zhang,et al.  Neural Network Methods for Natural Language Processing , 2017, Computational Linguistics.

[80]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.

[81]  J. Sutherland,et al.  An evaluation of the efficacy of various coal combustion models for predicting char burnout , 2017 .

[82]  Soowoong Kim,et al.  Deep belief network based statistical feature learning for fingerprint liveness detection , 2016, Pattern Recognit. Lett..

[83]  William Stafford Noble,et al.  Machine learning applications in genetics and genomics , 2015, Nature Reviews Genetics.

[84]  Kun Yu,et al.  Short-term Power Load Forecasting of Residential Community Based on GRU Neural Network , 2018, 2018 International Conference on Power System Technology (POWERCON).

[85]  Xuanyu Liu,et al.  Earth pressure balance control of shield tunneling machine based on nonlinear least squares support vector machine model predictive control , 2018, Measurement and Control.

[86]  R Halter,et al.  Energy from Waste--clean, efficient, renewable: transitions in combustion efficiency and NOx control. , 2013, Waste management.

[87]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[88]  M. Ye,et al.  Developing a Long Short-Term Memory (LSTM) based model for predicting water table depth in agricultural areas , 2018, Journal of Hydrology.

[89]  Prasad Krishna,et al.  Back propagation genetic and recurrent neural network applications in modelling and analysis of squeeze casting process , 2017, Appl. Soft Comput..

[90]  Geoffrey E. Hinton A Practical Guide to Training Restricted Boltzmann Machines , 2012, Neural Networks: Tricks of the Trade.

[91]  Jeremy N. Thornock,et al.  Application of LES-CFD for predicting pulverized-coal working conditions after installation of NOx control system , 2018, Energy.

[92]  Marc'Aurelio Ranzato,et al.  DeViSE: A Deep Visual-Semantic Embedding Model , 2013, NIPS.

[93]  Kody M. Powell,et al.  Nonlinear modeling, estimation and predictive control in APMonitor , 2014, Comput. Chem. Eng..

[94]  Lance D. Chambers Practical handbook of genetic algorithms , 1995 .

[95]  Zhenhao Tang,et al.  The multi-objective optimization of combustion system operations based on deep data-driven models , 2019, Energy.

[96]  Edwin Lughofer,et al.  Virtual Quality control using bidirectional LSTM networks and gradient boosting , 2019, 2019 IEEE 17th International Conference on Industrial Informatics (INDIN).

[97]  Yueting Zhuang,et al.  Fusing Geometric Features for Skeleton-Based Action Recognition Using Multilayer LSTM Networks , 2018, IEEE Transactions on Multimedia.

[98]  A. Jensen,et al.  Coupling thermal deactivation with oxidation for predicting the combustion of a solid fuel , 2001 .

[99]  Kody M. Powell,et al.  Development of a roadmap for dynamic process intensification by using a dynamic, data-driven optimization approach , 2019 .

[100]  Mustafa Unel,et al.  Estimating Soot Emission in Diesel Engines Using Gated Recurrent Unit Networks , 2019, IFAC-PapersOnLine.

[101]  Jizhen Liu,et al.  A novel least squares support vector machine ensemble model for NOx emission prediction of a coal-fired boiler , 2013 .

[102]  Tao Zhang,et al.  Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges , 2018, IEEE Signal Processing Magazine.

[103]  Diego Klabjan,et al.  Online Adaptive Machine Learning Based Algorithm for Implied Volatility Surface Modeling , 2017, Knowl. Based Syst..

[104]  B. R. Mehta,et al.  Advanced process control systems , 2015 .

[105]  Hao Zhou,et al.  Computational intelligence approach for NOx emissions minimization in a coal-fired utility boiler , 2010 .

[106]  Yitao Liu,et al.  Deep belief network based deterministic and probabilistic wind speed forecasting approach , 2016 .

[107]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[108]  Jiajun Zhang,et al.  Deep Neural Networks in Machine Translation: An Overview , 2015, IEEE Intelligent Systems.

[109]  K. Kyprianidis,et al.  Waste Fuel Combustion: Dynamic Modeling and Control , 2018, Processes.

[110]  Xinli Li,et al.  Prediction of the NO emissions from thermal power plant using long-short term memory neural network , 2020 .

[111]  Mohamed El Merouani,et al.  Predicting financial distress using hybrid feedforward neural network with cuckoo search algorithm , 2020, ANT/EDI40.

[112]  Kody M. Powell,et al.  Dynamic Real-Time Optimization of Air-Conditioning Systems in Residential Houses with a Battery Energy Storage under Different Electricity Pricing Structures , 2018 .

[113]  Georg Heigold,et al.  Multilingual acoustic models using distributed deep neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[114]  Li Li,et al.  Using LSTM and GRU neural network methods for traffic flow prediction , 2016, 2016 31st Youth Academic Annual Conference of Chinese Association of Automation (YAC).

[115]  M. A. Henson Announcing the 2018 Processes Travel Award for Post-Doctoral Fellows and Ph.D. Students , 2018 .

[116]  Feng Wu,et al.  Combining support vector regression and cellular genetic algorithm for multi-objective optimization of coal-fired utility boilers , 2009 .

[117]  K. Gnana Sheela,et al.  Review on Methods to Fix Number of Hidden Neurons in Neural Networks , 2013 .

[118]  Kuldip K. Paliwal,et al.  Bidirectional recurrent neural networks , 1997, IEEE Trans. Signal Process..

[119]  Xiao Wu,et al.  Data-Driven Modeling and Predictive Control for Boiler–Turbine Unit , 2013 .

[120]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[121]  Md. Khaledur Rahman,et al.  Transit network design by genetic algorithm with elitism , 2014 .