Dual-stage attention-based LSTM for simulating performance of brackish water treatment plant

Abstract The remarkable increment in the demand for freshwater in water-resource-stressed regions increases the necessity of saltwater desalination and the application of a brackish water treatment plant (BWTP). In that respect, model-based process analysis can play an essential role in optimizing BWTP operation and maintenance (O&M) and reducing costs. In modeling, it is challenging for either theoretical or numerical methods to sufficiently account for the complex causality and various correlations among the numerous process parameters or variables in the BWTP system. Contrastively, deep learning approaches are capable of modeling such a BWTP system as it can describe the complexity and nonlinearity of its variables with robust autonomous learning. In this study, we modeled an RO unit process of BWTP using conventional long short-term memory (Conv-LSTM) and dual-stage attention-based LSTM (DA-LSTM) based on hourly time-series data obtained from the actual BWTP operation during a one-year period. Hyperparameter optimization for Conv-LSTM and DA-LSTM was individually conducted to enhance the model prediction performance. The model prediction results demonstrated the superiority of DA-LSTM (R2 > 0.99) over Conv-LSTM (0.531 ≤ R2 ≤ 0.884). The sensitivity analysis offered straightforward interpretations of how the attention mechanisms of DA-LSTM used time-series data of the model input and output parameters for prediction.

[1]  Willi Gujer,et al.  Data-driven modeling approaches to support wastewater treatment plant operation , 2012, Environ. Model. Softw..

[2]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[3]  J. Georgiadis,et al.  Science and technology for water purification in the coming decades , 2008, Nature.

[4]  Jun Xie,et al.  Neural Machine Translation With GRU-Gated Attention Model , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[5]  Hung-yi Lee,et al.  Temporal pattern attention for multivariate time series forecasting , 2018, Machine Learning.

[6]  I. Wright,et al.  Increased Water Pollution After Closure of Australia’s Longest Operating Underground Coal Mine: a 13-Month Study of Mine Drainage, Water Chemistry and River Ecology , 2018, Water, Air, & Soil Pollution.

[7]  Amy McGovern,et al.  Making the Black Box More Transparent: Understanding the Physical Implications of Machine Learning , 2019, Bulletin of the American Meteorological Society.

[8]  Ibrahim S. Al-Mutaz,et al.  Development of a steady-state mathematical model for MEE-TVC desalination plants , 2014 .

[9]  Chris Eliasmith,et al.  Hyperopt: a Python library for model selection and hyperparameter optimization , 2015 .

[10]  Mika Liukkonen,et al.  Cluster analysis by self-organizing maps: An application to the modelling of water quality in a treatment process , 2013, Appl. Soft Comput..

[11]  Yue-Shan Chang,et al.  Air Pollution Forecasting Using RNN with LSTM , 2018, 2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress(DASC/PiCom/DataCom/CyberSciTech).

[12]  Mu-En Wu,et al.  Applied attention-based LSTM neural networks in stock prediction , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[13]  Abdel M. Bayoumi,et al.  Development of a Predictive Maintenance 4.0 Platform: Enhancing Product Design and Manufacturing , 2020 .

[14]  G. H. Huang,et al.  Mathematical Modeling for Water Quality Management under Interval and Fuzzy Uncertainties , 2013, J. Appl. Math..

[15]  Lei Li,et al.  Recent advances in artificial intelligence and machine learning for nonlinear relationship analysis and process control in drinking water treatment: A review , 2021, Chemical Engineering Journal.

[16]  A. Elmidaoui,et al.  Prediction of salt rejection by nanofiltration and reverse osmosis membranes using Spiegler-Kedem model and an optimisation procedure , 2018 .

[17]  Yu Lou,et al.  Forecasting Stock Prices Using a Hybrid Deep Learning Model Integrating Attention Mechanism, Multi-Layer Perceptron, and Bidirectional Long-Short Term Memory Neural Network , 2020, IEEE Access.

[18]  Bart De Moor,et al.  Hyperparameter tuning in Python using Optunity , 2014 .

[19]  Y. Shao,et al.  Residual networks with multi-attention mechanism for hyperspectral image classification , 2021, Arabian Journal of Geosciences.

[20]  Myungjoo Kang,et al.  Financial series prediction using Attention LSTM , 2019, ArXiv.

[21]  P. M. Biesheuvel,et al.  Water desalination using capacitive deionization with microporous carbon electrodes. , 2012, ACS applied materials & interfaces.

[22]  Yibin Li,et al.  The Influence of the Activation Function in a Convolution Neural Network Model of Facial Expression Recognition , 2020, Applied Sciences.

[23]  Eui Jin Hwang,et al.  Development and Validation of Deep Learning-based Automatic Detection Algorithm for Malignant Pulmonary Nodules on Chest Radiographs. , 2019, Radiology.

[24]  Tianrui Li,et al.  Multivariate time series forecasting via attention-based encoder-decoder framework , 2020, Neurocomputing.

[25]  Ulf Jeppsson,et al.  Benchmarking biological nutrient removal in wastewater treatment plants: influence of mathematical model assumptions. , 2012, Water science and technology : a journal of the International Association on Water Pollution Research.

[26]  Frank Hutter,et al.  Speeding Up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves , 2015, IJCAI.

[27]  Tao Yu,et al.  Generative Adversarial Networks Based on Collaborative Learning and Attention Mechanism for Hyperspectral Image Classification , 2020, Remote. Sens..

[28]  Yoshua Bengio,et al.  Fine-grained attention mechanism for neural machine translation , 2018, Neurocomputing.

[29]  Vivienne Sze,et al.  Efficient Processing of Deep Neural Networks: A Tutorial and Survey , 2017, Proceedings of the IEEE.

[30]  Yuteng Xiao,et al.  A dual‐stage attention‐based Conv‐LSTM network for spatio‐temporal correlation and multivariate time series prediction , 2021, Int. J. Intell. Syst..

[31]  G. Nabi,et al.  The crisis of water shortage and pollution in Pakistan: risk to public health, biodiversity, and ecosystem , 2019, Environmental Science and Pollution Research.

[32]  Grgoire Montavon,et al.  Neural Networks: Tricks of the Trade , 2012, Lecture Notes in Computer Science.

[33]  F. D. Owa,et al.  Water Pollution: Sources, Effects, Control and Management , 2013, International Letters of Natural Sciences.

[34]  H. Nagy,et al.  Prediction of Sediment Load Concentration in Rivers using Artificial Neural Network Model , 2002 .

[35]  Lutz Prechelt,et al.  Early Stopping - But When? , 2012, Neural Networks: Tricks of the Trade.

[36]  Junqiang Wang,et al.  Time-series well performance prediction based on Long Short-Term Memory (LSTM) neural network model , 2020 .

[37]  Yanjun Zhang,et al.  The Zhacang thermal field, Qinghai Province, China: its geology, geophysics, chemistry, and conceptual model , 2021, Arabian Journal of Geosciences.

[38]  Jun Zhang,et al.  Time series prediction using RNN in multi-dimension embedding phase space , 1998, SMC'98 Conference Proceedings. 1998 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.98CH36218).

[39]  Duo Zhang,et al.  Use Long Short-Term Memory to Enhance Internet of Things for Combined Sewer Overflow Monitoring , 2018 .

[40]  Yao Zhao,et al.  EA-LSTM: Evolutionary Attention-based LSTM for Time Series Prediction , 2018, Knowl. Based Syst..

[41]  Ameet Talwalkar,et al.  Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization , 2016, J. Mach. Learn. Res..

[42]  Chaker El Amrani,et al.  Sequence to Sequence Weather Forecasting with Long Short-Term Memory Recurrent Neural Networks , 2016 .

[43]  Changjun Zhou,et al.  Forecasting stock prices with long-short term memory neural network based on attention mechanism , 2020, PloS one.

[44]  Diana Marculescu,et al.  HyperPower: Power- and memory-constrained hyper-parameter optimization for neural networks , 2017, 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[45]  Hao Zhang,et al.  Attention-Based LSTM Network for Rotatory Machine Remaining Useful Life Prediction , 2020, IEEE Access.

[46]  Muhammad Zahid,et al.  Short Term Traffic State Prediction via Hyperparameter Optimization Based Classifiers , 2020, Sensors.

[47]  Laxman Yetukuri,et al.  Algorithms and tools for the preprocessing of LC–MS metabolomics data , 2011 .

[48]  Ramakrishnan Kannan,et al.  HyperSpace: Distributed Bayesian Hyperparameter Optimization , 2018, 2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD).

[49]  Raphael Semiat,et al.  Energy issues in desalination processes. , 2008, Environmental science & technology.

[50]  Jean Oh,et al.  Attention-based Multimodal Neural Machine Translation , 2016, WMT.

[51]  Diego Gómez,et al.  Intelligent System for the Predictive Analysis of an Industrial Wastewater Treatment Process , 2020 .

[52]  Tzahi Y Cath,et al.  Data-driven performance analyses of wastewater treatment plants: A review. , 2019, Water research.

[53]  Martin J. Sweetman,et al.  Activated Carbon, Carbon Nanotubes and Graphene: Materials and Composites for Advanced Water Purification , 2017 .

[54]  Andrea Cipollina,et al.  تحلية مياه البحر؛ سيرورات الطاقة التقليدية والمتجددة (Seawater Desalination. Conventional and Renewable Energy Processes) , 2009 .

[55]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[56]  Venet Osmani,et al.  Monitoring and detecting faults in wastewater treatment plants using deep learning , 2020, Environmental Monitoring and Assessment.

[57]  Yue Wu,et al.  Double-Branch Multi-Attention Mechanism Network for Hyperspectral Image Classification , 2019, Remote. Sens..

[58]  Garrison W. Cottrell,et al.  A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction , 2017, IJCAI.

[59]  Nicholas Bambos,et al.  Automatic Sleep Arousal Identification From Physiological Waveforms Using Deep Learning , 2018, 2018 Computing in Cardiology Conference (CinC).

[60]  Michael Y. Hu,et al.  Forecasting with artificial neural networks: The state of the art , 1997 .

[61]  Jennifer Whyte,et al.  Analysing Systems Interdependencies Using a Digital Twin , 2019 .

[62]  Ole Johan Jørgensen Lønnum Deep Learning Metocean Simulation and its Applications in Marine Simulation-based Design , 2018 .