ADMM Consensus for Deep LSTM Networks

In modern real-world applications, the need of using a decentralized data processing approach has progressively increased, facing complexity and handling issues. Pervasive data and ubiquitous computational capacity have enabled the proficient use of distributed implementation of machine learning algorithms, especially for forecasting problems. We provide in this paper a new, fully distributed prediction approach based on the Long Short-Term Memory deep neural network. When placed in a network of interconnected agents, the single predictors are able to improve the prediction accuracy by means of the Alternating Direction Method of Multipliers consensus procedure on some network parameters. Experimental tests on real-world time series prove the efficacy of the proposed approach, which regulates the information exchange in the network through high-level structures in the considered models.

[1]  Jaime Lloret,et al.  Artificial neural networks for short-term load forecasting in microgrids environment , 2014 .

[2]  H. Vincent Poor,et al.  Distributed learning in wireless sensor networks , 2005, IEEE Signal Processing Magazine.

[3]  Hermann Ney,et al.  LSTM Neural Networks for Language Modeling , 2012, INTERSPEECH.

[4]  Tamer Khatib,et al.  A review of islanding detection techniques for renewable distributed generation systems , 2013 .

[5]  Simone Scardapane,et al.  Distributed semi-supervised support vector machines , 2016, Neural Networks.

[6]  Alex Graves,et al.  Generating Sequences With Recurrent Neural Networks , 2013, ArXiv.

[7]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[8]  Stephen P. Boyd,et al.  Distributed average consensus with least-mean-square deviation , 2007, J. Parallel Distributed Comput..

[9]  Massimo Panella,et al.  Prediction in Photovoltaic Power by Neural Networks , 2017 .

[10]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[11]  Dianhui Wang,et al.  Distributed learning for Random Vector Functional-Link networks , 2015, Inf. Sci..

[12]  Mehmet Emre Çek,et al.  Analysis of observed chaotic data , 2004 .

[13]  Zoran Obradovic,et al.  The distributed boosting algorithm , 2001, KDD '01.

[14]  P. Lerman.  Fitting Segmented Regression Models by Grid Search , 1980 .

[15]  Henrik Madsen,et al.  Multi-site solar power forecasting using gradient boosted regression trees , 2017 .

[16]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[17]  Massimo Panella,et al.  A Distributed Algorithm for the Cooperative Prediction of Power Production in PV Plants , 2019, IEEE Transactions on Energy Conversion.

[18]  R. Urraca,et al.  Review of photovoltaic power forecasting , 2016 .

[19]  Sonia Leva,et al.  Physical and hybrid methods comparison for the day ahead PV output power forecast , 2017 .

[20]  Ali H. Sayed,et al.  Adaptive Networks , 2014, Proceedings of the IEEE.

[21]  Amit Kumar Yadav,et al.  Solar radiation prediction using Artificial Neural Network techniques: A review , 2014 .

[22]  Soteris A. Kalogirou,et al.  Machine learning methods for solar radiation forecasting: A review , 2017 .

[23]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[24]  Jinyu Wen,et al.  Determining the Minimal Power Capacity of Energy Storage to Accommodate Renewable Generation , 2017 .

[25]  Nikos D. Hatziargyriou,et al.  Integrating distributed generation into electric power systems: A review of drivers, challenges and opportunities , 2007 .

[26]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[27]  Jürgen Schmidhuber,et al.  Learning to Forget: Continual Prediction with LSTM , 2000, Neural Computation.

[28]  Bernhard Sick,et al.  Deep Learning for solar power forecasting — An approach using AutoEncoder and LSTM Neural Networks , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[29]  Dianhui Wang,et al.  Distributed music classification using Random Vector Functional-Link nets , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[30]  Samy Bengio,et al.  Show and tell: A neural image caption generator , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Angelika Bayer,et al.  Solar Engineering Of Thermal Processes , 2016 .

[32]  Alex Graves,et al.  DRAW: A Recurrent Neural Network For Image Generation , 2015, ICML.

[33]  Gonzalo Mateos,et al.  Modeling and Optimization for Big Data Analytics: (Statistical) learning tools for our era of data deluge , 2014, IEEE Signal Processing Magazine.

[34]  Dong Liu,et al.  Research on Stochastic Optimal Operation Strategy of Active Distribution Network Considering Intermittent Energy , 2017 .