Flow of Information in Feed-Forward Denoising Neural Networks

Due to inaccuracies in data acquisition, time series data often suffer from noise and instability which leads to inaccurate data mining results. The ability to handle noisy time series data is thus critical in many data-driven real-time applications. Using the locality feature of time series, feed-forward deep neural networks has been effectively used for time series denoising. In this paper, in order to understand the underling behavior of denoising neural networks, we use an information theoretic approach to study the flow of information and to determine how the entropy of information changes between consecutive layers. We develop analytical bounds for multi-layer feed-forward deep neural networks deployed in time series denoising. Numerical experiments support our theoretical conclusions.

[1]  Antonio Salmerón,et al.  Multivariate Imputation of Qualitative Missing Data Using Bayesian Networks , 2004 .

[2]  Xiaogang Wang,et al.  Deep Learning Face Representation from Predicting 10,000 Classes , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Ana Fred.  Structural, syntactic, and statistical pattern recognition : joint IAPR international workshops, SSPR 2004 and SPR 2004, Lisbon, Portugal, August 18-20, 2004 : proceedings , 2004 .

[4]  Naftali Tishby,et al.  The information bottleneck method , 2000, ArXiv.

[5]  Naren Ramakrishnan,et al.  An integrated data mining framework for analysis and prediction of battery characteristics , 2014, 2014 IEEE Innovative Smart Grid Technologies - Asia (ISGT ASIA).

[6]  Saifur Rahman,et al.  Analyzing Invariants in Cyber-Physical Systems using Latent Factor Regression , 2015, KDD.

[7]  David J. Schwab,et al.  An exact mapping between the Variational Renormalization Group and Deep Learning , 2014, ArXiv.

[8]  Naren Ramakrishnan,et al.  Time series forecasting via noisy channel reversal , 2015, 2015 IEEE 25th International Workshop on Machine Learning for Signal Processing (MLSP).

[9]  Kyoung-jae Kim,et al.  Financial time series forecasting using support vector machines , 2003, Neurocomputing.

[10]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[11]  Enhong Chen,et al.  Image Denoising and Inpainting with Deep Neural Networks , 2012, NIPS.

[12]  Naren Ramakrishnan,et al.  Online denoising of discrete noisy data , 2015, 2015 IEEE International Symposium on Information Theory (ISIT).

[13]  Naren Ramakrishnan,et al.  Wikipedia in the Tourism Industry: Forecasting Demand and Modeling Usage Behavior , 2016, AAAI.

[14]  Christopher M. Bishop.  Theoretical Foundations of Neural Networks , 1996 .

[15]  Naftali Tishby,et al.  Deep learning and the information bottleneck principle , 2015, 2015 IEEE Information Theory Workshop (ITW).

[16]  Honglak Lee,et al.  Deep learning for detecting robotic grasps , 2013, Int. J. Robotics Res..

[17]  Pejman Khadivi,et al.  Online Denoising Solutions for Forecasting Applications , 2016 .

[18]  Yossi Aviv,et al.  A Time-Series Framework for Supply-Chain Inventory Management , 2003, Oper. Res..

[19]  HuiMing Li.  Deep Learning for Image Denoising , 2014 .

[20]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[21]  Madhav V. Marathe,et al.  Forecasting a Moving Target: Ensemble Models for ILI Case Count Predictions , 2014, SDM.