An Empirical Study on Unsupervised Pre-training Approaches in Regression Problems

Unsupervised pre-training allows for efficient training of deep architectures. It provides a good set of initialised weights to the deep architecture that can provide better generalisation of the data. In this paper, we aim to empirically analyse the effect of different unsupervised pre-training approaches for the task of regression on different datasets. We have considered two most common pre-training methods namely deep belief network and stacked autoencoder, and compared the results with the standard training algorithm without pretraining. The models with pretraining performed better than the model without pretraining in terms of error, convergence and the prediction of pattern. The results of the experiments also show the importance of hyperparameters tuning, specially learning rate, in providing a better prediction result. This study once again confirmed the effectiveness and potential of pretraining approach in nonlinear regression problem.

[1]  E. Massera,et al.  On field calibration of an electronic nose for benzene estimation in an urban pollution monitoring scenario , 2008 .

[2]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[3]  Yoshua Bengio,et al.  Scaling learning algorithms towards AI , 2007 .

[4]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[5]  Luis M. Candanedo,et al.  Data driven prediction models of energy use of appliances in a low-energy house , 2017 .

[6]  Yulei Rao,et al.  A deep learning framework for financial time series using stacked autoencoders and long-short term memory , 2017, PloS one.

[7]  Thomas Hofmann,et al.  Greedy Layer-Wise Training of Deep Networks , 2007 .

[8]  I-Cheng Yeh,et al.  Modeling of strength of high-performance concrete using artificial neural networks , 1998 .

[9]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[10]  Shingo Mabu,et al.  Time Series Prediction Using DBN and ARIMA , 2015, 2015 International Conference on Computer Application Technologies.

[11]  Yurong Liu,et al.  A survey of deep neural network architectures and their applications , 2017, Neurocomputing.

[12]  L. Glass,et al.  Oscillation and chaos in physiological control systems. , 1977, Science.

[13]  Rashmi Dutta Baruah,et al.  Unsupervised Pre-training on Improving the Performance of Neural Network in Regression , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[14]  Fred Collopy,et al.  How effective are neural networks at forecasting and prediction? A review and evaluation , 1998 .

[15]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[16]  Danko Brezak,et al.  A comparison of feed-forward and recurrent neural networks in time series forecasting , 2012, 2012 IEEE Conference on Computational Intelligence for Financial Engineering & Economics (CIFEr).