On Training Deep Neural Networks Using a Streaming Approach

Abstract In recent years, many deep learning methods, allowed for a significant improvement of systems based on artificial intelligence methods. Their effectiveness results from an ability to analyze large labeled datasets. The price for such high accuracy is the long training time, necessary to process such large amounts of data. On the other hand, along with the increase in the number of collected data, the field of data stream analysis was developed. It enables to process data immediately, with no need to store them. In this work, we decided to take advantage of the benefits of data streaming in order to accelerate the training of deep neural networks. The work includes an analysis of two approaches to network learning, presented on the background of traditional stochastic and batch-based methods.

[1]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Piotr Duda,et al.  On applying the Restricted Boltzmann Machine to active concept drift detection , 2017, 2017 IEEE Symposium Series on Computational Intelligence (SSCI).

[3]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[4]  Albert Bifet,et al.  Deep learning in partially-labeled data streams , 2015, SAC.

[5]  David Reitter,et al.  Online Semi-Supervised Learning with Deep Hybrid Boltzmann Machines and Denoising Autoencoders , 2015, ArXiv.

[6]  Andrzej Cader,et al.  Resource-Aware Data Stream Mining Using the Restricted Boltzmann Machine , 2019, ICAISC.

[7]  Roberto Souto Maior de Barros,et al.  Wilcoxon Rank Sum Test Drift Detector , 2018, Neurocomputing.

[8]  Rodrigo Fernandes de Mello,et al.  On learning guarantees to unsupervised concept drift detection on data streams , 2019, Expert Syst. Appl..

[9]  Guido Bologna,et al.  Characterization of Symbolic Rules Embedded in Deep DIMLP Networks: A Challenge to Transparency of Deep Learning , 2017, J. Artif. Intell. Soft Comput. Res..

[10]  Gregory Ditzler,et al.  Learning in Nonstationary Environments: A Survey , 2015, IEEE Computational Intelligence Magazine.

[11]  Jean Paul Barddal,et al.  A Survey on Ensemble Learning for Data Stream Classification , 2017, ACM Comput. Surv..

[12]  Yoshua Bengio,et al.  Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies , 2001 .

[13]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Matthew D. Zeiler.  ADADELTA: An Adaptive Learning Rate Method , 2012, ArXiv.

[15]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[16]  Alex Zhavoronkov,et al.  Applications of Deep Learning in Biomedicine. , 2016, Molecular pharmaceutics.

[17]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[18]  Talel Abdessalem,et al.  Adaptive random forests for evolving data stream classification , 2017, Machine Learning.

[19]  Yoshua Bengio,et al.  Greedy Layer-Wise Training of Deep Networks , 2006, NIPS.

[20]  Jürgen Schmidhuber,et al.  Recurrent nets that time and count , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[21]  Zoran Bosni,et al.  Detecting concept drift in data streams using model explanation , 2018 .

[22]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[23]  Yevgeniy Bodyanskiy,et al.  Fast learning algorithm for deep evolving GMDH-SVM neural network in data stream mining tasks , 2016, 2016 IEEE First International Conference on Data Stream Mining & Processing (DSMP).

[24]  Lawrence B. Holder,et al.  On Graph Mining With Deep Learning: Introducing Model R for Link Weight Prediction , 2018, J. Artif. Intell. Soft Comput. Res..

[25]  Piotr Duda,et al.  Concept Drift Detection in Streams of Labelled Data Using the Restricted Boltzmann Machine , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[26]  Geoff Hulten,et al.  Mining high-speed data streams , 2000, KDD '00.

[27]  David B. Skillicorn,et al.  Streaming Random Forests , 2007, 11th International Database Engineering and Applications Symposium (IDEAS 2007).

[28]  Ricard Gavaldà,et al.  Adaptive Learning from Evolving Data Streams , 2009, IDA.

[29]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[30]  Diederik P. Kingma,et al.  Stochastic Gradient VB and the Variational Auto-Encoder , 2013 .

[31]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[32]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[33]  A. Bifet,et al.  Early Drift Detection Method , 2005 .

[34]  David B. Skillicorn,et al.  Classifying Evolving Data Streams Using Dynamic Streaming Random Forests , 2008, DEXA.

[35]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[37]  João Gama,et al.  Learning with Drift Detection , 2004, SBIA.

[38]  Zoran Bosnic,et al.  Detecting concept drift in data streams using model explanation , 2018, Expert Syst. Appl..

[39]  Brian Kingsbury,et al.  New types of deep neural network learning for speech recognition and related applications: an overview , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[40]  João Gama,et al.  Ensemble learning for data stream analysis: A survey , 2017, Inf. Fusion.