Estimation of Probability Density Function, Differential Entropy and Other Relative Quantities for Data Streams with Concept Drift

In this paper estimators of nonstationary probability density function are proposed. Additionally, applying the trapezoidal method of numerical integration, the estimators of two information-theoretic measures are presented: the differential entropy and the Renyi’s quadratic differential entropy. Finally, using an analogous methodology, estimators of the Cauchy-Schwarz divergence and the probability density function divergence are proposed, which are used to measure the differences between two probability density functions. All estimators are proposed in two variants: one with the sliding window and one with the forgetting factor. Performance of all the estimators is verified using numerical simulations.

[1]  Piotr Duda,et al.  Heuristic Regression Function Estimation Methods for Data Streams with Concept Drift , 2017, ICAISC.

[2]  Marcin Zalasinski,et al.  New Algorithm for On-line Signature Verification Using Characteristic Hybrid Partitions , 2015, ISAT.

[3]  Piotr Duda,et al.  New Splitting Criteria for Decision Trees in Stationary Data Streams , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Gregory Ditzler,et al.  Learning in Nonstationary Environments: A Survey , 2015, IEEE Computational Intelligence Magazine.

[5]  V. A. Epanechnikov.  Non-Parametric Estimation of a Multivariate Probability Density , 1969 .

[6]  Jaroslaw Bilski,et al.  Parallel Architectures for Learning the RTRN and Elman Dynamic Neural Networks , 2015, IEEE Transactions on Parallel and Distributed Systems.

[7]  A. Krzyżak,et al.  The pointwise rate of convergence of the kernel regression estimate , 1987 .

[8]  Donald F. Specht,et al.  Probabilistic neural networks , 1990, Neural Networks.

[9]  Michael Botsch,et al.  A Machine Learning Approach for the Segmentation of Driving Maneuvers and its Application in Autonomous Parking , 2017, J. Artif. Intell. Soft Comput. Res..

[10]  Peng Yan,et al.  MapReduce and Semantics Enabled Event Detection using Social Media , 2017, J. Artif. Intell. Soft Comput. Res..

[11]  W. Greblicki,et al.  Nonparametric system identification , 2008 .

[12]  Piotr Duda,et al.  On applying the Restricted Boltzmann Machine to active concept drift detection , 2017, 2017 IEEE Symposium Series on Computational Intelligence (SSCI).

[13]  Janusz T. Starczewski,et al.  Toward Work Groups Classification Based on Probabilistic Neural Network Approach , 2015, ICAISC.

[14]  L. Rutkowski.  Sequential Estimates of a Regression Function by Orthogonal Series with Applications in Discrimination , 1981 .

[15]  Yuji Sato,et al.  Swarm Intelligence Algorithm Based on Competitive Predators with Dynamic Virtual Teams , 2017, J. Artif. Intell. Soft Comput. Res..

[16]  Oscar Chang,et al.  A Novel Deep Neural Network that Uses Space-Time Features for Tracking and Recognizing a Moving Object , 2017, J. Artif. Intell. Soft Comput. Res..

[17]  Leszek Rutkowski,et al.  Generalized regression neural networks in time-varying environment , 2004, IEEE Transactions on Neural Networks.

[18]  Vincent Lemaire,et al.  A Survey on Supervised Classification on Data Streams , 2014, eBISS.

[19]  Piotr Duda,et al.  A New Method for Data Stream Mining Based on the Misclassification Error , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[20]  L. Rutkowski,et al.  Nonparametric fitting of multivariate functions , 1986 .

[21]  Piotr Duda,et al.  On ensemble components selection in data streams scenario with reoccurring concept-drift , 2017, 2017 IEEE Symposium Series on Computational Intelligence (SSCI).

[22]  E. Parzen.  On Estimation of a Probability Density Function and Mode , 1962 .

[23]  Piotr Duda,et al.  The Parzen kernel approach to learning in non-stationary environment , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[24]  V. Susheela Devi,et al.  Parallel MCNN (pMCNN) with Application to Prototype Selection on Large and Streaming Data , 2017, J. Artif. Intell. Soft Comput. Res..

[25]  João Gama,et al.  A survey on concept drift adaptation , 2014, ACM Comput. Surv..