Fast and unsupervised outlier removal by recurrent adaptive reconstruction extreme learning machine

Outlier removal is vital in machine learning. As massive unlabeled data are generated rapidly today, eliminating outliers from noisy data in a fast and unsupervised manner is gaining increasing attention in practical applications. This paper tackles this challenging problem by proposing a novel Recurrent Adaptive Reconstruction Extreme Learning Machine (RAR-ELM). Specifically, with the given noisy data collection, RAR-ELM recurrently learns to reconstruct data and automatically excludes those data with high reconstruction errors as outliers by a novel adaptive labeling mechanism. Compared with existing methods, the proposed RAR-ELM enjoys three major merits: first, RAR-ELM inherits the fast and sound learning property of original extreme learning machine (ELM). RAR-ELM can be implemented at a tens or hundreds of times faster speed while achieving a superior or comparable outlier removal performance to existing methods, which makes RAR-ELM particularly suitable for application scenarios like real-time outlier removal; secondly, instead of priorly specifying a decision threshold, RAR-ELM is able to adaptively find a reasonable decision threshold when processing data with different proportions of outliers, which is vital to the case of unsupervised outlier removal where no prior knowledge of outliers in the data is available; thirdly, we also propose Online Sequential RAR-ELM (OS-RAR-ELM) can be implemented by an online or sequential mode, which makes RAR-ELM easily applicable to massive noisy data or online sequential data. Extensive experiments on various datasets reveal that the proposed RAR-ELM can realize faster and better unsupervised outlier removal in contrast to existing methods.

[1]  Hans-Peter Kriegel,et al.  Angle-based outlier detection in high-dimensional data , 2008, KDD.

[2]  Karl Pearson F.R.S..  LIII. On lines and planes of closest fit to systems of points in space , 1901 .

[3]  Yiqiang Chen,et al.  Weighted extreme learning machine for imbalance learning , 2013, Neurocomputing.

[4]  Pasi Fränti,et al.  Outlier detection using k-nearest neighbour graph , 2004, ICPR 2004.

[5]  Yong Dou,et al.  An efficient and effective convolutional auto-encoder extreme learning machine network for 3d feature learning , 2016, Neurocomputing.

[6]  Robert P. W. Duin,et al.  Support Vector Data Description , 2004, Machine Learning.

[7]  Yihong Gong,et al.  Locality-constrained Linear Coding for image classification , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[8]  Judith Redi,et al.  Circular-ELM for the reduced-reference assessment of perceived image quality , 2013, Neurocomputing.

[9]  Victor C. M. Leung,et al.  Extreme Learning Machines [Trends & Controversies] , 2013, IEEE Intelligent Systems.

[10]  Gang Hua,et al.  Learning Discriminative Reconstructions for Unsupervised Outlier Removal , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[11]  Guang-Bin Huang,et al.  Extreme Learning Machine for Multilayer Perceptron , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[12]  Aleksandar Lazarevic,et al.  Incremental Local Outlier Detection for Data Streams , 2007, 2007 IEEE Symposium on Computational Intelligence and Data Mining.

[13]  F. E. Grubbs.  Procedures for Detecting Outlying Observations in Samples , 1969 .

[14]  Ligang Liu,et al.  Projective Feature Learning for 3D Shapes with Multi‐View Depth Images , 2015, Comput. Graph. Forum.

[15]  Hans-Peter Kriegel,et al.  A survey on unsupervised outlier detection in high‐dimensional numerical data , 2012, Stat. Anal. Data Min..

[16]  Hongming Zhou,et al.  Extreme Learning Machines [Trends & Controversies] , 2013 .

[17]  Nuno Vasconcelos,et al.  Anomaly detection in crowded scenes , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[18]  Guillermo Sapiro,et al.  See all by looking at a few: Sparse modeling for finding representative objects , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Ping-Lang Yen,et al.  Engineering Applications of Intelligent Monitoring and Control 2014 , 2013 .

[20]  Liyanaarachchi Lekamalage Chamara Kasun,et al.  Generic Object Recognition with Local Receptive Fields Based Extreme Learning Machine , 2015, INNS Conference on Big Data.

[21]  Stephen J. Roberts,et al.  A Probabilistic Resource Allocating Network for Novelty Detection , 1994, Neural Computation.

[22]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[23]  Guofei Gu,et al.  Using an Ensemble of One-Class SVM Classifiers to Harden Payload-based Anomaly Detection Systems , 2006, Sixth International Conference on Data Mining (ICDM'06).

[24]  Hongming Zhou,et al.  Extreme Learning Machine for Regression and Multiclass Classification , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[25]  Sanjoy Dasgupta,et al.  An elementary proof of a theorem of Johnson and Lindenstrauss , 2003, Random Struct. Algorithms.

[26]  Ming Shao,et al.  Locality linear fitting one-class SVM with low-rank constraints for outlier detection , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[27]  Heikki Mannila,et al.  Random projection in dimensionality reduction: applications to image and text data , 2001, KDD '01.

[28]  E. Parzen.  On Estimation of a Probability Density Function and Mode , 1962 .

[29]  Hongxing He,et al.  A comparative study of RNN for outlier detection in data mining , 2002, 2002 IEEE International Conference on Data Mining, 2002. Proceedings..

[30]  N. Otsu.  A threshold selection method from gray level histograms , 1979 .

[31]  Petia Radeva,et al.  Approximate polytope ensemble for one-class classification , 2014, Pattern Recognit..

[32]  Bernhard Schölkopf,et al.  Estimating the Support of a High-Dimensional Distribution , 2001, Neural Computation.

[33]  Jun Miao,et al.  One-Class Classification with Extreme Learning Machine , 2015 .

[34]  Ming Shao,et al.  Low-Rank Outlier Detection , 2014, Low-Rank and Sparse Modeling for Visual Analysis.

[35]  Cheng Wu,et al.  Semi-Supervised and Unsupervised Extreme Learning Machines , 2014, IEEE Transactions on Cybernetics.

[36]  John MacIntyre,et al.  Adaptive local fusion systems for novelty detection and diagnostics in condition monitoring , 1998, Defense, Security, and Sensing.

[37]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[38]  Narasimhan Sundararajan,et al.  A Fast and Accurate Online Sequential Learning Algorithm for Feedforward Networks , 2006, IEEE Transactions on Neural Networks.

[39]  Huangang Wang,et al.  Robust one-class SVM for fault detection , 2016 .

[40]  Xuelong Li,et al.  Relevance Preserving Projection and Ranking for Web Image Search Reranking , 2015, IEEE Transactions on Image Processing.

[41]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[42]  Kwontaeg Choi,et al.  Incremental face recognition for large-scale social network services , 2012, Pattern Recognit..

[43]  Vijay Manikandan Janakiraman,et al.  Anomaly detection in aviation data using extreme learning machines , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[44]  Jian Tang,et al.  Enhancing Effectiveness of Outlier Detections for Low Density Patterns , 2002, PAKDD.

[45]  Lei Zhang,et al.  Abnormal Odor Detection in Electronic Nose via Self-Expression Inspired Extreme Learning Machine , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[46]  Gang Hua,et al.  Unsupervised One-Class Learning for Automatic Outlier Removal , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[47]  Malik Yousef,et al.  One-class document classification via Neural Networks , 2007, Neurocomputing.

[48]  Lin Zhang,et al.  Two methods of selecting Gaussian kernel parameters for one-class SVM and their application to fault detection , 2014, Knowl. Based Syst..

[49]  Hans-Peter Kriegel,et al.  LOF: identifying density-based local outliers , 2000, SIGMOD 2000.

[50]  Charu C. Aggarwal,et al.  Outlier Analysis , 2013, Springer New York.

[51]  En Zhu,et al.  Video anomaly detection and localization by local motion based joint video representation and OCELM , 2018, Neurocomputing.

[52]  Antonio Criminisi,et al.  Harvesting Image Databases from the Web , 2007, ICCV.