A Neural Network-Based On-Device Learning Anomaly Detector for Edge Devices

Semi-supervised anomaly detection is an approach to identify anomalies by learning the distribution of normal data. Backpropagation neural networks (i.e., BP-NNs) based approaches have recently drawn attention because of their good generalization capability. In a typical situation, BP-NN-based models are iteratively optimized in server machines with input data gathered from the edge devices. However, (1) the iterative optimization often requires significant efforts to follow changes in the distribution of normal data (i.e., concept drift), and (2) data transfers between edge and server impose additional latency and energy consumption. To address these issues, we propose ONLAD and its IP core, named ONLAD Core. ONLAD is highly optimized to perform fast sequential learning to follow concept drift in less than one millisecond. ONLAD Core realizes on-device learning for edge devices at low power consumption, which realizes standalone execution where data transfers between edge and server are not required. Experiments show that ONLAD has favorable anomaly detection capability in an environment that simulates concept drift. Evaluations of ONLAD Core confirm that the training latency is 1.95x<inline-formula><tex-math notation="LaTeX">$\sim$</tex-math><alternatives><mml:math><mml:mo>∼</mml:mo></mml:math><inline-graphic xlink:href="tsukada-ieq1-2973631.gif"/></alternatives></inline-formula>6.58x faster than the other software implementations. Also, the runtime power consumption of ONLAD Core implemented on PYNQ-Z1 board, a small FPGA/CPU SoC platform, is 5.0x<inline-formula><tex-math notation="LaTeX">$\sim$</tex-math><alternatives><mml:math><mml:mo>∼</mml:mo></mml:math><inline-graphic xlink:href="tsukada-ieq2-2973631.gif"/></alternatives></inline-formula>25.4x lower than them.

[1]  Carla Purdy,et al.  Fast hardware assisted online learning using unsupervised deep learning structure for anomaly detection , 2018, 2018 International Conference on Information and Computer Technologies (ICICT).

[2]  Dong Sun Park,et al.  Online sequential extreme learning machine with forgetting mechanism , 2012, Neurocomputing.

[3]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[4]  Wei Xu,et al.  Improving one-class SVM for anomaly detection , 2003, Proceedings of the 2003 International Conference on Machine Learning and Cybernetics (IEEE Cat. No.03EX693).

[5]  Harish Kumar,et al.  An intrusion detection system using network traffic profiling and online sequential extreme learning machine , 2015, Expert Syst. Appl..

[6]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[7]  Taeho Jo,et al.  A Multiple Resampling Method for Learning from Imbalanced Data Sets , 2004, Comput. Intell..

[8]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[9]  Narasimhan Sundararajan,et al.  A Fast and Accurate Online Sequential Learning Algorithm for Feedforward Networks , 2006, IEEE Transactions on Neural Networks.

[10]  Antonio Liotta,et al.  Spatial anomaly detection in sensor networks using neighborhood information , 2017, Inf. Fusion.

[11]  Stanley Kok,et al.  Multi-layer Online Sequential Extreme Learning Machine for Image Classification , 2016 .

[12]  Christos-Savvas Bouganis,et al.  fpgaConvNet: A Framework for Mapping Convolutional Neural Networks on FPGAs , 2016, 2016 IEEE 24th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM).

[13]  Asit K. Mishra,et al.  From high-level deep neural models to FPGAs , 2016, 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO).

[14]  Christopher Leckie,et al.  High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning , 2016, Pattern Recognit..

[15]  Axel Jantsch,et al.  Neural network based ECG anomaly detection on FPGA and trade-off analysis , 2017, 2017 IEEE International Symposium on Circuits and Systems (ISCAS).

[16]  Koichiro Mashiko,et al.  FPGA implementation of extreme learning machine system for classification , 2017, TENCON 2017 - 2017 IEEE Region 10 Conference.

[17]  Anil K. Jain,et al.  Encyclopedia of Biometrics , 2015, Springer US.

[18]  Peter Richtárik,et al.  Federated Optimization: Distributed Machine Learning for On-Device Intelligence , 2016, ArXiv.

[19]  Satinder Singh,et al.  Unsupervised Anomaly Detection in Network Intrusion Detection Using Clusters , 2005, ACSC.

[20]  Hongming Zhou,et al.  Silicon spiking neurons for hardware implementation of extreme learning machines , 2013, Neurocomputing.

[21]  Paolo Gastaldo,et al.  Efficient Digital Implementation of Extreme Learning Machines for Classification , 2012, IEEE Transactions on Circuits and Systems II: Express Briefs.

[22]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[23]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[24]  Gene H. Golub,et al.  Matrix computations , 1983 .

[25]  Sujith Ravi,et al.  ProjectionNet: Learning Efficient On-Device Deep Networks Using Neural Projections , 2017, ArXiv.

[26]  Kevin Skadron,et al.  Scalable parallel programming , 2008, 2008 IEEE Hot Chips 20 Symposium (HCS).

[27]  Rui Araújo,et al.  An adaptive ensemble of on-line Extreme Learning Machines with variable forgetting factor for dynamic system prediction , 2016, Neurocomputing.

[28]  Marek Wegrzyn,et al.  Hardware implementation of real-time Extreme Learning Machine in FPGA: Analysis of precision, resource occupation and performance , 2016, Comput. Electr. Eng..

[29]  He Jiang,et al.  FP-ELM: An online sequential learning algorithm for dealing with concept drift , 2016, Neurocomputing.

[30]  Geoffrey I. Webb,et al.  # 2001 Kluwer Academic Publishers. Printed in the Netherlands. Machine Learning for User Modeling , 1999 .

[31]  Mehmet Karakose,et al.  FPGA based intelligent condition monitoring of induction motors: Detection, diagnosis, and prognosis , 2011, 2011 IEEE International Conference on Industrial Technology.

[32]  Georg Langs,et al.  Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery , 2017, IPMI.

[33]  Kush R. Varshney,et al.  Increasing Trust in AI Services through Supplier's Declarations of Conformity , 2018, IBM J. Res. Dev..

[34]  Georg Carle,et al.  Traffic Anomaly Detection Using K-Means Clustering , 2007 .

[35]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[36]  Hiroki Matsutani,et al.  OS-ELM-FPGA: An FPGA-Based Online Sequential Unsupervised Anomaly Detector , 2018, Euro-Par Workshops.

[37]  Bernhard Schölkopf,et al.  Kernel Principal Component Analysis , 1997, ICANN.

[38]  Philip Heng Wai Leong,et al.  Real-time FPGA-based Anomaly Detection for Radio Frequency Signals , 2018, 2018 IEEE International Symposium on Circuits and Systems (ISCAS).

[39]  Thomas S. Huang,et al.  One-class SVM for learning in image retrieval , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[40]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[41]  Gene H. Golub,et al.  Singular value decomposition and least squares solutions , 1970, Milestones in Matrix Computation.

[42]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[43]  Michael McCloskey,et al.  Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem , 1989 .

[44]  William A. Gale,et al.  A sequential algorithm for training text classifiers , 1994, SIGIR '94.

[45]  Hiroki Matsutani,et al.  An Adaptive Abnormal Behavior Detection using Online Sequential Learning , 2019, 2019 IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC).

[46]  Rajesh Gupta,et al.  Accelerating Binarized Convolutional Neural Networks with Software-Programmable FPGAs , 2017, FPGA.

[47]  V. Rao Vemuri,et al.  Use of K-Nearest Neighbor classifier for intrusion detection , 2002, Comput. Secur..

[48]  Takehisa Yairi,et al.  Anomaly Detection Using Autoencoders with Nonlinear Dimensionality Reduction , 2014, MLSDA'14.

[49]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[50]  Hiroki Nakahara,et al.  GUINNESS: A GUI Based Binarized Deep Neural Network Framework for Software Programmers , 2019, IEICE Trans. Inf. Syst..

[51]  Andrew S. Miner,et al.  Anomaly intrusion detection using one class SVM , 2004, Proceedings from the Fifth Annual IEEE SMC Information Assurance Workshop, 2004..

[52]  Kaibin Huang,et al.  Towards an Intelligent Edge: Wireless Communication Meets Machine Learning , 2018, ArXiv.

[53]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[54]  Mehdi Bennis,et al.  Wireless Network Intelligence at the Edge , 2018, Proceedings of the IEEE.

[55]  Sungzoon Cho,et al.  Variational Autoencoder based Anomaly Detection using Reconstruction Probability , 2015 .

[56]  Juan Barrios-Aviles,et al.  Moving Learning Machine towards Fast Real-Time Applications: A High-Speed FPGA-Based Implementation of the OS-ELM Training Algorithm , 2018, Electronics.

[57]  Yimin Yang,et al.  System-on-a-Chip (SoC)-Based Hardware Acceleration for an Online Sequential Extreme Learning Machine (OS-ELM) , 2019, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[58]  Hans-Peter Kriegel,et al.  LOF: identifying density-based local outliers , 2000, SIGMOD '00.

[59]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.

[60]  Nitesh V. Chawla,et al.  Editorial: special issue on learning from imbalanced data sets , 2004, SKDD.