Novelty Detection in Learning Systems

Novelty detection is concerned with recognising inputs that differ in some way from those that are usually seen. It is a useful technique in cases where an important class of data is under-represented in the training set. This means that the performance of the network will be poor for those classes. In some circumstances, such as medical data and fault detection, it is often precisely the class that is under-represented in the data, the disease or potential fault, that the network should detect. In novelty detection systems the network is trained only on the negative examples where that class is not present, and then detects inputs that do not fits into the model that it has acquired, that it, members of the novel class. This paper reviews the literature on novelty detection in neural networks and other machine learning techniques, as well as providing brief overviews of the related topics of statistical outlier detection and novelty detection in biological organisms.

[1]  R. Penrose.  A Generalized inverse for matrices , 1955 .

[2]  E. Gumbel,et al.  Statistics of extremes , 1960 .

[3]  K H PRIBRAM,et al.  A further experimental analysis of the behavioral deficit that follows injury to the primate frontal cortex. , 1961, Experimental neurology.

[4]  J. Orbach.  Principles of Neurodynamics. Perceptrons and the Theory of Brain Mechanisms. , 1962 .

[5]  R. F. Thompson,et al.  Habituation: a model phenomenon for the study of neuronal substrates of behavior. , 1966, Psychological review.

[6]  P. Sen,et al.  Theory of rank tests , 1969 .

[7]  John W. Sammon,et al.  A Nonlinear Mapping for Data Structure Analysis , 1969, IEEE Transactions on Computers.

[8]  P. Groves,et al.  Habituation: a dual-process theory. , 1970, Psychological review.

[9]  S. Grossberg.  A neural theory of punishment and avoidance, I: Qualitative theory , 1972 .

[10]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[11]  J. C. Stanley.  Computer simulation of a model of habituation , 1976, Nature.

[12]  T. J. Tighe,et al.  Habituation: Perspectives from child development, animal behavior, and neurophysiology , 1976 .

[13]  Erkki Oja.  S-Orthogonal Projection Operators as Asymptotic Solutions of a Class of Matrix Differential Equations , 1978 .

[14]  E R Kandel,et al.  Cellular analysis of long-term habituation of the gill-withdrawal reflex of Aplysia californica. , 1978, Science.

[15]  L. Denby,et al.  Robust Estimation of the First-Order Autoregressive Parameter , 1979 .

[16]  R. Passingham.  The hippocampus as a cognitive map J. O'Keefe & L. Nadel, Oxford University Press, Oxford (1978). 570 pp., £25.00 , 1979, Neuroscience.

[17]  L. Devroye,et al.  Detection of Abnormal Behavior Via Nonparametric Estimation of the Support , 1980 .

[18]  C. H. Bailey,et al.  Morphological basis of long-term habituation and sensitization in Aplysia. , 1983, Science.

[19]  F. Mosteller,et al.  Understanding robust and exploratory data analysis , 1985 .

[20]  L. Rabiner,et al.  An introduction to hidden Markov models , 1986, IEEE ASSP Magazine.

[21]  James L. McClelland,et al.  Parallel Distributed Processing: Explorations in the Microstructure of Cognition : Psychological and Biological Models , 1986 .

[22]  Peter J. Rousseeuw,et al.  Robust regression and outlier detection , 1987 .

[23]  C. D. Kemp,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[24]  James H. Schwartz,et al.  A molecular mechanism for long-term sensitization in Aplysia , 1987, Nature.

[25]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[26]  Stephen Grossberg,et al.  The ART of Adaptive Pattern Recognition Self-organizing by a Neu Network , 1988 .

[27]  Teuvo Kohonen,et al.  Self-organization and associative memory: 3rd edition , 1989 .

[28]  John Moody,et al.  Fast Learning in Networks of Locally-Tuned Processing Units , 1989, Neural Computation.

[29]  J. Rubner,et al.  A Self-Organizing Network for Principal-Component Analysis , 1989 .

[30]  John H. Byrne,et al.  Mathematical Model of Cellular and Molecular Processes Contributing to Associative and Nonassociative Learning in Aplysia , 1989 .

[31]  Daniel S. Levine,et al.  Modeling some effects of frontal lobe damage--Novelty and perseveration , 1989, Neural Networks.

[32]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[33]  G. Lynch,et al.  The neurobiology of learning and memory , 1989, Cognition.

[34]  P. S. Maybeck,et al.  The Kalman Filter: An Introduction to Concepts , 1990, Autonomous Robot Vehicles.

[35]  DeLiang Wang,et al.  SLONN: A Simulation Language for modeling of Neural Networks , 1990, Simul..

[36]  Hans G. C. Tråvén,et al.  A neural network approach to statistical pattern classification by 'semiparametric' estimation of probability density functions , 1991, IEEE Trans. Neural Networks.

[37]  Dirk Aeyels,et al.  On the dynamic behavior of the novelty detector and the novelty filter , 1991 .

[38]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[39]  Michael A. Arbib,et al.  Hierarchical dishabituation of visual discrimination in toads , 1991 .

[40]  W. J. Daunicht,et al.  Autoassociation and novelty detection by neuromechanics. , 1991, Science.

[41]  A. Jagota,et al.  Novelty detection on a very large number of memories stored in a Hopfield-style network , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[42]  M. Arozullah,et al.  Neural network based novelty filtering for signal detection enhancement , 1992, [1992] Proceedings of the 35th Midwest Symposium on Circuits and Systems.

[43]  L. Cooper,et al.  When Networks Disagree: Ensemble Methods for Hybrid Neural Networks , 1992 .

[44]  Haluk Ogmen,et al.  Some neural correlates of sensorial and cognitive control of behavior , 1992, Defense, Security, and Sensing.

[45]  Dean Pomerleau,et al.  Input Reconstruction Reliability Estimation , 1992, NIPS.

[46]  Padhraic J. Smyth,et al.  Hidden Markov models for fault detection in dynamic systems , 1993 .

[47]  J Metcalfe.  Novelty monitoring, metacognition, and control in a composite holographic associative recall model: implications for Korsakoff amnesia. , 1993, Psychological review.

[48]  John E. R. Staddon.  A note on rate-sensitive habituation , 1993 .

[49]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[50]  Christopher M. Bishop,et al.  Novelty detection and neural network validation , 1994 .

[51]  Stephen J. Roberts,et al.  A Probabilistic Resource Allocating Network for Novelty Detection , 1994, Neural Computation.

[52]  Padhraic Smyth,et al.  Markov monitoring with unknown states , 1994, IEEE J. Sel. Areas Commun..

[53]  Nathalie Japkowicz,et al.  A Novelty Detection Approach to Classification , 1995, IJCAI.

[54]  Vladimir Vapnik,et al.  The Nature of Statistical Learning , 1995 .

[55]  Joydeep Ghosh,et al.  A habituation based neural network for spatio-temporal classification , 1995, Proceedings of 1995 IEEE Workshop on Neural Networks for Signal Processing.

[56]  Michael Brady,et al.  Novelty detection for the identification of masses in mammograms , 1995 .

[57]  H. Elsimary.  Implementation of neural network and genetic algorithms for novelty filters for fault detection , 1996, Proceedings of the 39th Midwest Symposium on Circuits and Systems.

[58]  Dipankar Dasgupta,et al.  Novelty detection in time series data using ideas from immunology , 1996 .

[59]  R. Knight.  Contribution of human hippocampal region to novelty detection , 1996, Nature.

[60]  Stephen J. Roberts,et al.  Novelty, confidence and errors in connectionist systems , 1996 .

[61]  Peter Lozo.  Neural Circuit For Matchhnismatch, Familiarity/novelty And Synchronization Detection In Saart Neural Networks , 1996, Fourth International Symposium on Signal Processing and Its Applications.

[62]  Paul Helman,et al.  An immunological approach to change detection: algorithms, analysis and implications , 1996, Proceedings 1996 IEEE Symposium on Security and Privacy.

[63]  Lucas C. Parra,et al.  Statistical Independence and Novelty Detection with Information Preserving Nonlinear Maps , 1996, Neural Computation.

[64]  M. W. Brown.  Neuronal responses and recognition memory , 1996 .

[65]  Andreas Kurz.  Constructing maps for mobile robot navigation based on ultrasonic range data , 1996, IEEE Trans. Syst. Man Cybern. Part B.

[66]  Mohamed A. El-Sharkawi,et al.  Detection of shorted-turns in the field winding of turbine-generator rotors using novelty detectors-development and field test , 1996 .

[67]  Stephen J. Roberts,et al.  A Validation Index For Artificial Neural Networks , 1996 .

[68]  Keith Worden,et al.  STRUCTURAL FAULT DETECTION USING A NOVELTY MEASURE , 1997 .

[69]  R. R. Macdonald.  On statistical testing in psychology , 1997 .

[70]  Robert P. W. Duin,et al.  Novelty Detection Using Self-Organizing Maps , 1997, ICONIP.

[71]  Georges Linarès,et al.  Model Breaking Detection Using Independent Component Classifier , 1997, ICANN.

[72]  Jean Rouat,et al.  A Novelty Detector Using a Network of Integrate and Fire Neurons , 1997, ICANN.

[73]  Terrence J. Sejnowski,et al.  A Unifying Objective Function for Topographic Mappings , 1997, Neural Computation.

[74]  Lionel Tarassenko,et al.  Choosing an appropriate model for novelty detection , 1997 .

[75]  Jean Rouat,et al.  Novelty detection based on relaxation time of a network of integrate-and-fire neurons , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[76]  Alberto Muñoz,et al.  Self-organizing maps for outlier detection , 1998, Neurocomputing.

[77]  M. W. Brown,et al.  Recognition memory: neuronal substrates of the judgement of prior occurrence , 1998, Progress in Neurobiology.

[78]  J. C. BurgesChristopher.  A Tutorial on Support Vector Machines for Pattern Recognition , 1998 .

[79]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[80]  Robert P. W. Duin,et al.  Outlier Detection Using Classifier Instability , 1998, SSPR/SPR.

[81]  Samuel Kaski,et al.  Bibliography of Self-Organizing Map (SOM) Papers: 1981-1997 , 1998 .

[82]  Gilles Pagès,et al.  Theoretical aspects of the SOM algorithm , 1998, Neurocomputing.

[83]  John MacIntyre,et al.  Adaptive local fusion systems for novelty detection and diagnostics in condition monitoring , 1998, Defense, Security, and Sensing.

[84]  Haluk Ögmen,et al.  Self-organization via active exploration: hardware implementation of a neural robot , 1998, Robotica.

[85]  Amanda Parker,et al.  The von Restorff Effect in Visual Object Recognition Memory in Humans and Monkeys: The Role of Frontal/Perirhinal Interaction , 1998, Journal of Cognitive Neuroscience.

[86]  D. Martinez,et al.  Neural tree density estimation for novelty detection , 1998, IEEE Trans. Neural Networks.

[87]  Stefano Nolfi,et al.  Learning to perceive the world as articulated: an approach for hierarchical learning in sensory-motor systems , 1998, Neural Networks.

[88]  S. Roberts.  Novelty detection using extreme value statistics , 1999 .

[89]  Lionel Tarassenko,et al.  A System for the Analysis of Jet Engine Vibration Data , 1999, Integr. Comput. Aided Eng..

[90]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[91]  Christophe Giraud-Carrier,et al.  High Capacity Neural Networks for Familiarity Discrimination , 1999 .

[92]  Frank Montgomery,et al.  IDENTIFYING MOTORWAY INCIDENTS BY NOVELTY DETECTION , 1999 .

[93]  Jim Austin,et al.  Neural networks for novelty detection in airframe strain data , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[94]  U. Nehmzow,et al.  Novelty Detection on a Mobile Robot Using Habituation , 2000, ArXiv.

[95]  Nello Cristianini,et al.  An introduction to Support Vector Machines , 2000 .

[96]  Colin Campbell,et al.  A Linear Programming Approach to Novelty Detection , 2000, NIPS.

[97]  Florian Metze,et al.  Generalized radial basis function networks for classification and novelty detection: self-organization of optimal Bayesian decision , 2000, Neural Networks.

[98]  José Carlos Príncipe,et al.  On the use of neural networks in the generalized likelihood ratio test for detecting abrupt changes in signals , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[99]  S. Roberts.  EXTREME VALUE STATISTICS FOR NOVELTY DETECTION IN BIOMEDICAL DATA PROCESSING , 2000 .

[100]  Keith Worden,et al.  Detection of defects in composite plates using Lamb waves and novelty detection , 2000, Int. J. Syst. Sci..

[101]  Kimmo Hätönen,et al.  A computer host-based user anomaly detection system using the self-organizing map , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[102]  Lars Niklasson,et al.  Time series segmentation using an adaptive resource allocating vector quantization network based on change detection , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[103]  Hanseok Ko,et al.  Dynamical behavior of autoassociative memory performing novelty filtering for signal enhancement , 2000, IEEE Trans. Neural Networks Learn. Syst..

[104]  Stephen R. Marsland,et al.  Learning to select distinctive landmarks for mobile robot navigation , 2001, Robotics Auton. Syst..

[105]  Stephen Marsland,et al.  On-Line Novelty Detection through self-organisation with application to inspection robotics , 2001 .

[106]  Bernhard Schölkopf,et al.  Estimating the Support of a High-Dimensional Distribution , 2001, Neural Computation.

[107]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[108]  Stephen R. Marsland,et al.  A tale of two filters-on-line novelty detection , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[109]  Stephen R. Marsland,et al.  A self-organising network that grows when required , 2002, Neural Networks.

[110]  Paul A. Crook,et al.  A Robot Implementation of a Biologically Inspired Method for Novelty Detection , 2002 .

[111]  Stephen Grossberg,et al.  A Neural Theory of Punishment and Avoidance , II : Quantitative Theory , 2003 .

[112]  J. Ewert,et al.  Configurational pattern discrimination responsible for dishabituation in common toads Bufo bufo (L.): Behavioral tests of the predictions of a neural model , 1992, Journal of Comparative Physiology A.

[113]  M. Arbib,et al.  A model of the neural mechanisms responsible for pattern recognition and stimulus specific habituation in toads , 2004, Biological Cybernetics.

[114]  Michael A. Arbib,et al.  Modeling the dishabituation hierarchy: The role of the primordial hippocampus , 1992, Biological Cybernetics.

[115]  Teuvo Kohonen,et al.  Self-organized formation of topologically correct feature maps , 2004, Biological Cybernetics.

[116]  D. L. Reilly,et al.  A neural model for category learning , 1982, Biological Cybernetics.

[117]  J. -P. Ewert,et al.  Configurational prey-selection by individual experience in the toadBufo bufo , 2004, Journal of comparative physiology.

[118]  E. Oja,et al.  Fast adaptive formation of orthogonalizing filters and associative memory in recurrent networks of neuron-like elements , 1976, Biological Cybernetics.

[119]  M. A. Arbib,et al.  How does the toad's visual system discriminate different worm-like stimuli? , 2004, Biological Cybernetics.

[120]  Rafal Bogacz,et al.  Model of Familiarity Discrimination in the Perirhinal Cortex , 2004, Journal of Computational Neuroscience.

[121]  P. Rousseeuw,et al.  Wiley Series in Probability and Mathematical Statistics , 2005 .

[122]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.