A Decision Analysis Method for Evaluating Computer Intrusion Detection Systems

This paper presents a decision analysis method for evaluating computer intrusion detection systems. The method integrates and extends receiver operating characteristic (ROC) and cost analysis methods to provide an expected cost metric. We demonstrate that both the ROC analysis and cost analysis methods are incomplete. Furthermore, we demonstrate how a decision tree can combine and extend the ROC and cost analysis methods to provide an expected cost metric that reflects the intrusion detection system's ROC curve, costs, and assessments of the hostility of the environment as summarized by the prior probability of intrusion. We further demonstrate how this method can be used to decide the optimal operating point on an intrusion detector's ROC curve, choose the best intrusion detection system, compare the value of one intrusion detection system with another's, determine the value of an intrusion detector over no detector, and determine how to adjust the operation of an intrusion detector to respond to changes in its environment. General results are given and the method is illustrated in several numerical examples that involve both hypothetical and real intrusion detection systems. We demonstrate that, contrary to common advice, the value of an intrusion detection system depends not only on its ROC curve, but also on various costs (such as those associated with making incorrect decisions about detection) and the hostility of the operating environment. Conclusions are drawn about the design and evaluation of intrusion detection systems and the role for decision analysis in that design and evaluation.

[1]  Sean Dougherty,et al.  Edge detector evaluation using empirical ROC curves , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[2]  Salvatore J. Stolfo,et al.  A framework for constructing features and models for intrusion detection systems , 2000, TSEC.

[3]  Andrew P. Bradley,et al.  The use of the area under the ROC curve in the evaluation of machine learning algorithms , 1997, Pattern Recognit..

[4]  Thomas L. Magnanti,et al.  Applied Mathematical Programming , 1977 .

[5]  Douglas A. Reynolds,et al.  The NIST speaker recognition evaluation - Overview, methodology, systems, results, perspective , 2000, Speech Commun..

[6]  Salvatore J. Stolfo,et al.  Cost-based modeling for fraud and intrusion detection: results from the JAM project , 2000, Proceedings DARPA Information Survivability Conference and Exposition. DISCEX'00.

[7]  R. Richards-Kortum,et al.  A comparison of C/B ratios from studies using receiver operating characteristic curve analysis. , 1999, Journal of clinical epidemiology.

[8]  James O. Larimer,et al.  Evaluation of human vision models for predicting human observer performance , 1997, Medical Imaging.

[9]  Elizabeth B. Lennon.  Testing Intrusion Detection Systems , 2003 .

[10]  John McHugh,et al.  Testing Intrusion detection systems: a critique of the 1998 and 1999 DARPA intrusion detection system evaluations as performed by Lincoln Laboratory , 2000, TSEC.

[11]  Peter G. Neumann,et al.  EMERALD: Event Monitoring Enabling Responses to Anomalous Live Disturbances , 1997, CCS 2002.

[12]  John C. Hancock,et al.  Signal Detection Theory , 1966 .

[13]  R.K. Cunningham,et al.  Evaluating intrusion detection systems: the 1998 DARPA off-line intrusion detection evaluation , 2000, Proceedings DARPA Information Survivability Conference and Exposition. DISCEX'00.

[14]  Tom Fawcett,et al.  Robust Classification for Imprecise Environments , 2000, Machine Learning.

[15]  Ralph D. Hippenstiel,et al.  Detection Theory: Applications and Digital Signal Processing , 2001 .

[16]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[17]  Peter G. Neumann,et al.  Experience with EMERALD to Date , 1999, Workshop on Intrusion Detection and Network Monitoring.

[18]  Alvin F. Martin,et al.  The NIST 1999 Speaker Recognition Evaluation - An Overview , 2000, Digit. Signal Process..

[19]  John McHugh,et al.  Defending Yourself: The Role of Intrusion Detection Systems , 2000, IEEE Software.

[20]  S. Kent,et al.  On the trail of intrusions into information systems , 2000 .

[21]  John E. Gaffney,et al.  Evaluation of intrusion detectors: a decision theory approach , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[22]  H. Vincent Poor,et al.  An Introduction to Signal Detection and Estimation , 1994, Springer Texts in Electrical Engineering.

[23]  Eric Miller,et al.  Testing and evaluating computer intrusion detection systems , 1999, CACM.

[24]  Jacob Benesty,et al.  An objective technique for evaluating doubletalk detectors in acoustic echo cancelers , 1999, IEEE Trans. Speech Audio Process..

[25]  Stefan Axelsson,et al.  The base-rate fallacy and the difficulty of intrusion detection , 2000, TSEC.