Online Appendix To: Evaluating Computer Intrusion Detection Systems: a Survey of Common Practices A. Evaluation of Intrusion Detection Systems: Historical Overview

In Figure 11, we depict chronologically ordered dates that mark major developments in the area of intrusion detection system (IDS) evaluation from its inception until the present date. The earliest effort on evaluating IDSes in a systematic manner is the work of Puketza et al. [1996, 1997]. These authors presented an approach for evaluating IDSes based on principles of the field of software systems testing. They were the first to develop a framework for evaluating IDSes, which they describe in detail in their work from 1997. They used the framework to evaluate a network-based IDS in terms of attack detection accuracy, resource consumption, and performance under stress. multiple IDSes using generated trace files that contain host and network activities of benign and malicious nature. The latter are commonly known as the DARPA datasets (see Section 2.1). Cunningham et al. [1999] describe the approach taken to generate the DARPA datasets in detail. The DARPA datasets are still extensively used in IDS evaluation studies. Debar et al. [1998] from the IBM Zurich Research Laboratory developed a workbench for evaluating IDSes. The workbench enabled the execution of attack scripts stored in a database maintained internally at IBM and the generation of regular workloads for training anomaly based IDSes. Debar et al. demonstrated the use of the workbench by evaluating multiple host-based IDSes. A recent effort to support the rigorous evaluation of IDSes is being driven by Symantec. Dumitras and Shou [2011] presented Symantec's Worldwide Intelligence Network Environment (WINE) datasets, 45 which contain local and remote attacks (see Table I). They also presented an evaluation platform that makes use of the datasets and is available for use by researchers for evaluating security mechanisms. However, since the datasets are captured from real network infrastructures and systems, and therefore contain private user data, they can only be accessed on-site at Symantec to avoid legal issues. The large scale of this project is indicated by the fact that Syman-tec continuously monitors and records malicious activities using more than 240,000 sensors deployed in 200 countries. In addition to attacks, which can be used for evaluating IDSes, the WINE datasets contain samples of malware (i.e., malicious software like trojans or viruses), which 45