Evolving successful stack overflow attacks for vulnerability testing

The work presented in this paper is intended to test crucial system services against stack overflow vulnerabilities. The focus of the test is the user-accessible variables, that is to say, the inputs from the user as specified at the command line or in a configuration file. The tester is defined as a process for automatically generating a wide variety of user-accessible variables that result in malicious buffers (an exploit). In this work, the search for successful exploits is formulated as an optimization problem and solved using evolutionary computation. Moreover the resulting attacks are passed through the Snort misuse detection system to observe the detection (or not) of each exploit

[1]  Conor Ryan,et al.  Grammatical evolution , 2007, GECCO '07.

[2]  Vidroha Debroy,et al.  Genetic Programming , 1998, Lecture Notes in Computer Science.

[3]  Martin Roesch,et al.  Snort - Lightweight Intrusion Detection for Networks , 1999 .

[4]  A. E. Eiben,et al.  Introduction to Evolutionary Computing , 2003, Natural Computing Series.

[5]  Giovanni Vigna,et al.  Testing network-based intrusion detection signatures using mutant exploits , 2004, CCS '04.

[6]  Michael J. Shaw,et al.  Genetic algorithms with dynamic niche sharing for multimodal function optimization , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[7]  Kalyanmoy Deb,et al.  An Investigation of Niche and Species Formation in Genetic Function Optimization , 1989, ICGA.

[8]  David A. Wagner,et al.  Mimicry attacks on host-based intrusion detection systems , 2002, CCS '02.

[9]  Kalyanmoy Deb,et al.  A Comparative Analysis of Selection Schemes Used in Genetic Algorithms , 1990, FOGA.

[10]  Somesh Jha,et al.  Static Analysis of Executables to Detect Malicious Patterns , 2003, USENIX Security Symposium.

[11]  Somesh Jha,et al.  Automatic generation and analysis of NIDS attacks , 2004, 20th Annual Computer Security Applications Conference.

[12]  Conor Ryan,et al.  Grammatical Evolution , 2001, Genetic Programming Series.

[13]  Gerry V. Dozier,et al.  Vulnerability Analysis of Immunity-Based Intrusion Detection Systems Using Evolutionary Hackers , 2004, GECCO.

[14]  Kymie M. C. Tan,et al.  Undermining an Anomaly-Based Intrusion Detection System Using Common Exploits , 2002, RAID.

[15]  Riccardo Poli,et al.  Foundations of Genetic Programming , 1999, Springer Berlin Heidelberg.

[16]  James C. Foster.  Buffer overflow attacks : detect, exploit, prevent , 2005 .

[17]  Vitaly Osipov,et al.  Buffer Overflow Attacks , 2005 .