Evolving TCP/IP packets: A case study of port scans

In this work, we investigate the ability of genetic programming techniques to evolve valid network packets, including all relevant header values, towards a specific goal. We see this as a first step in building a fuzzing system that can learn to adapt for vulnerability analysis. By developing a system that learns the packets that are required to be transmitted towards targets, using feedback from an external network source, we make a step towards having a system that can intelligently explore the capabilities of a given security system. In order to validate our system's capabilities we evolve a variety of port scan patterns while running the packets through an IDS, with the goal to minimizes the alarms raised during the scanning process. Results show that the system not only successfully evolves valid TCP packets, but also remains stealthy in its activity.

[1]  Martin Roesch,et al.  Snort - Lightweight Intrusion Detection for Networks , 1999 .

[2]  Matthew M. Williamson,et al.  Throttling viruses: restricting propagation to defeat malicious mobile code , 2002, 18th Annual Computer Security Applications Conference, 2002. Proceedings..

[3]  Syed Ali Khayam,et al.  A Comparative Evaluation of Anomaly Detectors under Portscan Attacks , 2008, RAID.

[4]  David A. Wagner,et al.  Mimicry attacks on host-based intrusion detection systems , 2002, CCS '02.

[5]  Malcolm I. Heywood,et al.  Evolving Buffer Overflow Attacks with Detector Feedback , 2007, EvoWorkshops.

[6]  Stuart E. Schechter,et al.  Fast Detection of Scanning Worm Infections , 2004, RAID.

[7]  Hari Balakrishnan,et al.  Fast portscan detection using sequential hypothesis testing , 2004, IEEE Symposium on Security and Privacy, 2004. Proceedings. 2004.

[8]  Michel Cukier,et al.  An experimental evaluation to determine if port scans are precursors to an attack , 2005, 2005 International Conference on Dependable Systems and Networks (DSN'05).

[9]  Donald F. Towsley,et al.  Detecting anomalies in network traffic using maximum entropy estimation , 2005, IMC '05.

[10]  Vern Paxson,et al.  Bro: a system for detecting network intruders in real-time , 1998, Comput. Networks.

[11]  Salvatore J. Stolfo,et al.  Surveillance detection in high bandwidth environments , 2003, Proceedings DARPA Information Survivability Conference and Exposition.

[12]  Malcolm I. Heywood,et al.  On evolving buffer overflow attacks using genetic programming , 2006, GECCO '06.

[13]  Stuart Staniford-Chen,et al.  Practical Automated Detection of Stealthy Portscans , 2002, J. Comput. Secur..

[14]  Vern Paxson,et al.  Proceedings of the 13th USENIX Security Symposium , 2022 .