An Enhanced Automated Signature Generation Algorithm for Polymorphic Malware Detection

Polymorphic malware is a secure menace for application of computer network systems because hacker can evade detection and launch stealthy attacks. In this paper, a novel enhanced automated signature generation (EASG) algorithm to detect polymorphic malware is proposed. The EASG algorithm is composed of enhanced-expectation maximum algorithm and enhanced K-means clustering algorithm. In EASG algorithm, the fixed threshold value is replaced by the decision threshold of interval area. The false positive ratio can be controlled at low level, and the iterative operations and the execution time are effectively reduced. Moreover, the centroid updating is realized by application of similarity metric of Mahalanobis distance and incremental learning. Different malware group families are partitioned by the centroid updating.

[1]  Salvatore J. Stolfo,et al.  Anomalous Payload-Based Network Intrusion Detection , 2004, RAID.

[2]  Pranab Banerjee,et al.  A Multi-Layered Approach to Botnet Detection , 2008, Security and Management.

[3]  Biplab Sikdar,et al.  A Quasi-Species Approach for Modeling the Dynamics of Polymorphic Worms , 2006, Proceedings IEEE INFOCOM 2006. 25TH IEEE International Conference on Computer Communications.

[4]  Mattia Monga,et al.  LISABETH: automated content-based signature generator for zero-day polymorphic worms , 2008, SESS '08.

[5]  Ming-Yang Kao,et al.  Hamsa: fast signature generation for zero-day polymorphic worms with provable attack resilience , 2006, 2006 IEEE Symposium on Security and Privacy (S&P'06).

[6]  Ibrahim Sogukpinar,et al.  Polymorphic worm detection using token-pair signatures , 2008, SecPerU '08.

[7]  Siwei Luo,et al.  Entropy based soft K-means clustering , 2008, 2008 IEEE International Conference on Granular Computing.

[8]  Yan Chen,et al.  Botnet Research Survey , 2008, 2008 32nd Annual IEEE International Computer Software and Applications Conference.

[9]  Greg Goth.  Fast-Moving Zombies: Botnets Stay a Step Ahead of the Fixes , 2007, IEEE Internet Computing.

[10]  Stefan Savage,et al.  Inside the Slammer Worm , 2003, IEEE Secur. Priv..

[11]  Christopher Krügel,et al.  Polymorphic Worm Detection Using Structural Information of Executables , 2005, RAID.

[12]  Somesh Jha,et al.  Semantics-aware malware detection , 2005, 2005 IEEE Symposium on Security and Privacy (S&P'05).

[13]  Michael K. Reiter,et al.  Traffic Aggregation for Malware Detection , 2008, DIMVA.

[14]  James Newsome,et al.  Polygraph: automatically generating signatures for polymorphic worms , 2005, 2005 IEEE Symposium on Security and Privacy (S&P'05).

[15]  Giovanni Vigna,et al.  Feature Omission Vulnerabilities: Thwarting Signature Generation for Polymorphic Worms , 2007, Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007).

[16]  Jung-Min Park,et al.  An overview of anomaly detection techniques: Existing solutions and latest technological trends , 2007, Comput. Networks.

[17]  D.M. Mount,et al.  An Efficient k-Means Clustering Algorithm: Analysis and Implementation , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Philip S. Yu,et al.  Top 10 algorithms in data mining , 2007, Knowledge and Information Systems.

[19]  Kouichi Sakurai,et al.  Analyzing Maximum Length of Instruction Sequence in Network Packets for Polymorphic Worm Detection , 2008, 2008 International Conference on Multimedia and Ubiquitous Engineering (mue 2008).

[20]  Vinod Yegneswaran,et al.  An Inside Look at Botnets , 2007, Malware Detection.

[21]  Zhenkai Liang,et al.  Fast and automated generation of attack signatures: a basis for building self-protecting servers , 2005, CCS '05.

[22]  Yan Chen,et al.  Network-based and Attack-resilient Length Signature Generation for Zero-day Polymorphic Worms , 2007, 2007 IEEE International Conference on Network Protocols.