Ensemble Clustering for Internet Security Applications

Due to their damage to Internet security, malware and phishing website detection has been the Internet security topics that are of great interests. Compared with malware attacks, phishing website fraud is a relatively new Internet crime. However, they share some common properties: 1) both malware samples and phishing websites are created at a rate of thousands per day driven by economic benefits; and 2) phishing websites represented by the term frequencies of the webpage content share similar characteristics with malware samples represented by the instruction frequencies of the program. Over the past few years, many clustering techniques have been employed for automatic malware and phishing website detection. In these techniques, the detection process is generally divided into two steps: 1) feature extraction, where representative features are extracted to capture the characteristics of the file samples or the websites; and 2) categorization, where intelligent techniques are used to automatically group the file samples or websites into different classes based on computational analysis of the feature representations. However, few have been applied in real industry products. In this paper, we develop an automatic categorization system to automatically group phishing websites or malware samples using a cluster ensemble by aggregating the clustering solutions that are generated by different base clustering algorithms. We propose a principled cluster ensemble framework to combine individual clustering solutions that are based on the consensus partition, which can not only be applied for malware categorization, but also for phishing website clustering. In addition, the domain knowledge in the form of sample-level/website-level constraints can be naturally incorporated into the ensemble framework. The case studies on large and real daily phishing websites and malware collection from the Kingsoft Internet Security Laboratory demonstrate the effectiveness and efficiency of our proposed method.

[1]  Fei Wang,et al.  Generalized Cluster Aggregation , 2009, IJCAI.

[2]  Chris H. Q. Ding,et al.  Weighted Consensus Clustering , 2008, SDM.

[3]  Joydeep Ghosh,et al.  Cluster Ensembles --- A Knowledge Reuse Framework for Combining Multiple Partitions , 2002, J. Mach. Learn. Res..

[4]  Christopher K. I. Williams.  A MCMC Approach to Hierarchical Mixture Modelling , 1999, NIPS.

[5]  Nicholas Nethercote,et al.  Valgrind: a framework for heavyweight dynamic binary instrumentation , 2007, PLDI '07.

[6]  John Yearwood,et al.  Consensus Clustering and Supervised Classification for Profiling Phishing Emails in Internet Commerce Security , 2010, PKAW.

[7]  Aristides Gionis,et al.  Clustering aggregation , 2005, 21st International Conference on Data Engineering (ICDE'05).

[8]  Tao Li,et al.  An intelligent PE-malware detection system based on association mining , 2008, Journal in Computer Virology.

[9]  Y. Fukuyama,et al.  A new method of choosing the number of clusters for the fuzzy c-mean method , 1989 .

[10]  Lior Rokach,et al.  Improving malware detection by applying multi-inducer ensemble , 2009, Comput. Stat. Data Anal..

[11]  Jill P. Mesirov,et al.  Consensus Clustering: A Resampling-Based Method for Class Discovery and Visualization of Gene Expression Microarray Data , 2003, Machine Learning.

[12]  Lorrie Faith Cranor,et al.  Cantina: a content-based approach to detecting phishing web sites , 2007, WWW '07.

[13]  Hans-Peter Kriegel,et al.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise , 1996, KDD.

[14]  Yanfang Ye,et al.  IMDS: intelligent malware detection system , 2007, KDD '07.

[15]  Gang Liu,et al.  Automatic Detection of Phishing Target from Phishing Webpage , 2010, 2010 20th International Conference on Pattern Recognition.

[16]  Fadi A. Thabtah,et al.  Predicting Phishing Websites Using Classification Mining Techniques with Experimental Case Studies , 2010, 2010 Seventh International Conference on Information Technology: New Generations.

[17]  Olatz Arbelaitz,et al.  Evaluation of Malware clustering based on its dynamic behaviour , 2008, AusDM.

[18]  Simon Brown,et al.  Using Differencing to Increase Distinctiveness for Phishing Website Clustering , 2009, 2009 Symposia and Workshops on Ubiquitous, Autonomic and Trusted Computing.

[19]  Yanfang Ye,et al.  Intelligent file scoring system for malware detection from the gray list , 2009, KDD.

[20]  Fabrizio Sebastiani,et al.  Machine learning in automated text categorization , 2001, CSUR.

[21]  P. Rousseeuw,et al.  Partitioning Around Medoids (Program PAM) , 2008 .

[22]  Carsten Willems,et al.  Learning and Classification of Malware Behavior , 2008, DIMVA.

[23]  Maurice K. Wong,et al.  Algorithm AS136: A k-means clustering algorithm. , 1979 .

[24]  Min Wu.  Fighting phishing at the user interface , 2006 .

[25]  Christopher Krügel,et al.  Limits of Static Analysis for Malware Detection , 2007, Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007).

[26]  Yong Chen,et al.  Automatic malware categorization using cluster ensemble , 2010, KDD.

[27]  Salvatore J. Stolfo,et al.  Data mining methods for detection of new malicious executables , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[28]  Christopher Krügel,et al.  Scalable, Behavior-Based Malware Clustering , 2009, NDSS.

[29]  Masatoshi Yoshikawa,et al.  Refinement of TF-IDF schemes for web pages using their hyperlinked neighboring pages , 2003, HYPERTEXT '03.

[30]  Wenke Lee,et al.  PolyUnpack: Automating the Hidden-Code Extraction of Unpack-Executing Malware , 2006, 2006 22nd Annual Computer Security Applications Conference (ACSAC'06).

[31]  Anil K. Jain,et al.  Clustering ensembles: models of consensus and weak partitions , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Ali S. Hadi,et al.  Finding Groups in Data: An Introduction to Chster Analysis , 1991 .

[33]  Marius Gheorghescu.  AN AUTOMATED VIRUS CLASSIFICATION SYSTEM , 2006 .

[34]  Carla E. Brodley,et al.  Solving cluster ensemble problems by bipartite graph partitioning , 2004, ICML.

[35]  Somesh Jha,et al.  Synthesizing Near-Optimal Malware Specifications from Suspicious Behaviors , 2010, 2010 IEEE Symposium on Security and Privacy.

[36]  Xiaotie Deng,et al.  An antiphishing strategy based on visual similarity assessment , 2006, IEEE Internet Computing.

[37]  Yuval Elovici,et al.  Applying Machine Learning Techniques for Detection of Malicious Code in Network Traffic , 2007, KI.

[38]  Suku Nair,et al.  A comparison of machine learning techniques for phishing detection , 2007, eCrime '07.

[39]  Christopher Krügel,et al.  Exploring Multiple Execution Paths for Malware Analysis , 2007, 2007 IEEE Symposium on Security and Privacy (SP '07).

[40]  Lynn Margaret Batten,et al.  Function length as a tool for malware classification , 2008, 2008 3rd International Conference on Malicious and Unwanted Software (MALWARE).

[41]  Zhuoqing Morley Mao,et al.  Automated Classification and Analysis of Internet Malware , 2007, RAID.

[42]  Ian Davidson,et al.  Constrained Clustering: Advances in Algorithms, Theory, and Applications , 2008 .

[43]  John C. Mitchell,et al.  Client-Side Defense Against Web-Based Identity Theft , 2004, NDSS.

[44]  Rui Xu,et al.  Survey of clustering algorithms , 2005, IEEE Transactions on Neural Networks.

[45]  Cormac Herley,et al.  A profitless endeavor: phishing as tragedy of the commons , 2009, NSPW '08.

[46]  Yiming Yang,et al.  Text categorization , 2008, Scholarpedia.

[47]  Paul A. Watters,et al.  Determining provenance in phishing websites using automated conceptual analysis , 2009, 2009 eCrime Researchers Summit.

[48]  B. Ripley,et al.  Pattern Recognition , 1968, Nature.