Optimization of rootkit revealing system resources - A game theoretic approach

Malicious rootkit is a collection of programs designed with the intent of infecting and monitoring the victim computer without the user's permission. After the victim has been compromised, the remote attacker can easily cause further damage. In order to infect, compromise and monitor, rootkits adopt Native Application Programming Interface (API) hooking technique. To reveal the hidden rootkits, current rootkit detection techniques check different data structures which hold reference to Native APIs. To verify these data structures, a large amount of system resources are required. This is because of the number of APIs in these data structures being quite large. Game theoretic approach is a useful mathematical tool to simulate network attacks. In this paper, a mathematical model is framed to optimize resource consumption using game-theory. To the best of our knowledge, this is the first work to be proposed for optimizing resource consumption while revealing rootkit presence using game theory. Non-cooperative game model is taken to discuss the problem. Analysis and simulation results show that our game theoretic model can effectively reduce the resource consumption by selectively monitoring the number of APIs in windows platform.

[1]  Yu Liu,et al.  Modelling misbehaviour in ad hoc networks: a game theoretic approach for intrusion detection , 2006, Int. J. Secur. Networks.

[2]  Mishaal Abdullah Al-Kadhi.  Assessment of the status of spam in the Kingdom of Saudi Arabia , 2011, J. King Saud Univ. Comput. Inf. Sci..

[3]  Abdulkader A. Alfantookh,et al.  DoS Attacks Intelligent Detection using Neural Networks , 2006, J. King Saud Univ. Comput. Inf. Sci..

[4]  J. Neumann,et al.  Theory of games and economic behavior , 1945, 100 Years of Math Milestones.

[5]  Lin Chen,et al.  A Game Theoretical Framework on Intrusion Detection in Heterogeneous Networks , 2009, IEEE Transactions on Information Forensics and Security.

[6]  Aaron Emigh.  The Crimeware Landscape: Malware, Phishing, Identity Theft and Beyond , 2006, J. Digit. Forensic Pract..

[7]  Yueming Cai,et al.  Resource allocation based on integer programming and game theory in uplink multi-cell cooperative OFDMA systems , 2011, EURASIP J. Wirel. Commun. Netw..

[8]  Liviu Iftode,et al.  Monitoring Integrity Using Limited Local Memory , 2013, IEEE Transactions on Information Forensics and Security.

[9]  Yoshiyasu Takefuji,et al.  Towards a tamper-resistant kernel rootkit detector , 2007, SAC '07.

[10]  Murali S. Kodialam,et al.  Detecting network intrusions via sampling: a game theoretic approach , 2003, IEEE INFOCOM 2003. Twenty-second Annual Joint Conference of the IEEE Computer and Communications Societies (IEEE Cat. No.03CH37428).

[11]  Prabir Bhattacharya,et al.  Game theoretic models for detecting network intrusions , 2008, Comput. Commun..

[12]  Yu Liu,et al.  A Game Theoretic Approach to Efficient Mixed Strategies for Intrusion Detection , 2006, 2006 IEEE International Conference on Communications.

[13]  T. Basar,et al.  A game theoretic analysis of intrusion detection in access control systems , 2004, 2004 43rd IEEE Conference on Decision and Control (CDC) (IEEE Cat. No.04CH37601).

[14]  Hesham Altwaijry,et al.  Bayesian based intrusion detection system , 2012, J. King Saud Univ. Comput. Inf. Sci..

[15]  Hesham Altwaijry,et al.  Bayesian based intrusion detection system , 2012, J. King Saud Univ. Comput. Inf. Sci..

[16]  Shrisha Rao,et al.  Resource Allocation in Cloud Computing Using the Uncertainty Principle of Game Theory , 2016, IEEE Systems Journal.

[17]  Shuai Liu,et al.  A Game Theoretic Approach to Optimize the Performance of Host-Based IDS , 2008, 2008 IEEE International Conference on Wireless and Mobile Computing, Networking and Communications.

[18]  Huiqun Yu,et al.  A Game Theory Approach to Fair and Efficient Resource Allocation in Cloud Computing , 2014 .

[19]  Partha Dasgupta,et al.  Kernel and Application Integrity Assurance: Ensuring Freedom from Rootkits and Malware in a Computer System , 2007, 21st International Conference on Advanced Information Networking and Applications Workshops (AINAW'07).

[20]  Peng Liu.  A Game Theoretic Approach to Cyber Attack Prediction , 2005 .

[21]  Salim Hariri,et al.  Game Theory Based Network Security , 2010, J. Information Security.