A Malware Detector Placement Game for Intrusion Detection

We propose and investigate a game-theoretic approach to the malware filtering and detector placement problem which arises in network security. Our main objective is to develop optimal detector algorithms taking into account attacker strategies and actions. Assuming rational and intelligent attackers, we present a two-person zero-sum non-cooperative Markov security game framework as a basis for modeling the interaction between the attackers who generate malware traffic on a network and a corresponding intrusion detection system (IDS). Thus, we establish a formal model of the detector placement problem based on game theory and derive optimal strategies for both players. In addition, we test the strategies obtained in a realistic agent-based network simulation environment and compare the results of static and dynamic placement scenarios. The obtained IDS strategies and the corresponding simulation results provide interesting insights into how to optimally deploy malware detectors in a network environment.

[1]  Murali S. Kodialam,et al.  Detecting network intrusions via sampling: a game theoretic approach , 2003, IEEE INFOCOM 2003. Twenty-second Annual Joint Conference of the IEEE Computer and Communications Societies (IEEE Cat. No.03CH37428).

[2]  T. Başar,et al.  Dynamic Noncooperative Game Theory , 1982 .

[3]  Lixia Zhang,et al.  On the placement of Internet instrumentation , 2000, Proceedings IEEE INFOCOM 2000. Conference on Computer Communications. Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat. No.00CH37064).

[4]  George Varghese,et al.  Building a better NetFlow , 2004, SIGCOMM 2004.

[5]  Michael Bloem,et al.  Malware Filtering for Network Security Using Weighted Optimality Measures , 2007, 2007 IEEE International Conference on Control Applications.

[6]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[7]  Christophe Diot,et al.  Reformulating the Monitor Placement Problem: Optimal Network-Wide Sampling , 2006 .

[8]  T. Başar,et al.  An Intrusion Detection Game with Limited Observations , 2005 .

[9]  Alejandro López-Ortiz,et al.  On the number of distributed measurement points for network tomography , 2003, IMC '03.

[10]  Jan Keiser,et al.  Agent-based telematic services and telecom applications , 2001, CACM.

[11]  T. Başar,et al.  Dynamic Noncooperative Game Theory, 2nd Edition , 1998 .

[12]  Michael L. Littman,et al.  Markov Games as a Framework for Multi-Agent Reinforcement Learning , 1994, ICML.

[13]  U. Brandes A faster algorithm for betweenness centrality , 2001 .