Beware the Soothsayer: From Attack Prediction Accuracy to Predictive Reliability in Security Games

Interdicting the flow of illegal goods (such as drugs and ivory) is a major security concern for many countries. The massive scale of these networks, however, forces defenders to make judicious use of their limited resources. While existing solutions model this problem as a Network Security Game (NSG), they do not consider humans’ bounded rationality. Previous human behavior modeling works in Security Games, however, make use of large training datasets that are unrealistic in real-world situations; the ability to effectively test many models is constrained by the time-consuming and complex nature of field deployments. In addition, there is an implicit assumption in these works that a model’s prediction accuracy strongly correlates with the performance of its corresponding defender strategy (referred to as predictive reliability). If the assumption of predictive reliability does not hold, then this could lead to substantial losses for the defender. In the following paper, we (1) first demonstrate that predictive reliability is indeed strong for previous Stackelberg Security Game experiments. We also run our own set of human subject experiments in such a way that models are restricted to learning on dataset sizes representative of real-world constraints. In the analysis on that data, we demonstrate that (2) predictive reliability is extremely weak for NSGs. Following that discovery, however, we identify (3) key factors that influence predictive reliability results: the training set’s exposed attack surface and graph structure.

[1]  Sarit Kraus,et al.  Game-Theoretic Patrolling with Dynamic Execution Uncertainty and a Case Study on a Real Transit System , 2014, J. Artif. Intell. Res..

[2]  Rong Yang,et al.  Modeling Human Bounded Rationality to Improve Defender Strategies in Network Security Games , 2012 .

[3]  David P. Morton,et al.  Models for nuclear smuggling interdiction , 2007 .

[4]  Milind Tambe Security and Game Theory - Algorithms, Deployed Systems, Lessons Learned , 2011 .

[5]  David Eppstein,et al.  Studying (non-planar) road networks through an algorithmic lens , 2008, GIS '08.

[6]  Gary Charness,et al.  Experimental methods: Between-subject and within-subject design , 2012 .

[7]  Milind Tambe,et al.  Game-Theoretic Security Patrolling with Dynamic Execution Uncertainty and a Case Study on a Real Transit System , 2014 .

[8]  Vincent Conitzer,et al.  Security scheduling for real-world networks , 2013, AAMAS.

[9]  Milind Tambe,et al.  Urban security: game-theoretic resource allocation in networked physical domains , 2010, AAAI 2010.

[10]  Milind Tambe,et al.  "A Game of Thrones": When Human Behavior Models Compete in Repeated Stackelberg Security Games , 2015, AAMAS.

[11]  Richard S. John,et al.  Empirical Comparisons of Descriptive Multi-objective Adversary Models in Stackelberg Security Games , 2014, GameSec.

[12]  Vincent Conitzer,et al.  A double oracle algorithm for zero-sum security games on graphs , 2011, AAMAS.

[13]  Feng Pan,et al.  Optimal Interdiction of Unreactive Markovian Evaders , 2009, CPAIOR.

[14]  Bo An,et al.  PROTECT: a deployed game theoretic system to protect the ports of the United States , 2012, AAMAS.

[15]  Nicole D. Sintov,et al.  Human Adversaries in Opportunistic Crime Security Games: Evaluating Competing Bounded Rationality Models , 2015 .

[16]  Amos Azaria,et al.  Analyzing the Effectiveness of Adversary Modeling in Security Games , 2013, AAAI.

[17]  Rong Yang,et al.  Computing optimal strategy against quantal response in security games , 2012, AAMAS.

[18]  M G H Bell,et al.  Attacker–defender models and road network vulnerability , 2008, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[19]  Jeannette M. Wing,et al.  Measuring a System's Attack Surface , 2004 .

[20]  Colin Camerer Behavioral Game Theory: Experiments in Strategic Interaction , 2003 .

[21]  Milind Tambe,et al.  Comparing human behavior models in repeated Stackelberg security games: An extended study , 2016, Artif. Intell..

[22]  José R. Correa,et al.  Fare Evasion in Transit Networks , 2017, Oper. Res..