Comparing Agents' Success against People in Security Domains

The interaction of people with autonomous agents has become increasingly prevalent. Some of these settings include security domains, where people can be characterized as uncooperative, hostile, manipulative, and tending to take advantage of the situation for their own needs. This makes it challenging to design proficient agents to interact with people in such environments. Evaluating the success of the agents automatically before evaluating them with people or deploying them could alleviate this challenge and result in better designed agents. In this paper we show how Peer Designed Agents (PDAs) – computer agents developed by human subjects – can be used as a method for evaluating autonomous agents in security domains. Such evaluation can reduce the effort and costs involved in evaluating autonomous agents interacting with people to validate their efficacy. Our experiments included more than 70 human subjects and 40 PDAs developed by students. The study provides empirical support that PDAs can be used to compare the proficiency of autonomous agents when matched with people in security domains.

[1]  Sarit Kraus,et al.  Adversarial Uncertainty in Multi-Robot Patrol , 2009, IJCAI.

[2]  A. Roth,et al.  Predicting How People Play Games: Reinforcement Learning in Experimental Games with Unique, Mixed Strategy Equilibria , 1998 .

[3]  R. McKelvey,et al.  An experimental study of the centipede game , 1992 .

[4]  M. Benda,et al.  On Optimal Cooperation of Knowledge Sources , 1985 .

[5]  Sarit Kraus,et al.  Multi-robot perimeter patrol in adversarial settings , 2008, 2008 IEEE International Conference on Robotics and Automation.

[6]  Sarit Kraus,et al.  Using Game Theory for Los Angeles Airport Security , 2009, AI Mag..

[7]  Shou-De Lin,et al.  Designing the Market Game for a Trading Agent Competition , 2001, IEEE Internet Comput..

[8]  Ya'akov Gal,et al.  Facilitating the Evaluation of Automated Negotiators using Peer Designed Agents , 2010, AAAI.

[9]  Sarit Kraus,et al.  The impact of adversarial knowledge on adversarial planning in perimeter patrol , 2008, AAMAS.

[10]  Sarit Kraus,et al.  The influence of social dependencies on decision-making: initial investigations with a new game , 2004, Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, 2004. AAMAS 2004..

[11]  James K. Sebenius,et al.  Thinking Coalitionally: Party Arithmetic, Process Opportunism, and Strategic Sequencing , 1992 .

[12]  Nicola Basilico,et al.  Finding the optimal strategies for robotic patrolling with adversaries in topologically-represented environments , 2009, 2009 IEEE International Conference on Robotics and Automation.

[13]  Sarit Kraus,et al.  Robust solutions to Stackelberg games: Addressing bounded rationality and limited observations in human cognition , 2010, Artif. Intell..