A human-in-the-loop approach to understanding situation awareness in cyber defence analysis

In this paper we argue for a human-in-the-loop approach to the study of situation awareness in computer defence analysis (CDA). The cognitive phenomenon of situation awareness (SA) has received significant attention in cybersecurity/CDA research. Yet little of this work has attended to the cognitive aspects of situation awareness in the CDA context; instead, the human operator has been treated as an abstraction within the larger human-technology system. A more human-centric approach that seeks to understand the socio-cognitive work of human operators as they perform CDA will yield greater insights into the design of tools and interfaces for CDA. As support for this argument, we present our own work employing the Living Lab Framework through which we ground our experimental findings in contextual knowledge of real-world practice.

[1]  Guy H. Walker,et al.  Event analysis of systemic teamwork (EAST): a novel integration of ergonomics methods to analyse C4i activity , 2006, Ergonomics.

[2]  Michael W. Boyce,et al.  Human Performance in Cybersecurity , 2011 .

[3]  Neville Stanton,et al.  Situation awareness measurement: a review of applicability for C4i environments. , 2006, Applied ergonomics.

[4]  N A Stanton,et al.  Distributed situation awareness in dynamic systems: theoretical development and application of an ergonomics methodology , 2006, Ergonomics.

[5]  John J. Salerno,et al.  Realizing situation awareness within a cyber environment , 2006, SPIE Defense + Commercial Sensing.

[6]  Gregory J. Conti,et al.  Toward Instrumenting Network Warfare Competitions to Generate Labeled Datasets , 2009, CSET.

[7]  Mica R. Endsley,et al.  Theoretical Underpinnings of Situation Awareness, A Critical Review , 2000 .

[8]  Aleksander P. J. Ellis.  System Breakdown: The Role of Mental Models and Transactive Memory in the Relationship between Acute Stress and Team Performance , 2006 .

[9]  Adam Stotz,et al.  INformation fusion engine for real-time decision-making (INFERD): A perceptual system for cyber attack tracking , 2007, 2007 10th International Conference on Information Fusion.

[10]  Wayne G. Lutters,et al.  I know my network: collaboration and expertise in intrusion detection , 2004, CSCW.

[11]  Mica R. Endsley,et al.  Toward a Theory of Situation Awareness in Dynamic Systems , 1995, Hum. Factors.

[12]  Kyle Lewis,et al.  Knowledge and Performance in Knowledge-Worker Teams: A Longitudinal Study of Transactive Memory Systems , 2004, Manag. Sci..

[13]  Michael D. McNeese,et al.  Advancing Socio-Technical Systems Design Via the Living Laboratory , 2000 .

[14]  Nancy J. Cooke,et al.  Measuring team situation awareness in decentralized command and control environments , 2006, Ergonomics.

[15]  S. Gronlund,et al.  Situation Awareness , 2006 .

[16]  N. Stanton,et al.  Is situation awareness all in the mind? , 2010 .

[17]  George P. Tadda,et al.  Overview of Cyber Situation Awareness , 2010, Cyber Situational Awareness.

[18]  Ann Blandford,et al.  Situation awareness in emergency medical dispatch , 2004, Int. J. Hum. Comput. Stud..

[19]  D. Wegner,et al.  Cognitive interdependence in close relationships , 1985 .

[20]  Pascal Vasseur,et al.  Introduction to multi-sensor data fusion , 2004 .

[21]  Mica R. Endsley,et al.  Situation Awareness Information Requirements Analysis for En Route Air Traffic Control , 1994 .

[22]  Mica R. Endsley,et al.  Situation awareness global assessment technique (SAGAT) , 1988, Proceedings of the IEEE 1988 National Aerospace and Electronics Conference.

[23]  Michael D. McNeese,et al.  Using the Neocities 3.1 Simulation to Study and Measure Team Cognition , 2010 .

[24]  Richard Lippmann,et al.  The 1999 DARPA off-line intrusion detection evaluation , 2000, Comput. Networks.

[25]  Aleksander P. J. Ellis,et al.  The Effects of Critical Team Member Assertiveness on Team Performance and Satisfaction , 2006 .

[26]  John Yen,et al.  RPD-based Hypothesis Reasoning for Cyber Situation Awareness , 2010, Cyber Situational Awareness.

[27]  Guy H. Walker,et al.  What Really Is Going on? Review, Critique and Extension of Situation Awareness Theory , 2007, HCI.

[28]  Rajeev Sharma,et al.  GeoCollaborative crisis management: designing technologies to meet real-world needs , 2006, DG.O.

[29]  Jakob E. Bardram,et al.  AwareMedia: a shared interactive display supporting social, temporal, and spatial awareness in surgery , 2006, CSCW '06.

[30]  A. D'Amico,et al.  Methods of visualizing temporal patterns in and mission impact of computer security breaches , 2001, Proceedings DARPA Information Survivability Conference and Exposition II. DISCEX'01.

[31]  R. Moreland,et al.  Exploring the Performance Benefits of Group Training: Transactive Memory or Improved Communication? , 2000 .

[32]  Brian Hazlehurst,et al.  Distributed cognition in the heart room: How situation awareness arises from coordinated communications during cardiac surgery , 2007, J. Biomed. Informatics.

[33]  William M. Jones,et al.  Situation Awareness Information Requirements for Commercial Airline Pilots , 1998 .

[34]  Robert F. Mills,et al.  Developing Systems for Cyber Situational Awareness , 2009 .

[35]  Adam Stotz,et al.  High level information fusion for tracking and projection of multistage cyber attacks , 2009, Inf. Fusion.

[36]  Lundy Lewis,et al.  Insider threat detection using situation-aware MAS , 2008, 2008 11th International Conference on Information Fusion.

[37]  Han Tin French,et al.  Measurement Of Situation Awareness In A C4ISR Experiment , 2002 .

[38]  Michael Tyworth,et al.  An Alternative Framework for Research on Situational Awareness in Computer Network Defense , 2012 .

[39]  Peng Liu,et al.  Using Bayesian networks for cyber security analysis , 2010, 2010 IEEE/IFIP International Conference on Dependable Systems & Networks (DSN).

[40]  Rashaad E. T. Jones,et al.  The Neocities Simulation: Understanding the Design and Experimental Methodology Used to Develop a Team Emergency Management Simulation , 2005 .

[41]  Shambhu J. Upadhyaya,et al.  An alert fusion framework for situation awareness of coordinated multistage attacks , 2005, Third IEEE International Workshop on Information Assurance (IWIA'05).

[42]  Shanchieh Jay Yang,et al.  Intrusion activity projection for cyber situational awareness , 2008, 2008 IEEE International Conference on Intelligence and Security Informatics.

[43]  John Yen,et al.  Cyber SA: Situational Awareness for Cyber Defense , 2010, Cyber Situational Awareness.

[44]  J. E. Groves,et al.  Made in America: Science, Technology and American Modernist Poets , 1989 .

[45]  John D. Lee,et al.  Augmenting the operator function model with cognitive operations: assessing the cognitive demands of technological innovation in ship navigation , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[46]  Guy H. Walker,et al.  What really is going on? Review of situation awareness models for individuals and teams , 2008 .

[47]  Susan Leigh Star,et al.  The Structure of Ill-Structured Solutions: Boundary Objects and Heterogeneous Distributed Problem Solving , 1989, Distributed Artificial Intelligence.

[48]  Rashaad E. T. Jones,et al.  A Distributed Cognition Simulation Involving Homeland Security and Defense: The Development of Neocities , 2004 .

[49]  Wanda J. Orlikowski,et al.  Research Commentary: Desperately Seeking the "IT" in IT Research - A Call to Theorizing the IT Artifact , 2001, Inf. Syst. Res..

[50]  Daniel R. Tesone,et al.  Achieving Cyber Defense Situational Awareness: A Cognitive Task Analysis of Information Assurance Analysts , 2005 .

[51]  Anthony C. Robinson,et al.  The geoviz toolkit: using component-oriented coordination methods for geographic visualization and analysis , 2011, Int. J. Geogr. Inf. Sci..

[52]  R.K. Cunningham,et al.  Evaluating intrusion detection systems: the 1998 DARPA off-line intrusion detection evaluation , 2000, Proceedings DARPA Information Survivability Conference and Exposition. DISCEX'00.

[53]  Mica R. Endsley,et al.  Measurement of Situation Awareness in Dynamic Systems , 1995, Hum. Factors.

[54]  Mica R. Endsley,et al.  A Survey of Situation Awareness Requirements in Air-to-Air Combat Fighters , 1993 .

[55]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[56]  Anna De Fina,et al.  The ethnographic interview , 2019, The Routledge Handbook of Linguistic Ethnography.