Situation Awareness and campaign assessments

Military-led nation building campaigns can face many challenges as advances in certain aspects such as the economy and infrastructure may well be accompanied by setbacks in other areas such as security, making it difficult to judge overall progress. In this context, campaign assessments provide vital decision support for adapting strategy and deploying resources to best effect. The value of such assessments is enhanced when they report not only what has happened but why and when they also indicate likely future events—mirroring the three levels in Endsley’s model of Situation Awareness. Our aim is to develop a process for evaluating changes to the complex organisational structure and processes used for producing assessment documents, even if it is difficult to understand the linkages between such changes and output quality. Using 2 years’ worth of Australian Defence Force quarterly campaign assessments, we study the effect of adoption of a new analysis framework in their production. Using the conventions of thematic analysis and Endsley’s model of Situation Awareness, we introduce the Situation Awareness Elicitation method for obtaining metrics pertaining to the value of assessments before and after introduction of the framework and perform statistical analysis on this data.

[1]  Guy H. Walker,et al.  Is it really better to share? Distributed situation awareness and its implications for collaborative system design , 2010 .

[2]  Alison Hickman.  Assisting Strategic Analysis in an Uncertain World , 2008 .

[3]  Miguel P Caldas,et al.  Research design: qualitative, quantitative, and mixed methods approaches , 2003 .

[4]  J. Boyd,et al.  A Discourse on Winning and Losing , 1987 .

[5]  Guy H. Walker,et al.  Event analysis of systemic teamwork (EAST): a novel integration of ergonomics methods to analyse C4i activity , 2006, Ergonomics.

[6]  H. Rittel,et al.  Dilemmas in a general theory of planning , 1973 .

[7]  David J. Weiss,et al.  A Science of Decision Making , 2008 .

[8]  Irena M. Ali,et al.  Methodological Approaches for Researching Complex Organizational Phenomena , 2014, Informing Sci. Int. J. an Emerg. Transdiscipl..

[9]  Mica R. Endsley,et al.  Toward a Theory of Situation Awareness in Dynamic Systems , 1995, Hum. Factors.

[10]  Peter A. Tamás,et al.  Should I take this seriously? A simple checklist for calling bullshit on policy supporting research , 2014 .

[11]  Jonathan Rosenhead,et al.  Problem structuring methods in action , 2004, Eur. J. Oper. Res..

[12]  Programme Manager's Planning Monitoring & Evaluation Toolkit Tool Number 4: Stakeholder Participation in Monitoring and Evaluation , 2022 .

[13]  C. Cassell,et al.  Essential guide to qualitative methods in organizational research , 2004 .

[14]  Jason Scholz,et al.  A Dialectic for Network Centric Warfare , 2005 .

[15]  Edward H. S. Lo,et al.  Evaluating impacts of system integration on joint fires operations , 2014, 2014 Military Communications and Information Systems Conference (MilCIS).

[16]  Keith Punch,et al.  Introduction to Social Research: Quantitative and Qualitative Approaches , 1998 .

[17]  L. Goddard,et al.  Operations Research (OR) , 2007 .

[18]  Ben Connable,et al.  Embracing the Fog of War: Assessment and Metrics in Counterinsurgency , 2012 .

[19]  Van Nederveen,et al.  Science, Strategy and War: The Strategic Theory of John Boyd , 2008 .

[20]  Jeff Conklin,et al.  Dialogue Mapping: Building Shared Understanding of Wicked Problems , 2005 .

[21]  Charles Tustin Kamps.  Effects-Based Operations , 2004 .

[22]  Eduardo Salas,et al.  Situation Awareness in Team Performance: Implications for Measurement and Training , 1995, Hum. Factors.

[23]  David Wainwright,et al.  Issues and challenges in the use of template analysis: twocomparative case studies from the field , 2008 .

[24]  Mica R. Endsley,et al.  Design and Evaluation for Situation Awareness Enhancement , 1988 .

[25]  K. Sutcliffe,et al.  Controlling Decision-Making Practice in Organizations , 2001 .

[26]  L. Alberto Franco,et al.  Taking problem structuring methods forward , 2007, J. Oper. Res. Soc..

[27]  Edward A. Smith.  Complexity, Networking, & Effects-Based Approaches to Operations , 2006 .

[28]  N A Stanton,et al.  Distributed situation awareness in dynamic systems: theoretical development and application of an ergonomics methodology , 2006, Ergonomics.

[29]  David Lindley,et al.  Introduction to the Practice of Statistics , 1990, The Mathematical Gazette.

[30]  Barry D. Watts.  Clausewitzian Friction and Future War, Revised Edition , 2004 .

[31]  Nigel King,et al.  Using templates in the thematic analysis of text , 2004 .

[32]  Phuong La,et al.  The Situation Awareness Weighted Network (SAWN) Model , 2014 .

[33]  Mica R. Endsley,et al.  Measurement of Situation Awareness in Dynamic Systems , 1995, Hum. Factors.

[34]  H. J. Arnold.  Introduction to the Practice of Statistics , 1990 .

[35]  Patricia J. Rogers,et al.  Purposeful Program Theory: Effective Use of Theories of Change and Logic Models , 2011 .

[36]  U. Neisser.  Cognitive Psychology. (Book Reviews: Cognition and Reality. Principles and Implications of Cognitive Psychology) , 1976 .

[37]  Neville A Stanton,et al.  From ethnography to the EAST method: A tractable approach for representing distributed cognition in Air Traffic Control , 2010, Ergonomics.

[38]  B. P. Welford,et al.  The Science of Decision Making. , 1969 .

[39]  Nimon,et al.  Intelligence Collection: How to Plan and Execute Intelligence Collection in Complex Environments. By Wayne Michael Hall and Gary Citrenbaum. Santa Barbara, CA, ABC-CLIO, LLC. 2012. , 2013 .

[40]  Mica R. Endsley,et al.  Theoretical Underpinnings of Situation Awareness, A Critical Review , 2000 .

[41]  Simon French,et al.  Cynefin, statistics and decision analysis , 2013, J. Oper. Res. Soc..

[42]  D. Deptula.  Effects-Based Operations , 2006 .

[43]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[44]  N. Smirnov.  Table for Estimating the Goodness of Fit of Empirical Distributions , 1948 .

[45]  P. J. Dortmans,et al.  Employing integrated reference models to represent interdependency in a complex enterprise , 2013, J. Oper. Res. Soc..