Measuring Resilience

Objective: As human factors and ergonomics (HF/E) moves to embrace a greater systems perspective concerning human–machine technologies, new and emergent properties, such as resilience, have arisen. Our objective here is to promote discussion as to how to measure this latter, complex phenomenon. Background: Resilience is now a much-referenced goal for technology and work system design. It subsumes the new movement of resilience engineering. As part of a broader systems approach to HF/E, this concept requires both a definitive specification and an associated measurement methodology. Such an effort epitomizes our present work. Method: Using rational analytic and synthetic methods, we offer an approach to the measurement of resilience capacity. Results: We explicate how our proposed approach can be employed to compare resilience across multiple systems and domains, and emphasize avenues for its future development and validation. Conclusion: Emerging concerns for the promise and potential of resilience and associated concepts, such as adaptability, are highlighted. Arguments skeptical of these emerging dimensions must be met with quantitative answers; we advance one approach here. Application: Robust and validated measures of resilience will enable coherent and rational discussions of complex emergent properties in macrocognitive system science.

[1]  Kevin C. Stagl,et al.  Understanding team adaptation: a conceptual analysis and model. , 2006, The Journal of applied psychology.

[2]  Robert R. Hoffman,et al.  "Complex Sociotechnical Joint Cognitive Work Systems"? , 2009, IEEE Intelligent Systems.

[3]  Stephen M. Stigler,et al.  Fisher and the 5% Level , 2008 .

[4]  Darren D. Medlin Information Age Transformation: Getting to a 21st Century Military , 2008 .

[5]  Robert R. Hoffman,et al.  Toward a Theory of Complex and Cognitive Systems , 2005, IEEE Intell. Syst..

[6]  Gary Klein,et al.  Flexecution as a Paradigm for Replanning, Part 1 , 2007, IEEE Intelligent Systems.

[7]  Tammy L. Rapp,et al.  Team Effectiveness 1997-2007: A Review of Recent Advancements and a Glimpse Into the Future , 2008 .

[8]  Kevin A. Gluck,et al.  Robustness in a variable environment , 2012 .

[9]  Nancy J. Cooke,et al.  Advances in Measuring Team Cognition , 2003 .

[10]  Suniya S Luthar,et al.  The Multidimensional Nature of Resilience to Spousal Loss , 2017, Journal of personality and social psychology.

[11]  Victor Smetacek,et al.  Making sense , 2004, Nature.

[12]  Michael J. McCloskey,et al.  Envisioning Desirements , 2013, IEEE Intelligent Systems.

[13]  David Woods,et al.  On the Origins of Cognitive Systems Engineering , 2017 .

[14]  David D. Woods John Wreathall STRESS-STRAIN PLOTS AS A BASIS FOR ASSESSING SYSTEM RESILIENCE , 2016 .

[15]  Patricia L. McDermott,et al.  Measurement for evaluating the learnability and resilience of methods of cognitive work , 2010 .

[16]  Christopher Nemeth,et al.  Remaining sensitive to the possibility of failure , 2008 .

[17]  Jeffrey M. Bradshaw,et al.  Metrics, Metrics, Metrics, Part 2: Universal Metrics? , 2010, IEEE Intelligent Systems.

[18]  G. Zajicek,et al.  The Wisdom of the Body , 1934, Nature.

[19]  Leonard R. Sussman,et al.  Nominal, Ordinal, Interval, and Ratio Typologies are Misleading , 1993 .

[20]  Barrett S. Caldwell,et al.  Cognitive Challenges to Resilience Dynamics in Managing Large-Scale Event Response , 2014 .

[21]  Erik Hollnagel,et al.  Joint Cognitive Systems: Patterns in Cognitive Systems Engineering , 2006 .

[22]  David D Woods,et al.  Steering the Reverberations of Technology Change on Fields of Practice: Laws that Govern Cognitive Work , 2019, Proceedings of the Twenty-Fourth Annual Conference of the Cognitive Science Society.

[23]  P A Hancock,et al.  Influence of task demand characteristics on workload and performance. , 1995, The International journal of aviation psychology.

[24]  P. Hancock,et al.  Adaptive control in human-machine systems , 1987 .

[25]  Robert R. Hoffman Some Challenges for Macrocognitive Measurement , 2018 .

[26]  Jeffrey M. Bradshaw,et al.  Seven Cardinal Virtues of Human-Machine Teamwork: Examples from the DARPA Robotic Challenge , 2014, IEEE Intelligent Systems.

[27]  Richard E. Hayes,et al.  Power to the Edge: Command, Control in the Information Age , 2003 .

[28]  Gary Klein,et al.  Making Sense of Sensemaking 2: A Macrocognitive Model , 2006, IEEE Intelligent Systems.

[29]  D. Woods Coping with complexity: the psychology of human behaviour in complex systems , 1988 .

[30]  Jeffrey R. Stevens,et al.  Evolution and the mechanisms of decision making , 2012 .

[31]  David Woods,et al.  Basic Patterns in How Adaptive Systems Fail , 2017 .

[32]  T. Belfield,et al.  Words matter! , 2010, Journal of Family Planning and Reproductive Health Care.

[33]  Gary Klein Seeing What Others Don't: The Remarkable Ways We Gain Insights , 2013 .

[34]  P A Hancock,et al.  Automation: how much is too much? , 2014, Ergonomics.

[35]  Paul J. Feltovich,et al.  Accelerated Expertise: Training for High Proficiency in a Complex World , 2013 .

[36]  P. Hancock,et al.  A Dynamic Model of Stress and Sustained Attention , 1989, Human factors.

[37]  Karol G. Ross,et al.  Macrocognition , 2003, IEEE Intell. Syst..

[38]  Jeffrey M. Bradshaw,et al.  Ten Challenges for Making Automation a "Team Player" in Joint Human-Agent Activity , 2004, IEEE Intell. Syst..

[39]  Laura G. Militello,et al.  Macrocognition, Mental Models, and Cognitive Task Analysis Methodology , 2017 .

[40]  Gary Klein,et al.  Naturalistic investigations and models of reasoning about complex indeterminate causation , 2011, Inf. Knowl. Syst. Manag..

[41]  Huib de Ridder,et al.  Hysteresis in Mental Workload and Task Performance , 2016, Hum. Factors.

[42]  Peter A. Hancock,et al.  Hedonomics: The Power of Positive and Pleasurable Ergonomics , 2005 .

[43]  J. Gregory Trafton,et al.  Minding the Weather: How Expert Forecasters Think , 2017 .

[44]  Peter A. Hancock,et al.  The Effect of Prior Task Loading on Mental Workload , 2011, Hum. Factors.

[45]  Robert R. Hoffman,et al.  Theory → Concepts → Measures but Policies → Metrics , 2018, Macrocognition Metrics and Scenarios.

[46]  Gary Klein,et al.  Making Sense of Sensemaking 1: Alternative Perspectives , 2006, IEEE Intelligent Systems.

[47]  Peter A. Hancock Performance on the Very Edge , 2009 .

[48]  Ann Blandford,et al.  A resilience markers framework for small teams , 2011, Reliab. Eng. Syst. Saf..

[49]  Gary Klein,et al.  Working Minds: A Practitioner's Guide to Cognitive Task Analysis , 2006 .

[50]  Mark H. Chignell,et al.  Mental workload dynamics in adaptive interface design , 1988, IEEE Trans. Syst. Man Cybern..

[51]  Merriam Webster Merriam-Webster's Collegiate Dictionary , 2016 .

[52]  Ronald F. Piccolo,et al.  A META‐ANALYSIS OF TEAMWORK PROCESSES: TESTS OF A MULTIDIMENSIONAL MODEL AND RELATIONSHIPS WITH TEAM EFFECTIVENESS CRITERIA , 2008 .

[53]  Robert R. Hoffman,et al.  Beyond Simon's Slice: Five Fundamental Trade-Offs that Bound the Performance of Macrocognitive Work Systems , 2011, IEEE Intelligent Systems.

[54]  P A Hancock,et al.  Imposing limits on autonomous systems , 2017, Ergonomics.

[55]  Leslie A. DeChurch,et al.  The cognitive underpinnings of effective teamwork: a meta-analysis. , 2010, The Journal of applied psychology.

[56]  Lia DiBello,et al.  Intuitive Expertise and Empowerment: The Long-Term Impact of Simulation Training on Changing Accountabilities in a Biotech Firm , 2009 .

[57]  Kevin A. Gluck,et al.  A quantification of robustness , 2013 .

[58]  Gary Klein,et al.  The Power of Intuition: How to Use Your Gut Feelings to Make Better Decisions at Work , 2004 .

[59]  David D. Woods,et al.  Four concepts for resilience and the implications for the future of resilience engineering , 2015, Reliab. Eng. Syst. Saf..

[60]  E. Hollnagel The Etto Principle: Efficiency-Thoroughness Trade-Off: Why Things That Go Right Sometimes Go Wrong , 2009 .

[61]  E. Pulakos,et al.  Adaptability in the workplace: development of a taxonomy of adaptive performance. , 2000, The Journal of applied psychology.

[62]  Emily S. Patterson,et al.  Macrocognition Metrics and Scenarios: Design and Evaluation for Real-World Teams , 2012 .