Reasoning difficulty in analytical activity

We review the consensus of expert opinion concerning the psychology of intelligence analysis, as a form of critical thinking. This consensus details a number of ways in which the cognitive work is difficult. Many senior analysts have commented upon the requirements of intelligence analysis – the reasoning traps to which novices fall victim, and the required knowledge and skills of experts. There remain gaps in our understanding, not just because the research is classified. There simply has not been that much systematic research. If the empirical base were broadened, headway might be made in training and techniques to help analysts cope with difficulty. We hope that this article contributes by presenting an overview and rationale for empirical study of the cognitive ergonomics of intelligence analysis.

[1]  M. Grant.  Teaching Students to Think Like Scientists , 2013 .

[2]  Emily S. Patterson,et al.  Macrocognition Metrics and Scenarios: Design and Evaluation for Real-World Teams , 2012 .

[3]  Rob Johnston.  Analytic Culture in the U.S. Intelligence Community: An Ethnographic Study , 2012 .

[4]  David T. Moore,et al.  Sensemaking: A Structure for an Intelligence Revolution , 2011 .

[5]  Robert R. Hoffman,et al.  Skills in Applied Concept Mapping , 2011 .

[6]  Ann Blandford,et al.  Discovery-led refinement in e-discovery investigations: sensemaking, cognitive ergonomics and system design , 2010, Artificial Intelligence and Law.

[7]  Chaoming Song,et al.  Limits of Predictability in Human Mobility , 2010, Science.

[8]  Robert R. Hoffman,et al.  Perspectives on Cognitive Task Analysis: Historical Origins and Modern Communities of Practice , 2008 .

[9]  Jordan Litman.  Interest and deprivation factors of epistemic curiosity , 2008 .

[10]  Paul E. Lehner,et al.  Confirmation Bias in Complex Analyses , 2008, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[11]  Jennifer K. Phillips,et al.  A Data–Frame Theory of Sensemaking , 2007 .

[12]  Susan G. Hutchins,et al.  What Makes Intelligence Analysis Difficult?: A Cognitive Task Analysis , 2007 .

[13]  D. T. Moore.  Critical Thinking and Intelligence Analysis , 2007 .

[14]  Gary Klein,et al.  Making Sense of Sensemaking 2: A Macrocognitive Model , 2006, IEEE Intelligent Systems.

[15]  Henry L. Roediger,et al.  Critical thinking in psychology , 2006 .

[16]  Gary Klein,et al.  Making Sense of Sensemaking 1: Alternative Perspectives , 2006, IEEE Intelligent Systems.

[17]  Paul J Silvia,et al.  The Latent Structure of Trait Curiosity: Evidence for Interest and Deprivation Curiosity Dimensions , 2006, Journal of personality assessment.

[18]  Stephen Marrin,et al.  Improving Intelligence Analysis by Looking to the Medical Profession , 2005 .

[19]  Jordan Litman.  Curiosity and the pleasures of learning: Wanting and liking new information , 2005 .

[20]  William C. Elm,et al.  Finding Decision Support Requirements for Effective Intelligence Analysis Tools , 2005 .

[21]  Jordan A. Litman,et al.  Epistemic curiosity, feeling-of-knowing, and exploratory behaviour , 2005 .

[22]  Philip Koopman,et al.  Work-arounds, Make-work, and Kludges , 2003, IEEE Intell. Syst..

[23]  Lisa Krizan,et al.  Bringing Intelligence About: Practitioners Reflect on Best Practices , 2003 .

[24]  Paul J. Feltovich,et al.  Learners' (mis)understanding of important and difficult concepts: a challenge to smart machines in education , 2001 .

[25]  Emily S. Patterson,et al.  Voice Loops as Coordination Aids in Space Shuttle Mission Control , 1999, Computer Supported Cooperative Work (CSCW).

[26]  D. Halpern.  Thought and Knowledge: An Introduction to Critical Thinking , 1995 .

[27]  J. Shanteau.  Competence in experts: The role of task characteristics , 1992 .

[28]  A. Tversky,et al.  Judgment Under Uncertainty: Heuristics and Biases. , 1984 .

[29]  Robert V. Katter,et al.  Human Processes in Intelligence Analysis: Phase I Overview , 1979 .

[30]  Daniel Kahneman,et al.  Availability: A heuristic for judging frequency and probability , 1973 .

[31]  Celeste N. Godfrey,et al.  of the Human Factors and Ergonomics Society , 2010 .

[32]  Gary Klein,et al.  Antipatterns in the Creation of Intelligent Systems , 2007, IEEE Intelligent Systems.

[33]  Robert R. Hoffman,et al.  HCC Implications for the Procurement Process , 2006, IEEE Intelligent Systems.

[34]  F. Mihelic.  Generalist Function in Intelligence Analysis , 2005 .

[35]  Robert R. Hoffman,et al.  How Might "Transformational" Technologies and Concepts be Barriers to Sensemaking in Intelligence Analysis? , 2005 .

[36]  Herbert S. Lin,et al.  A Review of the FBI ’ s Trilogy Information Technology Modernization Program , 2004 .

[37]  Olaf R. P. Bininda-Emonds,et al.  Garbage in, Garbage out , 2004 .

[38]  J. Austin,et al.  About the authors , 2004, Artificial Intelligence Review.

[39]  Gary H. Mills.  The role of rhetorical theory in military intelligence analysis: A soldier's guide to rhetorical theory , 2003 .

[40]  Richards J. Heuer,et al.  Psychology of Intelligence Analysis , 1999 .

[41]  P. Todd,et al.  Simple Heuristics That Make Us Smart , 1999 .

[42]  Peter J. Williamson,et al.  Strategy as Options on the Future , 1999 .

[43]  Philip A. Schrodt,et al.  The Impact of Early Warning on Institutional Responses to Complex Humanitarian Crises , 1998 .

[44]  Jared Freeman,et al.  Critical thinking skills in tactical decision making: A model and a training strategy. , 1998 .

[45]  R. Flin.  Decision making under stress : emerging themes and applications , 1997 .

[46]  Beverly T. Watkins.  Teaching Students to Think Like Scientists. , 1992 .

[47]  J. R. Thompson,et al.  The Cognitive Bases of Intelligence Analysis. , 1984 .

[48]  J. Mill.  A System of Logic , 1843 .

[49]  J. Bradshaw,et al.  Institute for Human and Machine Cognition , 2022 .