A Framework for Analyst Focus from Computed Significance

Attention is the critical resource for intelligence analysts, so tools that provide focus are useful. One way to determine focus is by computing significance. In the context of a known model, new data can be placed on a spectrum defined by: normal, anomalous, interesting, novel, or random; and significance is greatest towards the middle of this spectrum. However, significance also depends on the mental state of the analyst (and the organization). A framework for understanding significance is defined, and its impact on the knowledge–discovery process is explored.

[1]  Kurt Bryan,et al.  The $25,000,000,000 Eigenvector: The Linear Algebra behind Google , 2006, SIAM Rev..

[2]  Simeon J. Simoff,et al.  Digging in the Details: A Case Study in Network Data Mining , 2005, ISI.

[3]  Jürgen Schmidhuber,et al.  Simple Algorithmic Principles of Discovery, Subjective Beauty, Selective Attention, Curiosity & Creativity , 2007, Discovery Science.

[4]  Jon Kleinberg,et al.  The link prediction problem for social networks , 2003, CIKM '03.

[5]  Gregory F. Treverton.  Reshaping National Intelligence for an Age of Information: A reshaped intelligence , 2001 .

[6]  Jürgen Schmidhuber,et al.  Optimal Artificial Curiosity, Creativity, Music, and the Fine Arts , 2005 .

[7]  David B. Skillicorn,et al.  Detecting Anomalies in Graphs , 2007, 2007 IEEE Intelligence and Security Informatics.

[8]  Owen Conlan,et al.  The Challenge of Content Creation to Facilitate Personalized E-Learning Experiences , 2006 .

[9]  David M. J. Tax,et al.  One-class classification , 2001 .

[10]  Ulrike von Luxburg,et al.  A tutorial on spectral clustering , 2007, Stat. Comput..

[11]  David B. Skillicorn,et al.  Hardening adversarial prediction with anomaly tracking , 2009, 2009 IEEE International Conference on Intelligence and Security Informatics.

[12]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[13]  J. Leydold,et al.  Discrete Nodal Domain Theorems , 2000, math/0009120.

[14]  Rajeev Motwani,et al.  The PageRank Citation Ranking : Bringing Order to the Web , 1999, WWW 1999.

[15]  W. Greub.  Linear Algebra , 1981 .

[16]  Xiaojin Zhu,et al.  Introduction to Semi-Supervised Learning , 2009, Synthesis Lectures on Artificial Intelligence and Machine Learning.

[17]  David B. Skillicorn,et al.  Classification Using Streaming Random Forests , 2011, IEEE Transactions on Knowledge and Data Engineering.

[18]  David B. Skillicorn,et al.  Understanding Complex Datasets: Data Mining with Matrix Decompositions , 2007 .

[19]  Simeon J. Simoff,et al.  Network Data Mining: Discovering Patterns of Interaction Between Attributes , 2006, PAKDD.

[20]  J. Bouchaud,et al.  RANDOM MATRIX THEORY AND FINANCIAL CORRELATIONS , 2000 .