Performance Evaluation for Process Refinement Stage of SWA System

In periodic manner the analysts teams are in the process of designing, updating and verifying the situational awareness SWA system. Initially, at the designing stage the risk assessment model has little information about the dynamic environment. Hence, any missing information can directly impact the situational assessment capabilities. With this in mind, researchers relied on various performance metrics in order to verify how well they were doing in assessing different situations. In fact, before measuring the ranking capabilities of the SWA system, the underlying performance metrics should be examined against its intended purpose. In this paper, we have conducted quality based evaluations for the performance metrics, namely "The Ranking Capability Score". The results obtained showed that the proposed performance metrics have scaled well over a number of scenarios. Indeed, from the data fusion perspectives the underlying metrics have adequately satisfied different SWA system needs and configurations.

[1]  John J. Salerno,et al.  Measuring the worthiness of situation assessment , 2011, Proceedings of the 2011 IEEE National Aerospace and Electronics Conference (NAECON).

[2]  George Rebovich.  Systems thinking for the enterprise: new and emerging perspectives , 2006, 2006 IEEE/SMC International Conference on System of Systems Engineering.

[3]  Alexander Karlsson,et al.  Dependable and generic high-level information fusion methods and algorithms for uncertainty management , 2007 .

[4]  Moises Sudit,et al.  Issues and challenges in higher level fusion: Threat/impact assessment and intent modeling (a panel summary) , 2010, 2010 13th International Conference on Information Fusion.

[5]  Shambhu J. Upadhyaya,et al.  An alert fusion framework for situation awareness of coordinated multistage attacks , 2005, Third IEEE International Workshop on Information Assurance (IWIA'05).

[6]  John J. Salerno.  Where's level 2/3 fusion - a look back over the past 10 years , 2007, 2007 10th International Conference on Information Fusion.

[7]  Moises Sudit,et al.  TANDI: threat assessment of network data and information , 2006, SPIE Defense + Commercial Sensing.

[8]  Alan N. Steinberg,et al.  Rethinking the JDL Data Fusion Levels , 2005 .

[9]  Shanchieh Jay Yang,et al.  FuSIA: Future Situation and Impact Awareness , 2008, 2008 11th International Conference on Information Fusion.

[10]  Edgar Toshiro Yano,et al.  A Cyber Security Situational Awareness Framework to Track and Project Multistage Cyber Attacks , 2014 .

[11]  G. Klein,et al.  A recognition-primed decision (RPD) model of rapid decision making. , 1993 .

[12]  Jinqiao Shi,et al.  A Multi-Level Analysis Framework in Network Security Situation Awareness , 2013, ITQM.

[13]  Erik Blasch,et al.  Revisiting the JDL model for information exploitation , 2013, Proceedings of the 16th International Conference on Information Fusion.

[14]  Sushil Jajodia,et al.  Advanced Cyber Attack Modeling Analysis and Visualization , 2010 .

[15]  Mica R. Endsley,et al.  Theoretical Underpinnings of Situation Awareness, A Critical Review , 2000 .

[16]  Gee Wah Ng,et al.  High-level Information Fusion: An Overview , 2013, J. Adv. Inf. Fusion.

[17]  Karen Walzer.  Temporal complex event processing with rule based systems using the rete algorithm , 2009 .

[18]  Moises Sudit,et al.  Enhancing situation awareness via automated situation assessment , 2010, IEEE Communications Magazine.

[19]  Liam M. Mayron,et al.  A perceptually-relevant model-based cyber threat prediction method for enterprise mission assurance , 2011, 2011 IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support (CogSIMA).

[20]  Adam Stotz,et al.  High Level Fusion in the Cyber Domain , 2005 .

[21]  Shanchieh Jay Yang,et al.  Elements of impact assessment: a case study with cyber attacks , 2009, Defense + Commercial Sensing.

[22]  Ronald R. Yager,et al.  INFERD and Entropy for Situational Awareness , 2007, J. Adv. Inf. Fusion.

[23]  Pierre Valin,et al.  Using the C-OODA model for CIMIC analysis , 2011, Proceedings of the 2011 IEEE National Aerospace and Electronics Conference (NAECON).

[24]  Adam Stotz,et al.  Situational awareness of a coordinated cyber attack , 2005, SPIE Defense + Commercial Sensing.

[25]  Dale A. Lambert,et al.  STDF model based maritime situation assessments , 2007, 2007 10th International Conference on Information Fusion.

[26]  Adam Stotz,et al.  High level information fusion for tracking and projection of multistage cyber attacks , 2009, Inf. Fusion.

[27]  Anders Dahlbom.  Petri nets for situation recognition , 2011 .

[28]  Risto Vaarandi,et al.  SEC - a lightweight event correlation tool , 2002, IEEE Workshop on IP Operations and Management.

[29]  E. Bosse,et al.  Framework for the Analysis of Information Relevance (FAIR) , 2012, 2012 IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support.

[30]  Adam Stotz,et al.  INformation fusion engine for real-time decision-making (INFERD): A perceptual system for cyber attack tracking , 2007, 2007 10th International Conference on Information Fusion.

[31]  Stuart C. Shapiro,et al.  Systemic test and evaluation of a hard+soft information fusion framework: Challenges and current approaches , 2014, 17th International Conference on Information Fusion (FUSION).

[32]  Mica R. Endsley,et al.  Toward a Theory of Situation Awareness in Dynamic Systems , 1995, Hum. Factors.

[33]  T. Pica,et al.  "Information Gap" Tasks: Do They Facilitate Second Language Acquisition? , 1986 .

[34]  John P. Rouillard.  Real-time Log File Analysis Using the Simple Event Correlator (SEC) , 2004, LISA.

[35]  Alan N. Steinberg,et al.  Revisions to the JDL data fusion model , 1999, Defense, Security, and Sensing.

[36]  Joseph G. Wohl,et al.  Force Management Decision Requirements for Air Force Tactical Command and Control , 1981, IEEE Transactions on Systems, Man, and Cybernetics.

[37]  Jan H. P. Eloff,et al.  Simulating adversarial interactions between intruders and system administrators using OODA-RR , 2007, SAICSIT '07.

[38]  Doug Maughan.  National cyber security research assessment and roadmap , 2009, CSIIRW '09.

[39]  Daniel A. Keim,et al.  NStreamAware: real-time visual analytics for data streams to enhance situational awareness , 2014, VizSec '14.

[40]  Shanchieh Jay Yang,et al.  Projecting Cyberattacks Through Variable-Length Markov Models , 2008, IEEE Transactions on Information Forensics and Security.

[41]  George P. Tadda,et al.  Overview of Cyber Situation Awareness , 2010, Cyber Situational Awareness.

[42]  S. Upadhyaya,et al.  Real-time multistage attack awareness through enhanced intrusion alert clustering , 2005, MILCOM 2005 - 2005 IEEE Military Communications Conference.

[43]  Shanchieh Jay Yang,et al.  Toward Ensemble Characterization and Projection of Multistage Cyber Attacks , 2010, 2010 Proceedings of 19th International Conference on Computer Communications and Networks.

[44]  James Llinas,et al.  Handbook of Multisensor Data Fusion : Theory and Practice, Second Edition , 2008 .

[45]  Gregory Epiphaniou,et al.  The Dark Web: Cyber-Security Intelligence Gathering Opportunities, Risks and Rewards , 2014, CIT 2014.

[46]  Eric Michael Hutchins,et al.  Intelligence-Driven Computer Network Defense Informed by Analysis of Adversary Campaigns and Intrusion Kill Chains , 2010 .

[47]  S. Plano,et al.  Proactive decision fusion for site security , 2005, 2005 7th International Conference on Information Fusion.

[48]  Zoheir A. Sabeur,et al.  Agile Data Fusion and Knowledge Base Architecture for Critical Decision Support , 2013, Int. J. Decis. Support Syst. Technol..

[49]  Moises Sudit,et al.  Impact Assessment , 2014, Cyber Defense and Situational Awareness.