A Public-Private-Partnership Model for National Cyber Situational Awareness

Current data mining tools are characterized by a plethora of algorithms but a lack of guidelines to select the right method according to the nature of the problem under analysis. Producing such guidelines is a primary goal by the field of meta-learning; the research objective is to understand the interaction between the mechanism of learning and the concrete contexts in which that mechanism is applicable. The field of meta-learning has seen continuous growth in the past years with interesting new developments in the construction of practical model-selection assistants, task-adaptive learners, and a solid conceptual framework. In this paper, we give an overview of different techniques necessary to build meta-learning systems. We begin by describing an idealized meta-learning architecture comprising a variety of relevant component techniques. We then look at how each technique has been studied and implemented by previous research. In addition, we show how metalearning has already been identified as an important component in real-world applications.

[1]  Carlos Soares,et al.  Ranking Learning Algorithms: Using IBL and Meta-Learning on Accuracy and Time Results , 2003, Machine Learning.

[2]  Lorien Y. Pratt,et al.  A Survey of Connectionist Network Reuse Through Transfer , 1998, Learning to Learn.

[3]  Ricardo Vilalta,et al.  Introduction to the Special Issue on Meta-Learning , 2004, Machine Learning.

[4]  Stefan Arnborg,et al.  Information awareness in command and control: precision, quality, utility , 2000, Proceedings of the Third International Conference on Information Fusion.

[5]  Salvatore J. Stolfo,et al.  On the Accuracy of Meta-learning for Scalable Data Mining , 2004, Journal of Intelligent Information Systems.

[6]  Christophe Giraud-Carrier.  Beyond predictive accuracy : what? , 1998 .

[7]  Paul E. Utgoff,et al.  Shift of bias for inductive concept learning , 1984 .

[8]  David J. Spiegelhalter,et al.  Machine Learning, Neural and Statistical Classification , 2009 .

[9]  Donald Perlis,et al.  Explicitly biased generalization , 1989, Comput. Intell..

[10]  Eric A. M. Luiijf,et al.  Nineteen national cyber security strategies , 2013, Int. J. Crit. Infrastructures.

[11]  Stephanie S. Barrett,et al.  Inter-Organization Information Sharing Systems , 1982, MIS Q..

[12]  Robert Engels,et al.  Using a Data Metric for Preprocessing Advice for Data Mining Applications , 1998, ECAI.

[13]  Rich Caruana,et al.  Multitask Learning , 1997, Machine Learning.

[14]  M. Pazzani,et al.  Error Reduction through Learning Multiple Descriptions , 1996, Machine Learning.

[15]  T. Lewis.  Critical Infrastructure Protection in Homeland Security: Defending a Networked Nation , 2006 .

[16]  So Young Sohn,et al.  Meta Analysis of Classification Algorithms for Pattern Recognition , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Hilan Bensusan,et al.  Discovering Task Neighbourhoods Through Landmark Learning Performances , 2000, PKDD.

[18]  Abraham Bernstein,et al.  An Intelligent Assistant for the Knowledge Discovery Process , 2001, IJCAI 2001.

[19]  David B. Skalak,et al.  Prototype Selection for Composite Nearest Neighbor Classifiers , 1995 .

[20]  Peter A. Flach,et al.  Decision tree-based characterization for meta-learning , 2002 .

[21]  David W. Aha,et al.  Generalizing from Case studies: A Case Study , 1992, ML.

[22]  Xiaohua Yan,et al.  A Early Detection of Cyber Security Threats using Structured Behavior Modeling , 2013 .

[23]  Lukasz Golab,et al.  Issues in data stream management , 2003, SGMD.

[24]  João Gama,et al.  Cascade Generalization , 2000, Machine Learning.

[25]  Mica R. Endsley,et al.  Theoretical Underpinnings of Situation Awareness, A Critical Review , 2000 .

[26]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[27]  J. Costa,et al.  A WEIGHTED RANK MEASURE OF CORRELATION , 2005 .

[28]  Sebastian Thrun,et al.  Lifelong Learning Algorithms , 1998, Learning to Learn.

[29]  J. Friedman.  Special Invited Paper-Additive logistic regression: A statistical view of boosting , 2000 .

[30]  Melanie Hilario,et al.  Fusion of Meta-knowledge and Meta-data for Case-Based Model Selection , 2001, PKDD.

[31]  Gregory A. Witte,et al.  Framework for Improving Critical Infrastructure Cybersecurity | NIST , 2014 .

[32]  Mica R. Endsley,et al.  Toward a Theory of Situation Awareness in Dynamic Systems , 1995, Hum. Factors.

[33]  Dino Pedreschi,et al.  Machine Learning: ECML 2004 , 2004, Lecture Notes in Computer Science.

[34]  Carlos Soares,et al.  Sampling-Based Relative Landmarks: Systematically Test-Driving Algorithms Before Choosing , 2001, EPIA.

[35]  George P. Tadda,et al.  Overview of Cyber Situation Awareness , 2010, Cyber Situational Awareness.

[36]  João Gama,et al.  Characterization of Classification Algorithms , 1995, EPIA.

[37]  Ricardo Vilalta,et al.  A Perspective View and Survey of Meta-Learning , 2002, Artificial Intelligence Review.

[38]  Silvana Trimi,et al.  Big-data applications in the government sector , 2014, Commun. ACM.

[39]  Jürgen Schmidhuber.  Discovering Solutions with Low Kolmogorov Complexity and High Generalization Capability , 1995, ICML.

[40]  Takahira Yamaguchi,et al.  Constructing Inductive Applications by Meta-Learning with Method Repositories , 2002, Progress in Discovery Science.

[41]  Saso Dzeroski,et al.  Combining Classifiers with Meta Decision Trees , 2003, Machine Learning.

[42]  Ian H. Witten,et al.  Stacked generalization: when does it work? , 1997, IJCAI 1997.

[43]  Ian S. Paterson.  New Models for Data Envelopment Analysis. Measuring Efficiency Outwith the VRS Frontier , 2000 .

[44]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[45]  Carla E. Brodley,et al.  Recursive automatic bias selection for classifier construction , 1995, Machine Learning.

[46]  Ulrik Franke,et al.  Cyber situational awareness - A systematic review of the literature , 2014, Comput. Secur..

[47]  Hilan Bensusan,et al.  A Higher-order Approach to Meta-learning , 2000, ILP Work-in-progress reports.

[48]  L. Breiman.  Stacked Regressions , 1996, Machine Learning.

[49]  Alexander Schnabl,et al.  Development of Multi-Criteria Metrics for Evaluation of Data Mining Algorithms , 1997, KDD.

[50]  Gerhard Widmer,et al.  Tracking Context Changes through Meta-Learning , 1997, Machine Learning.

[51]  Hilan Bensusan,et al.  Meta-Learning by Landmarking Various Learning Algorithms , 2000, ICML.

[52]  Marie desJardins,et al.  Evaluation and selection of biases in machine learning , 1995, Machine Learning.

[53]  Katharina Morik,et al.  The MiningMart Approach to Knowledge Discovery in Databases , 2004 .

[54]  Florian Skopik,et al.  Designing a Cyber Attack Information System for National Situational Awareness , 2012, Future Security.

[55]  Bernard Zenko,et al.  Is Combining Classifiers Better than Selecting the Best One , 2002, ICML.

[56]  Pavel Brazdil,et al.  Improving Progressive Sampling via Meta-learning on Learning Curves , 2004, ECML.

[57]  Christopher J. Merz,et al.  Dynamical Selection of Learning Algorithms , 1995, AISTATS.

[58]  Jonathan Baxter,et al.  Theoretical Models of Learning to Learn , 1998, Learning to Learn.

[59]  James Moffat.  Mathematical modelling of information age conflict , 2006, Adv. Decis. Sci..

[60]  P. Andersen,et al.  A procedure for ranking efficient units in data envelopment analysis , 1993 .

[61]  Johannes Fürnkranz,et al.  An Evaluation of Landmarking Variants , 2001 .