A multi-disciplinary review of knowledge acquisition methods: From human to autonomous eliciting agents

This paper offers a multi-disciplinary review of knowledge acquisition methods in human activity systems. The review captures the degree of involvement of various types of agencies in the knowledge acquisition process, and proposes a classification with three categories of methods: the human agent, the human-inspired agent, and the autonomous machine agent methods. In the first two categories, the acquisition of knowledge is seen as a cognitive task analysis exercise, while in the third category knowledge acquisition is treated as an autonomous knowledge-discovery endeavour. The motivation for this classification stems from the continuous change over time of the structure, meaning and purpose of human activity systems, which are seen as the factor that fuelled researchers' and practitioners' efforts in knowledge acquisition for more than a century.We show through this review that the KA field is increasingly active due to the higher and higher pace of change in human activity, and conclude by discussing the emergence of a fourth category of knowledge acquisition methods, which are based on red-teaming and co-evolution.

[1]  Ralph L. Keeney,et al.  Foundations for Group Decision Analysis , 2013, Decis. Anal..

[2]  Gary Klein Using Cognitive Task Analysis to Build a Cognitive Model , 2000 .

[3]  Nigel Shadbolt,et al.  Use of the Critical Decision Method to Elicit Expert Knowledge: A Case Study in the Methodology of Cognitive Task Analysis , 1998, Hum. Factors.

[4]  Joseph Y. Halpern,et al.  Coevolution of learning and data-acquisition mechanisms: a model for cognitive evolution , 2012, Philosophical Transactions of the Royal Society B: Biological Sciences.

[5]  Hussein A. Abbass,et al.  The Art of Red Teaming , 2015 .

[6]  Ilona Jagielska,et al.  An investigation into the application of neural networks, fuzzy logic, genetic algorithms, and rough sets to automated knowledge acquisition for classification problems , 1999, Neurocomputing.

[7]  E. Salas,et al.  Human Factors : The Journal of the Human Factors and Ergonomics Society , 2012 .

[8]  Susan Palmiter,et al.  Designing Help Using a GOMS Model: An Information Retrieval Evaluation , 1991 .

[9]  Anders Jansson,et al.  Collegial verbalisation – the value of an independent observer: an ecological approach , 2015 .

[10]  Richard E. Clark,et al.  Cognitive Task Analysis for Expert-Based Instruction in Healthcare , 2014 .

[11]  Emilie M. Roth,et al.  Uncovering the Requirements of Cognitive Work , 2008, Hum. Factors.

[12]  Francisco Herrera,et al.  Using evolutionary algorithms as instance selection for data reduction in KDD: an experimental study , 2003, IEEE Trans. Evol. Comput..

[13]  Robert R. Hoffman,et al.  Doing Psychology in an AI Context: A Personal Perspective and Introduction to This Volume , 1992 .

[14]  Linda J. Viswat,et al.  RATING VERSUS RANKING : WHAT IS THE BEST WAY TO REDUCE RESPONSE AND LANGUAGE BIAS IN CROSS-NATIONAL RESEARCH ? , 2009 .

[15]  Ana L. C. Bazzan,et al.  A Multi-agent System to Facilitate Knowledge Discovery : an application to Bioinformatics , 2002 .

[16]  Mohammed Ouassaid,et al.  An implementation of FIPA contract net interaction protocol adapted for smart home agents simulation , 2015, 2015 3rd International Renewable and Sustainable Energy Conference (IRSEC).

[17]  Hisao Ishibuchi,et al.  Comparison between Fuzzy and Interval Partitions in Evolutionary Multiobjective Design of Rule-Based Classification Systems , 2005, The 14th IEEE International Conference on Fuzzy Systems, 2005. FUZZ '05..

[18]  Ian M. Neale,et al.  First generation expert systems: a review of knowledge acquisition methodologies , 1988, The Knowledge Engineering Review.

[19]  Wayne D. Gray Cognitive Architectures: Choreographing the Dance of Mental Operations With the Task Environment , 2008, Hum. Factors.

[20]  Gail A. Carpenter,et al.  Self-organizing information fusion and hierarchical knowledge discovery: a new framework using ARTMAP neural networks , 2005, Neural Networks.

[21]  Stephen R. Marsland,et al.  Machine Learning - An Algorithmic Perspective , 2009, Chapman and Hall / CRC machine learning and pattern recognition series.

[22]  Li Chen,et al.  Critiquing-based recommenders: survey and emerging trends , 2012, User Modeling and User-Adapted Interaction.

[23]  P. Gazarian,et al.  Use of the Critical Decision Method in Nursing Research: An Integrative Review , 2013, ANS. Advances in nursing science.

[24]  Robert R. Hoffman,et al.  The Problem of Extracting the Knowledge of Experts from the Perspective of Experimental Psychology , 1987, AI Mag..

[25]  A. How to Apply the Anthropological Technique of Participant Observation to Knowledge Acquisition for Expert Systems , 2022 .

[26]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[27]  Larry J. Eshelman,et al.  MOLE: A Tenacious Knowledge-Acquisition Tool , 1987, Int. J. Man Mach. Stud..

[28]  Rachel Manber,et al.  Detecting critical decision points in psychotherapy and psychotherapy + medication for chronic depression. , 2013, Journal of consulting and clinical psychology.

[29]  Lisanne Bainbridge,et al.  Verbal reports as evidence of the process operator's knowledge , 1999, Int. J. Hum. Comput. Stud..

[30]  Ujjwal Maulik,et al.  Survey of Multiobjective Evolutionary Algorithms for Data Mining: Part II , 2014, IEEE Transactions on Evolutionary Computation.

[31]  E. N. Corlett,et al.  Evaluation of human work : a practical ergonomics methodology , 1991 .

[32]  Hussein A. Abbass,et al.  An agent-based model to simulate and analyse behaviour under noisy and deceptive information , 2013, Adapt. Behav..

[33]  Maria K. E. Lahman,et al.  Photo-elicitation: reflexivity on method, analysis, and graphic portraits , 2015 .

[34]  John R. Anderson A spreading activation theory of memory. , 1983 .

[35]  C. Wannheden,et al.  HIV and tuberculosis coinfection: a qualitative study of treatment challenges faced by care providers. , 2013, The international journal of tuberculosis and lung disease : the official journal of the International Union against Tuberculosis and Lung Disease.

[36]  Harleen Kaur,et al.  A Multi-Intelligent Agent for Knowledge Discovery in Database (MIAKDD): Cooperative Approach with Domain Expert for Rules Extraction , 2014, ICIC.

[37]  Chris Reed,et al.  Araucaria: Software for Argument Analysis, Diagramming and Representation , 2004, Int. J. Artif. Intell. Tools.

[38]  Wayne Zachary,et al.  Experimental Validation of the Attention Switching Component of the COGNET Framework , 1991 .

[39]  Chengqi Zhang,et al.  Agent-Mining Interaction: An Emerging Area , 2007, AIS-ADM.

[40]  Nigel Shadbolt,et al.  A comparison of sorting techniques in knowledge acquisition , 1992 .

[41]  Hussein A. Abbass,et al.  On the Role of Working Memory in Trading-Off Skills and Situation Awareness in Sudoku , 2014, ICONIP.

[42]  Elissaveta Gourova,et al.  Audit of knowledge flows and critical business processes , 2012, EuroPLoP '12.

[43]  Chris Dede,et al.  Virtual Performance Assessment in Immersive Virtual Environments , 2012 .

[44]  Richard F. Gunst,et al.  Applied Regression Analysis , 1999, Technometrics.

[45]  Michael E. Atwood,et al.  Project Ernestine: Validating a GOMS Analysis for Predicting and Explaining Real-World Task Performance , 1993, Hum. Comput. Interact..

[46]  Rasmus Brun Pedersen,et al.  Process-Tracing Methods: Foundations and Guidelines , 2013 .

[47]  Gary L. Geis,et al.  137: LEVERAGING CRITICAL DECISION METHOD AND SIMULATION-BASED TRAINING TO ACCELERATE SEPSIS RECOGNITION , 2013 .

[48]  Monireh Hosseini,et al.  Knowledge-collector agents: Applying intelligent agents in marketing decisions with knowledge management approach , 2013, Knowl. Based Syst..

[49]  Dan Zhou,et al.  A Multi-Agent Distributed Data Mining Model Based on Algorithm Analysis and Task Prediction , 2010, 2010 2nd International Conference on Information Engineering and Computer Science.

[50]  Jan Peters,et al.  Reinforcement learning in robotics: A survey , 2013, Int. J. Robotics Res..

[51]  Lucas A. Overbey,et al.  Automated sensing and social network analysis in virtual worlds , 2010, 2010 IEEE International Conference on Intelligence and Security Informatics.

[52]  Rajib Mall,et al.  Application of elitist multi-objective genetic algorithm for classification rule generation , 2008, Appl. Soft Comput..

[53]  Thomas R. Gruber,et al.  Automated Knowledge Acquisition for Strategic Knowledge , 2005, Machine Learning.

[54]  Geoff Kuehne,et al.  "I Don't Know What's Right Anymore": Engaging Distressed Interviewees Using Graphic-Elicitation , 2013 .

[55]  Chi-Bin Cheng,et al.  Intelligent agents for e-marketplace: Negotiation with issue trade-offs by fuzzy inference systems , 2006, Decis. Support Syst..

[56]  Andreas Holzinger,et al.  Data Mining with Decision Trees: Theory and Applications , 2015, Online Inf. Rev..

[57]  Béatrice S. Hasler,et al.  Virtual research assistants: Replacing human interviewers by automated avatars in virtual worlds , 2013, Comput. Hum. Behav..

[58]  Ujjwal Maulik,et al.  Multiobjective Genetic Clustering for Pixel Classification in Remote Sensing Imagery , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[59]  John R. Anderson ACT: A simple theory of complex cognition. , 1996 .

[60]  Jacqueline A Guhde,et al.  An Evaluation Tool to Measure Interdisciplinary Critical Incident Verbal Reports , 2014, Nursing education perspectives.

[61]  Robert R. Hoffman,et al.  Perspectives on Cognitive Task Analysis: Historical Origins and Modern Communities of Practice , 2008 .

[62]  Igor Vatolkin,et al.  Exploration of Two-Objective Scenarios on Supervised Evolutionary Feature Selection: A Survey and a Case Study (Application to Music Categorisation) , 2015, EMO.

[63]  Chi Fai Cheung,et al.  A systematic approach for knowledge auditing: a case study in transportation sector , 2007, J. Knowl. Manag..

[64]  Abdelmonem A. Afifi,et al.  Statistical Analysis: A Computer Oriented Approach. , 1973 .

[65]  Nur Syufiza Ahmad Shukor,et al.  Knowledge audit process - tales of two organizations , 2011, 2011 International Conference on Research and Innovation in Information Systems.

[66]  Peter Essens,et al.  COADE: A Framework for Cognitive Analysis, Design, and Evaluation. , 1995 .

[67]  A Naweed,et al.  Investigations into the skills of modern and traditional train driving. , 2014, Applied ergonomics.

[68]  Udo Arend,et al.  Analysing complex tasks with an extended GOMS model , 1989, Informatics and Psychology Workshop.

[69]  James Geiwitz,et al.  Empirical Comparisons of Knowledge Acquisition Techniques , 1991 .

[70]  Ron Sun,et al.  Cognition and Multi-Agent Interaction: The CLARION Cognitive Architecture: Extending Cognitive Modeling to Social Simulation , 2005 .

[71]  Håkan Alm,et al.  Context adaptable driver information--or, what do whom need and want when? , 2014, Applied ergonomics.

[72]  Jörg Beringer,et al.  Object-based action planning , 1989, Informatics and Psychology Workshop.

[73]  Neville A Stanton,et al.  Combining network analysis with Cognitive Work Analysis: insights into social organisational and cooperation analysis , 2015, Ergonomics.

[74]  V. Krovi,et al.  Robotic Minimally Invasive Surgical skill assessment based on automated video-analysis motion studies , 2012, 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob).

[75]  Yili Liu,et al.  Queueing Network-Model Human Processor (QN-MHP): A computational architecture for multitask performance in human-machine systems , 2006, TCHI.

[76]  Hussein A. Abbass,et al.  Society of Mind cognitive agent architecture applied to drivers adapting in a traffic context , 2014, Adapt. Behav..

[77]  Li Xiu,et al.  Application of data mining techniques in customer relationship management: A literature review and classification , 2009, Expert Syst. Appl..

[78]  Vladimir V. Rybakov,et al.  Logic of knowledge and discovery via interacting agents - Decision algorithm for true and satisfiable statements , 2009, Inf. Sci..

[79]  Klaus Krippendorff,et al.  On the reliability of identifying design moves in protocol analysis , 2013 .

[80]  Mike Sharples,et al.  Socio-cognitive engineering: A methodology for the design of human-centred technology , 2002, European Journal of Operational Research.

[81]  R. J. Kuo,et al.  Integration of ART2 neural network and genetic K-means algorithm for analyzing Web browsing paths in electronic commerce , 2005, Decis. Support Syst..

[82]  Christophe Kolski,et al.  State of the Art on the Cognitive Walkthrough Method, Its Variants and Evolutions , 2010, Int. J. Hum. Comput. Interact..

[83]  J. M. Orasanu Decision Making (Naturalistic), Psychology of , 2001 .

[84]  Hussein A. Abbass,et al.  Discovering Delay Patterns in Arrival Traffic with Dynamic Continuous Descent Approaches Using Co-Evolutionary Red Teaming , 2012 .

[85]  Tünde Varga-Atkins,et al.  From drawings to diagrams: maintaining researcher control during graphic elicitation in qualitative interviews , 2009 .

[86]  Gavriel Salvendy,et al.  Development of a Methodology for Optimizing the Elicited Knowledge , 1993, HCI.

[87]  Robert R. Hoffman,et al.  The Forgotten History of Cognitive Task Analysis , 2008 .

[88]  A. Mechelli,et al.  Using Support Vector Machine to identify imaging biomarkers of neurological and psychiatric disease: A critical review , 2012, Neuroscience & Biobehavioral Reviews.

[89]  H. Seo,et al.  Neural basis of reinforcement learning and decision making. , 2012, Annual review of neuroscience.

[90]  David E. Kieras,et al.  Towards a Practical GOMS Model Methodology for User Interface Design , 1988 .

[91]  A. Muijtjens,et al.  Workplace-based assessment: raters’ performance theories and constructs , 2012, Advances in Health Sciences Education.

[92]  Edmon Begoli,et al.  Design Principles for Effective Knowledge Discovery from Big Data , 2012, 2012 Joint Working IEEE/IFIP Conference on Software Architecture and European Conference on Software Architecture.

[93]  Lee T. Ostrom,et al.  Task Analysis Techniques , 2012 .

[94]  Patricia R Ebright,et al.  Advancing the science of research in nursing education: contributions of the critical decision method. , 2014, The Journal of nursing education.

[95]  John E. Laird,et al.  The Soar Cognitive Architecture , 2012 .

[96]  Natalia Juristo Juzgado,et al.  Effectiveness of Requirements Elicitation Techniques: Empirical Results Derived from a Systematic Review , 2006, 14th IEEE International Requirements Engineering Conference (RE'06).

[97]  Nathalie Aussenac-Gilles,et al.  Making a method of problem solving explicit with MACAO , 1994, Int. J. Hum. Comput. Stud..

[98]  Manojit Chattopadhyay,et al.  Comparison of visualization of optimal clustering using self-organizing map and growing hierarchical self-organizing map in cellular manufacturing system , 2014, Appl. Soft Comput..

[99]  T. Landauer,et al.  Handbook of Human-Computer Interaction , 1997 .

[100]  Mark C. Fox,et al.  Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. , 2011, Psychological bulletin.

[101]  Anil K. Jain,et al.  Data clustering: a review , 1999, CSUR.

[102]  Dennis E. Egan,et al.  Handbook of Human Computer Interaction , 1988 .

[103]  John H. Boose,et al.  Recent progress in ACQUINAS: a knowledge acquisition workbench , 1989 .

[104]  Michael Georgiopoulos,et al.  APHID: An architecture for private, high-performance integrated data mining , 2010, Future Gener. Comput. Syst..

[105]  Arthur D. Fisk,et al.  Application of Automatic/Controlled Processing Theory to Training Tactical Command and Control Skills: 1. Background and Task Analytic Methodology , 1988 .

[106]  Byung Ro Moon,et al.  Multiobjective evolutionary algorithms for dynamic social network clustering , 2010, GECCO '10.

[107]  Elizabeth M. Borycki,et al.  Integrating Heuristic Evaluation with Cognitive Walkthrough: Development of a Hybrid Usability Inspection Method , 2015, ITCH.

[108]  John H. Boose,et al.  A survey of knowledge acquisition techniques and tools , 1993 .

[109]  Nikos Papamarkos,et al.  Development and Evaluation of Text Localization Techniques Based on Structural Texture Features and Neural Classifiers , 2009, 2009 10th International Conference on Document Analysis and Recognition.

[110]  Ujjwal Maulik,et al.  A Survey of Multiobjective Evolutionary Algorithms for Data Mining: Part I , 2014, IEEE Transactions on Evolutionary Computation.

[111]  Catherine Garbay,et al.  A knowledge-driven agent-centred framework for data mining in EMG. , 2002, Comptes rendus biologies.

[112]  Qingfu Zhang,et al.  Multiobjective evolutionary algorithms: A survey of the state of the art , 2011, Swarm Evol. Comput..

[113]  E M Roth,et al.  Cognitive simulation as a tool for cognitive task analysis. , 1992, Ergonomics.

[114]  Kenneth A. Yates,et al.  TOWARDS A TAXONOMY OF COGNITIVE TASK ANALYSIS METHODS: A SEARCH FOR COGNITION AND TASK ANALYSIS INTERACTIONS , 2007 .

[115]  Gary S. Kahn,et al.  MORE: An Intelligent Knowledge Acquisition Tool , 1985, IJCAI.

[116]  David F. Feldon,et al.  Cognitive task analysis , 2009 .

[117]  R. Sun Cognition and Multi-Agent Interactions: From Cognitive Modeling to Social Simulation , 2005 .

[118]  Neville A Stanton,et al.  What is on your mind? Using the perceptual cycle model and critical decision method to understand the decision-making process in the cockpit , 2013, Ergonomics.

[119]  A. Mark Williams,et al.  Skill-based differences in option generation in a complex task: a verbal protocol analysis , 2011, Cognitive Processing.

[120]  W. Bainbridge The Scientific Research Potential of Virtual Worlds , 2007, Science.

[121]  Avelino J. Gonzalez,et al.  Learning tactical human behavior through observation of human performance , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[122]  Rhona Flin,et al.  Surgeons' intraoperative decision making and risk management. , 2011, American journal of surgery.

[123]  Krassen Stefanov,et al.  Applied Cognitive Task Analysis in the Context of Serious Games Development , 2011 .

[124]  Martine B. Powell,et al.  Improving child investigative interviewer performance through computer-based learning activities , 2016 .

[125]  Francisco Herrera,et al.  Multiobjective genetic fuzzy rule selection of single granularity-based fuzzy classification rules and its interaction with the lateral tuning of membership functions , 2011, Soft Comput..

[126]  Catherine Garbay,et al.  Knowledge construction from time series data using a collaborative exploration system , 2007, J. Biomed. Informatics.

[127]  Li Chen,et al.  Usability Guidelines for Product Recommenders Based on Example Critiquing Research , 2011, Recommender Systems Handbook.

[128]  J. Lemke Analyzing Verbal Data: Principles, Methods, and Problems , 2012 .

[129]  Shay Sayre Analyzing Verbal Data , 2001 .

[130]  MusílekPetr,et al.  A survey of Knowledge Discovery and Data Mining process models , 2006 .

[131]  Jeremy N. Bailenson,et al.  A Method for Longitudinal Behavioral Data Collection in Second Life , 2008, PRESENCE: Teleoperators and Virtual Environments.

[132]  Karen Salmon,et al.  Do Non‐Verbal Aids Increase the Effectiveness of ‘Best Practice’ Verbal Interview Techniques? An Experimental Study , 2012 .

[133]  Martha Driessnack,et al.  Arts-based data collection techniques used in child research. , 2012, Journal for specialists in pediatric nursing : JSPN.

[134]  Thomas R. G. Green,et al.  The Structure of Command Languages: An Experiment on Task-Action Grammar , 1989, Int. J. Man Mach. Stud..

[135]  B. Woodward Knowledge acquisition at the front end: defining the domain , 1990 .

[136]  A. Blackwell,et al.  Graphic elicitation: using research diagrams as interview stimuli , 2006 .

[137]  Hussein A. Abbass,et al.  On trust and influence: A computational red teaming game theoretic perspective , 2014, the 2014 Seventh IEEE Symposium on Computational Intelligence for Security and Defense Applications (CISDA).

[138]  Roberta Calderwood,et al.  Critical decision method for eliciting knowledge , 1989, IEEE Trans. Syst. Man Cybern..

[139]  Terri Gullickson Evaluation of Human Work: A Practical Ergonomics Methodology. , 1996 .

[140]  Jeffrey M. Bradshaw,et al.  Knowledge acquisition as a constructive modeling activity , 1993, Int. J. Intell. Syst..

[141]  Ronald R. Yager,et al.  The use of ordered weighted averaging method for decision making under uncertainty , 2014, Int. Trans. Oper. Res..

[142]  Habiba Drias,et al.  From data mining to knowledge mining: Application to intelligent agents , 2015, Expert Syst. Appl..

[143]  Richard T. Gill,et al.  Conceptual Graph Analysis: Knowledge Acquisition for Instructional System Design , 1993 .

[144]  Gillian Ragsdell,et al.  Knowledge Audit: Findings from the Energy Sector , 2014 .

[145]  Catrien Notermans,et al.  Researching religion: the iconographic elicitation method , 2013 .

[146]  Frans Coenen,et al.  EMADS: An extendible multi-agent data miner , 2009, Knowl. Based Syst..

[147]  D. Edwards Data Mining: Concepts, Models, Methods, and Algorithms , 2003 .

[148]  R. Hutton,et al.  Applied cognitive task analysis (ACTA): a practitioner's toolkit for understanding cognitive task demands. , 1998, Ergonomics.

[149]  Brian R. Gaines,et al.  Integrated knowledge acquisition architectures , 1992, Journal of Intelligent Information Systems.

[150]  Moshe Leshno,et al.  Statistical Methods for Data Mining , 2010, Data Mining and Knowledge Discovery Handbook.

[151]  Célia Ghedini Ralha,et al.  A multi-agent data mining system for cartel detection in Brazilian government procurement , 2012, Expert Syst. Appl..

[152]  Allen Newell,et al.  The model human processor: An engineering model of human performance. , 1986 .

[153]  Nick F. Pidgeon,et al.  The Use of Grounded Theory for Conceptual Analysis in Knowledge Elicitation , 1991, Int. J. Man Mach. Stud..

[154]  Thomas R. Gruber,et al.  The Acquisition of Strategic Knowledge , 1989 .

[155]  Ameet Talwalkar,et al.  Foundations of Machine Learning , 2012, Adaptive computation and machine learning.

[156]  M. Angela Sasse,et al.  Eliciting and modelling expertise for serious games in project management , 2011, Entertain. Comput..

[157]  Otto Petrovic,et al.  Serious Games on the Move , 2009 .

[158]  John R. Anderson,et al.  Enhancing ACT-R's Perceptual-Motor Abilities , 2008 .

[159]  Patrick J. F. Groenen,et al.  Modern Multidimensional Scaling: Theory and Applications , 2003 .

[160]  Jeffrey M. Bradshaw,et al.  Decision Analysis Techniques for Knowledge Acquisition: Combining Information and Preferences Using Aquinas and Axotl , 1990, Int. J. Man Mach. Stud..

[161]  John F. Sowa,et al.  Conceptual analysis as a basis for knowledge acquisition , 1992 .

[162]  Rhona Flin,et al.  Intra-operative decision making by ophthalmic surgeons , 2013, British Journal of Ophthalmology.

[163]  D.M. Mount,et al.  An Efficient k-Means Clustering Algorithm: Analysis and Implementation , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[164]  David Embrey,et al.  Task Analysis Techniques , 2019, Risk Assessment.

[165]  Siti Salwah Salim,et al.  Investigating cognitive task difficulties and expert skills in e-Learning storyboards using a cognitive task analysis technique , 2012, Comput. Educ..

[166]  David H. Jonassen,et al.  Structural Knowledge: Techniques for Representing, Conveying, and Acquiring Structural Knowledge , 1993 .

[167]  Kenneth A. Yates,et al.  Advancing the practice of cognitive task analysis: a call for taxonomic research , 2011 .

[168]  Gavriel Salvendy,et al.  The cognitive task analysis methods for job and task design: review and reappraisal , 2004, Behav. Inf. Technol..

[169]  Aditya Kalyanpur,et al.  Automatic knowledge extraction from documents , 2012, IBM J. Res. Dev..

[170]  Joan M. Ryder,et al.  Integrating cognitive task analysis into instructional systems development , 1993 .

[171]  Phyllis Reisner,et al.  Formal Grammar and Human Factors Design of an Interactive Graphics System , 1981, IEEE Transactions on Software Engineering.

[172]  Ian N. Durbach,et al.  Modeling uncertainty in multi-criteria decision analysis , 2012, Eur. J. Oper. Res..

[173]  Hsinchun Chen,et al.  Design and evaluation of a multi-agent collaborative Web mining system , 2003, Decis. Support Syst..

[174]  John P. McDermott,et al.  SALT: A Knowledge Acquisition Language for Propose-and-Revise Systems , 1993, Artif. Intell..

[175]  Ujjwal Maulik,et al.  A multiobjective approach to MR brain image segmentation , 2011, Appl. Soft Comput..

[176]  Richard E. Neapolitan,et al.  Probabilistic reasoning in expert systems - theory and algorithms , 2012 .

[177]  Petri Ylikoski Process-Tracing Methods: Foundations and Guidelines , 2015 .

[178]  Jan de Leeuw,et al.  Modern Multidimensional Scaling: Theory and Applications (Second Edition) , 2005 .

[179]  Gordon Weller,et al.  Decision making strategies used by experts and the potential for training intuitive skills: A preliminary study , 2013 .

[180]  Kyung Hyan Yoo,et al.  Implications for Recommender System Design , 2013 .

[181]  Dario D. Salvucci Integration and Reuse in Cognitive Skill Acquisition , 2013, Cogn. Sci..

[182]  Stephen Marsland,et al.  Machine Learning: An Algorithmic Perspective, Second Edition , 2014 .

[183]  P. Holland Statistics and Causal Inference , 1985 .

[184]  Philip S. Yu,et al.  A brief introduction to agent mining , 2012, Autonomous Agents and Multi-Agent Systems.

[185]  Norita B. Ahmad,et al.  The KCRM knowledge audit: model and case illustration , 2013 .

[186]  Zhengxin Chen,et al.  An integrated interactive environment for knowledge discovery from heterogeneous data resources , 2001, Inf. Softw. Technol..

[187]  W. Bodmer Principles of Scientific Management , 1993, FASEB journal : official publication of the Federation of American Societies for Experimental Biology.

[188]  T. Ormerod,et al.  Naturalistic Decision Making and Macrocognition , 2012 .

[189]  M. Powell,et al.  The effects of e-simulation interview training on teachers' use of open-ended questions. , 2015, Child abuse & neglect.

[190]  Eric Chown,et al.  Cognitive Modeling , 2014, Computing Handbook, 3rd ed..

[191]  Jeffrey M. Bradshaw,et al.  New approaches to constructivist knowledge acquisition tool development , 1993 .

[192]  David Farrell,et al.  Adapting Cognitive Walkthrough to Support Game Based Learning Design , 2014, Int. J. Game Based Learn..

[193]  D A DaRosa,et al.  Faculty evaluation of simulation-based modules for assessment of intraoperative decision making. , 2011, Surgery.

[194]  Inn Seock Kim Human reliability analysis in the man–machine interface design review , 2001 .

[195]  Longbing Cao,et al.  Agent Mining: The Synergy of Agents and Data Mining , 2009, IEEE Intelligent Systems.

[196]  David F. Feldon,et al.  Cognitive Task Analysis–Based Training , 2013 .

[197]  J. Cullen,et al.  The Knowledge Acquisition Bottleneck: Time for Reassessment? , 1988 .

[198]  Joseph L. Kannry,et al.  Integrating usability testing and think-aloud protocol analysis with "near-live" clinical simulations in evaluating clinical decision support , 2012, Int. J. Medical Informatics.

[199]  Neville A Stanton,et al.  The process of processing: exploring the validity of Neisser's perceptual cycle model with accounts from critical decision-making in the cockpit , 2015, Ergonomics.

[200]  N. Shadbolt,et al.  Eliciting Knowledge from Experts: A Methodological Analysis , 1995 .

[201]  Nancy J. Cooke,et al.  Varieties of knowledge elicitation techniques , 1994, Int. J. Hum. Comput. Stud..

[202]  Marc Linster Explicit and operational models as a basis for second generation knowledge acquisition tools , 1993 .

[203]  Sunhee Park,et al.  Development of professional engineers' authentic contexts in blended learning environments , 2012, Br. J. Educ. Technol..

[204]  Igor Linkov,et al.  Multi-criteria decision analysis in environmental sciences: ten years of applications and trends. , 2011, The Science of the total environment.

[205]  John P. Gallagher,et al.  Cognitive/information processing psychology and instruction: Reviewing recent theory and practice , 1979 .

[206]  Ana L. C. Bazzan,et al.  Integrating Knowledge Through Cooperative Negotiation - A Case Study in Bioinformatics , 2005, AIS-ADM.

[207]  Gary Klein,et al.  Working Minds: A Practitioner's Guide to Cognitive Task Analysis , 2006 .

[208]  Olivier Poch,et al.  Knowledge Discovery in Variant Databases Using Inductive Logic Programming , 2013, Bioinformatics and biology insights.

[209]  Alessandra Russo,et al.  Inductive Logic Programming in Answer Set Programming , 2011, ILP.

[210]  Raymond Y. K. Lau,et al.  Knowledge discovery for adaptive negotiation agents in e-marketplaces , 2008, Decis. Support Syst..

[211]  Marion Bogo,et al.  Acting Like It Matters: A Scoping Review of Simulation in Child Welfare Training , 2014 .

[212]  Ingo Ruhmann,et al.  KRITON: A Knowledge-Acquisition Tool for Expert Systems , 1987, Int. J. Man Mach. Stud..

[213]  Sherrie P. Gott,et al.  A Procedural Guide to Cognitive Task Analysis: The PARI Methodology. , 1995 .

[214]  Ann M. Bisantz,et al.  Discussion Panel , 2014, 2021 IEEE International Conference on Automation/XXIV Congress of the Chilean Association of Automatic Control (ICA-ACCA).

[215]  John H. Holmes,et al.  Intelligent data analysis in biomedicine , 2007, J. Biomed. Informatics.

[216]  Guy H. Walker,et al.  Human Factors Methods: A Practical Guide for Engineering and Design , 2012 .

[217]  Katsuhiko Ogasawara,et al.  Constructing Mammography Examination Process Ontology Using Affinity Diagram and Hierarchical Task Analysis , 2013, MedInfo.

[218]  AbbassH.,et al.  Computational Red Teaming , 2015 .