Acquiring and Assessing Knowledge From Multiple Experts Using Graphical Representations

Publisher Summary The integrity of the process of eliciting and representing human expertise is fundamental to any knowledge-based system. This process is referred to as “knowledge acquisition.” Human expertise can be found in documents and databases that store records of past transactions as well as the guidelines and procedures used to process those transactions. An active area of current research involves “mining” these data to capture knowledge to develop rules for incorporation into knowledge-based systems. This chapter presents a thorough review of current practice in eliciting, representing, and amalgamating knowledge from multiple experts with a focus on the use of graphical representations to support the process. The important points of the discussion are illustrated by presenting a specific methodology for eliciting and combining knowledge from multiple experts. This methodology provides a statistically defensive summarization for assessment. The results of a pilot test of its implementation over the Internet are also presented in the chapter. This chapter concludes with a discussion on the need for knowledge acquisition techniques that permit the quality of the rules developed based on the acquired knowledge to be quantitatively assessed and subsequently embedded into decision support systems.

[1]  Kathleen M. Carley Extracting culture through textual analysis , 1994 .

[2]  Joseph P. Romano,et al.  Bootstrap technology and applications , 1992 .

[3]  Tracy L. Wells Hypertext as a means for knowledge acquisition , 1989, SGAR.

[4]  Kathleen M. Carley,et al.  Extracting, Representing, and Analyzing Mental Models , 1992 .

[5]  Nancy J. Cooke,et al.  Varieties of knowledge elicitation techniques , 1994, Int. J. Hum. Comput. Stud..

[6]  Paul K. Moser,et al.  Knowledge and Evidence , 1992 .

[7]  Kathleen McKeown,et al.  Generating Concise Natural Language Summaries , 1995, Inf. Process. Manag..

[8]  Eric Horvitz,et al.  Decision Analysis and Expert Systems , 1991, AI Mag..

[9]  C. Hookway,et al.  Warrant: The Current Debate.@@@Warrant and Proper Function. , 1995 .

[10]  Randy H. Katz,et al.  Adaptation and mobility in wireless information systems , 2002, IEEE Communications Magazine.

[11]  Kathleen M. Carley Extracting team mental models through textual analysis , 1997 .

[12]  Martha Grabowski,et al.  The relationship between three-dimensional imaging and group decision making: an exploratory study , 1997, IEEE Trans. Syst. Man Cybern. Part A.

[13]  Olivier Corby,et al.  Knowledge Acquisition for Explainable, Multi-Expert, Knowledge-Based Design Systems , 1992, EKAW.

[14]  Anne P. Massey,et al.  Understanding and facilitating group problem structuring and formulation: Mental representations, interaction, and representation aids , 1996, Decis. Support Syst..

[15]  Chris Carter,et al.  Multiple decision trees , 2013, UAI.

[16]  John F. Lemmer,et al.  Causal Modeling , 1993, UAI.

[17]  Mark Klein,et al.  Detecting and resolving conflicts among cooperating human and machine-based design agents , 1992, Artif. Intell. Eng..

[18]  R. M. Dudley,et al.  Influence Diagrams, Belief Nets and Decision Analysis. , 1991 .

[19]  B. Russell,et al.  “Human Knowledge—Its Scope and Limits” , 1949, Philosophy.

[20]  Olivier Corby,et al.  Agent-Based Knowledge Acquisition , 1994, EKAW.

[21]  Donald A. Waterman,et al.  A Guide to Expert Systems , 1986 .

[22]  Marc Linster,et al.  Integrating a knowledge acquisition tool, an expert system shell, and a hypermedia system , 1991 .

[23]  William A. Wallace,et al.  Ethics in Modeling , 1994 .

[24]  Mildred L. G. Shaw,et al.  Validation in a Knowledge Support System: Construing and Consistency with Multiple Experts , 1988, Int. J. Man Mach. Stud..

[25]  John Scott What is social network analysis , 2010 .

[26]  Fred R. McMorris,et al.  The median procedure for n-trees as a maximum likelihood method , 1990 .

[27]  B. Efron,et al.  Bootstrap confidence intervals , 1996 .

[28]  Anne P. Massey,et al.  Focus Groups as a Knowledge Elicitation Technique: An Exploratory Survey , 1991, IEEE Trans. Knowl. Data Eng..

[29]  Cynthia A. Phillips,et al.  The Asymmetric Median Tree - A New Model for Building Consensus Trees , 1996, Discret. Appl. Math..

[30]  M. Bougon Cognition in Organizations: An Analysis of the Utrecht Jazz Orchestra. , 1977 .

[31]  Lisa F. Rau,et al.  Information extraction and text summarization using linguistic knowledge acquisition , 1989, Inf. Process. Manag..

[32]  John L. Bennett,et al.  Learning from user experience with groupware , 1990, CSCW '90.

[33]  Barry G. Silverman,et al.  Coping with ongoing knowledge acquisition from collaborating hierarchies of experts , 1989, SGAR.

[34]  Roberta M. Roth,et al.  A Delphi approach to acquiring knowledge from single and multiple experts , 1990, SIGBDP '90.

[35]  John H. Boose,et al.  Rapid Acquisition and Combination of Knowledge from Multiple Experts in the Same Domain , 1985, Conference on Artificial Intelligence Applications.

[36]  R. M. Oliver,et al.  Influence diagrams, belief nets and decision analysis , 1992 .

[37]  S. Wasserman,et al.  Social Network Analysis: Computer Programs , 1994 .

[38]  W. Harvey Hegarty,et al.  Decision Makers' Beliefs About the Causes and Effects of Structure: An Exploratory Study , 1984 .

[39]  William H. Batchelder,et al.  Consensus analysis of three‐way social network data , 1997 .

[40]  William J. Burns,et al.  Covariance structure models and influence diagrams , 1993 .

[41]  Jeffrey M. Bradshaw,et al.  Knowledge acquisition techniques for group decision support , 1993 .

[42]  Kathleen M. Carley,et al.  Metric inference for social networks , 1994 .

[43]  Sofiane Labidi,et al.  Method for Knowledge Acquisition from Multiple Experts , 1995, SBIA.

[44]  Larry R. Medsker,et al.  Computerized conferencing for knowledge acquisition from multiple experts , 1991 .

[45]  J. Marden Analyzing and Modeling Rank Data , 1996 .

[46]  A. Carlisle Scott,et al.  Practical guide to knowledge acquisition , 1991 .

[47]  Deborah Estrin Interconnection of private networks: A link between industrial and telecommunications policy☆ , 1987 .

[48]  Fausto Giunchiglia,et al.  Understanding Scene Descriptions by Integrating Different Sources of Knowledge , 1992, Int. J. Man Mach. Stud..

[49]  D. Madigan,et al.  Bayesian model averaging and model selection for markov equivalence classes of acyclic digraphs , 1996 .

[50]  Alvin Plantinga,et al.  Warrant: The Current Debate , 1993 .

[51]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[52]  Anupam Basu,et al.  A knowledge based system using multiple expert modules for monitoring leprosy/spl minus/an endemic disease , 1994 .

[53]  Andrew Wirth,et al.  Measuring Differences Between Cognitive Maps , 1992 .

[54]  R. Garner Metacognition and reading comprehension , 1987 .

[55]  N. Shadbolt,et al.  Eliciting Knowledge from Experts: A Methodological Analysis , 1995 .

[56]  Michael P. Wellman,et al.  Real-world applications of Bayesian networks , 1995, CACM.

[57]  Richard L. Lewis,et al.  Combining multiple knowledge sources in an integrated intelligent system , 1993, IEEE Expert.

[58]  Elisabeth Neugebauer,et al.  Professional summarizing: no cognitive simulation without observation , 1998 .

[59]  Colin Eden,et al.  Messing about in problems , 1983 .

[60]  David S. Prerau Selection of an Appropriate Domain for an Expert System , 1985, AI Mag..

[61]  Steve Easterbrook,et al.  Handling conflict between domain descriptions with computer-supported negotiation , 1991 .

[62]  B. Efron Better Bootstrap Confidence Intervals , 1987 .

[63]  Richard A. Johnson,et al.  Applied Multivariate Statistical Analysis , 1983 .

[64]  Hermann Kaindl,et al.  Knowledge acquisition using hypertext , 1992 .

[65]  Eric W. Stein,et al.  A Method to Identify Candidates for Knowledge Acquisition , 1992, J. Manag. Inf. Syst..

[66]  William A. Wallace,et al.  Elecitation of Knowledge from Multiple Experts Using Network Inference , 1997, IEEE Trans. Knowl. Data Eng..

[67]  Marco Guida,et al.  Exploiting multiple knowledge models in the development of real-time diagnostic expert systems for highly automated manufacturing plants , 1993, J. Intell. Manuf..

[68]  Robert E. Kraut,et al.  Computer-mediated communication for intellectual teamwork: a field experiment in group writing , 1990, CSCW '90.

[69]  K. Biolsi,et al.  Techniques for representing expert knowledge. , 1991 .

[70]  R. Rice Computer-Mediated Communication and Organizational Innovation , 1987 .

[71]  William A. Wallace,et al.  Induction of Rules Subject to a Quality Constraint: Probabilistic Inductive Learning , 1993, IEEE Trans. Knowl. Data Eng..

[72]  J. L. Rasmussen "Bootstrap confidence intervals: Good or bad": Comments on Efron (1988) and Strube (1988) and further evaluation. , 1988 .

[73]  Chris D. Paice,et al.  Constructing literature abstracts by computer: Techniques and prospects , 1990, Inf. Process. Manag..

[74]  D. Madigan,et al.  Model Selection and Accounting for Model Uncertainty in Graphical Models Using Occam's Window , 1994 .

[75]  Patrick Wilson,et al.  Cognition and Complexity: The Cognitive Science of Managing Complexity , 1997 .

[76]  David Vance,et al.  On the Requirements of Knowledge Transfer Using Information Systems: A Schema Whereby Such Transfer Is Enhanced , 1998 .

[77]  Jukka Rantanen Hypermedia in Knowledge Acquisition and Specification of User Interface for KBS: An Approach and a Case Study , 1989, SCAI.

[78]  David Krackhardt,et al.  Cognitive social structures , 1987 .

[79]  Robert Johansen,et al.  Groupware: Computer Support for Business Teams , 1988 .

[80]  G. S. Ribar Expert systems technology at peat marwick main , 1987 .

[81]  Andrew B. Whinston,et al.  Decision Support Systems: A Knowledge Based Approach : , 1996 .

[82]  Clive L. Dym,et al.  Knowledge Acquisition from Multiple Experts , 1984, AI Mag..

[83]  Robert T. Clemen,et al.  Making Hard Decisions: An Introduction to Decision Analysis , 1997 .

[84]  Edna Schechtman,et al.  Efficient bootstrap simulation , 1986 .

[85]  Nigel Shadbolt,et al.  Applying the REKAP Methodology to Situation Assessment , 1994, EKAW.

[86]  Efraim Turban,et al.  Decision support and expert systems , 1993 .

[87]  François-Joseph Lapointe,et al.  THE AVERAGE CONSENSUS PROCEDURE: COMBINATION OF WEIGHTED TREES CONTAINING IDENTICAL OR OVERLAPPING SETS OF TAXA , 1997 .

[88]  Y. I. Liou,et al.  Collaborative knowledge acquisition , 1992 .

[89]  Gerard Salton,et al.  Automatic Text Structuring and Summarization , 1997, Inf. Process. Manag..

[90]  Tung Lai Lai,et al.  Supporting inter-expert cooperative work in expert system development , 1994, Proceedings of TENCON'94 - 1994 IEEE Region 10's 9th Annual International Conference on: 'Frontiers of Computer Technology'.

[91]  Ross D. Shachter Evaluating Influence Diagrams , 1986, Oper. Res..

[92]  Bernard Monjardet,et al.  The median procedure in cluster analysis and social choice theory , 1981, Math. Soc. Sci..

[93]  Steven R. LeClair,et al.  Interactive learning: a multiexpert paradigm for acquiring new knowledge , 1989, SGAR.

[94]  John S. Edwards,et al.  Knowledge elicitation and knowledge representation in a large domain with multiple experts , 1995 .

[95]  Robert E. Kraut,et al.  The VideoWindow system in informal communication , 1990, CSCW '90.

[96]  Jim McGovern,et al.  Knowledge acquisition for intelligent decision systems , 1991, Decis. Support Syst..

[97]  Jeffrey M. Bradshaw,et al.  Expertise Transfer and Complex Problems: Using AQUINAS as a Knowledge-Acquisition Workbench for Knowledge-Based Systems , 1993, Int. J. Man Mach. Stud..

[98]  Brian R. Gaines,et al.  Comparing the Conceptual Systems of Experts , 1989, IJCAI.

[99]  Karen L. McGraw,et al.  Knowledge Acquisition: Principles and Guidelines , 1989 .

[100]  Roberta M. Roth,et al.  A workshop approach to acquiring knowledge from single and multiple experts , 1990, SIGBDP '90.

[101]  Ronald A. Howard,et al.  Knowledge Maps , 1989 .

[102]  Joseph P. Romano,et al.  A Review of Bootstrap Confidence Intervals , 1988 .