Concept Development for Future Domains: A New Method of Knowledge Elicitation

Abstract : During the development of operational concepts for the Future Combat System of Systems' Unit of Employment and Unit of Action it became clear that the Army needed a more effective and efficient method for envisioning the future. A review of existing knowledge elicitation techniques indicated that current methods are not easily applied to the development of future concepts. They do not adequately address the cognitive impact new concepts and technology have on Soldiers. In addition, existing methods do not thoroughly examine the potential unforeseen impacts the introduction of the new technology will have. The inherent difficulty in envisioning new concepts requires a more systematic approach to elicit knowledge from domain experts. This report reviews existing methods and describes a new method of knowledge elicitation to more effectively support the development of future concepts, evaluate the impact of new technology, and solve difficult problems where information and expertise is dispersed among many individuals.

[1]  Valerie J. Shute,et al.  Exploratory Test of an Automated Knowledge Elicitation and Organization Tool , 1999 .

[2]  Experience, Expertise and the Forecasting Performance of Security Analysts , 1997 .

[3]  Roberta Calderwood,et al.  Critical decision method for eliciting knowledge , 1989, IEEE Trans. Syst. Man Cybern..

[4]  N. Shadbolt,et al.  Eliciting Knowledge from Experts: A Methodological Analysis , 1995 .

[5]  J. C. Flanagan.  Psychological Bulletin THE CRITICAL INCIDENT TECHNIQUE , 2022 .

[6]  David Woods,et al.  Commentary Designs are hypotheses about how artifacts shape cognition and collaboration , 1998 .

[7]  H. Johnson,et al.  Task knowledge structures: Psychological basis and integration into system design ☆ , 1991 .

[8]  Mary Beth Rosson,et al.  Getting around the task-artifact cycle: how to make claims and design by scenario , 1992, TOIS.

[9]  P. Feltovich,et al.  Issues of expert flexibility in contexts characterized by complexity and change , 1997 .

[10]  Ashton B. Carter,et al.  Keeping the edge : managing defense for the future , 2001 .

[11]  Gerhard Rosegger,et al.  The economics of production and innovation , 1980 .

[12]  William David Salisbury,et al.  Understanding the influence of organizational change strategies on information technology and knowledge management strategies , 2001, Decis. Support Syst..

[13]  Brian R. Gaines,et al.  Knowledge acquisition tools based on personal construct psychology , 1993, The Knowledge Engineering Review.

[14]  C. C. Leek,et al.  Job Analysis , 1970 .

[15]  Elizabeth S. Cordingley,et al.  Knowledge elicitation techniques for knowledge-based systems , 1989 .

[16]  Gary Klein,et al.  Applied Cognitive Task Analysis (ACTA) Methodology , 1997 .

[17]  Huba Wass de Czege,et al.  Optimizing Future Battle Command Technologies , 1998 .

[18]  G. Kelly.  The Psychology of Personal Constructs , 2020 .

[19]  G. Tichy.  Over-Optimism Among Experts in Assessment and Foresight , 2002 .

[20]  J. St B. T. Evans.  The knowledge elicitation problem: a psychological perspective , 1988 .

[21]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[22]  Sidney Dekker,et al.  Anticipating the effects of technological change: A new era of dynamics for human factors , 2000 .

[23]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[24]  K. R. Chervinskaya,et al.  Some methodological aspects of tacit knowledge elicitation , 2000, J. Exp. Theor. Artif. Intell..

[25]  Sherrie P. Gott,et al.  Rediscovering Learning: Acquiring Expertise in Real World Problem Solving Tasks , 1998 .

[26]  R. S. Isenson,et al.  Technological Forecasting in Perspective , 1966 .

[27]  Fine Sa,et al.  An introduction to functional job analysis: a scaling of selected tasks from the social welfare field. , 1971 .

[28]  F. Aguilar.  Scanning the business environment , 1967 .

[29]  Han Reichgelt,et al.  ALTO: An automated laddering tool. , 1990 .

[30]  Jane M. Booker,et al.  Eliciting and analyzing expert judgement - a practical guide , 2001, ASA-SIAM series on statistics and applied probability.

[31]  Mark Cantley,et al.  Technological forecasting in perspective , 1967 .

[32]  Benjamin S. Bloom,et al.  Taxonomy of Educational Objectives: The Classification of Educational Goals. , 1957 .

[33]  Sherrie P. Gott,et al.  A Procedural Guide to Cognitive Task Analysis: The PARI Methodology. , 1995 .

[34]  D. Woods,et al.  Automation Surprises , 2001 .

[35]  David H. Jonassen,et al.  Handbook of Task Analysis Procedures , 1989 .

[36]  Mica R. Endsley,et al.  A Survey of Situation Awareness Requirements in Air-to-Air Combat Fighters , 1993 .

[37]  David E. Kieras,et al.  Towards a Practical GOMS Model Methodology for User Interface Design , 1988 .

[38]  Fraser Hamilton,et al.  Predictive evaluation using task knowledge structures , 1996, CHI Conference Companion.

[39]  Mica R. Endsley,et al.  Direct Measurement of Situation Awareness: Validity and Use of SAGAT , 2000 .

[40]  Gary A. Klein,et al.  The Development of Knowledge Elicitation Methods for Capturing Military Expertise. , 1996 .

[41]  David D Woods,et al.  Steering the Reverberations of Technology Change on Fields of Practice: Laws that Govern Cognitive Work , 2019, Proceedings of the Twenty-Fourth Annual Conference of the Cognitive Science Society.

[42]  Colin Camerer,et al.  The process-performance paradox in expert judgment - How can experts know so much and predict so badly? , 1991 .

[43]  John P. Robinson,et al.  Futures under glass , 1990 .

[44]  Valerie L. Shalin,et al.  Cognitive task analysis , 2000 .

[45]  Robert C. Mecham,et al.  A study of job characteristics and job dimensions as based on the Position Analysis Questionnaire (PAQ). , 1972 .

[46]  M Hoejer,et al.  HISTORICAL DETERMINISM AND BACKCASTING IN FUTURE STUDIES , 2000 .

[47]  Shawn P. Curley,et al.  How Do I Know That You Know? Cues Used to Infer Another's Expertise and to Signal One's Own Expertise , 2003 .

[48]  J. W. Cunningham,et al.  Systematically Derived Work Dimensions: Factor Analyses of the Occupation Analysis Inventory , 1983 .

[49]  W. Borman,et al.  A Cognitively-Oriented Approach to Task Analysis and Test Development , 1995 .

[50]  John Robinson.  Energy backcasting A proposed method of policy analysis , 1982 .

[51]  Norman Crolee Dalkey,et al.  An experimental study of group opinion , 1969 .