Human-Automated Judge Learning: A Methodology for Examining Human Interaction With Information Analysis Automation

Human-automated judge learning (HAJL) is a methodology providing a three-phase process, quantitative measures, and analytical methods to support design of information analysis automation. HAJL's measures capture the human and automation's judgment processes, relevant features of the environment, and the relationships between each. Specific measures include achievement of the human and the automation, conflict between them, compromise and adaptation by the human toward the automation, and the human's ability to predict the automation. HAJL's utility is demonstrated herein using a simplified air traffic conflict prediction task. HAJL was able to capture patterns of behavior within and across the three phases with measures of individual judgments and human-automation interaction. Its measures were also used for statistical tests of aggregate effects across human judges. Two between-subject manipulations were crossed to investigate HAJL's sensitivity to interventions in the human's training (sensor noise during training) and in display design (information from the automation about its judgment strategy). HAJL identified that the design intervention impacted conflict and compromise with the automation, participants learned from the automation over time, and those with higher individual judgment achievement were also better able to predict the automation.

[1]  A. Kirlik,et al.  A Methodology for Measuring the Judgmental Components of Situation Awareness , 2000 .

[2]  Kenneth R. Hammond,et al.  Entangled task dismensions: An impediment to interpersonal learning , 1974 .

[3]  H. J. Einhorn.  The use of nonlinear, noncompensatory models in decision making. , 1970, Psychological bulletin.

[4]  Ellen J. Bass,et al.  Human-Automated Judgment Learning: A Methodology to Investigate Human Interaction with Automated Judges , 2002 .

[5]  T. R. Stewart,et al.  Effects of improved information on the components of skill in weather forecasting , 1992 .

[6]  E. Brunswik.  Perception and the Representative Design of Psychological Experiments , 1957 .

[7]  Ray W. Cooksey,et al.  Judgment analysis : theory, methods, and applications , 1996 .

[8]  Peter Freebody,et al.  Cue subset contributions in the hierarchical multivariate lens model: Judgments of children's reading achievement , 1987 .

[9]  K. R. Hammond,et al.  ANALYZING THE COMPONENTS OF CLINICAL INFERENCE. , 1964, Psychological review.

[10]  Lisanne Bainbridge,et al.  Ironies of automation , 1982, Autom..

[11]  Ellen J. Bass,et al.  Architecture and development environment of a knowledge-based monitor that facilitate incremental knowledge-base development , 2004, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[12]  Robert S. Wigton,et al.  Chapter 7 Applications of Judgment Analysis and Cognitive Feedback to Medicine , 1988 .

[13]  L. Dalgleish,et al.  Chapter 10 Decision Making in Child Abuse Cases: Applications of Social Judgment Theory and Signal Detection Theory , 1988 .

[14]  Raja Parasuraman,et al.  Humans and Automation: Use, Misuse, Disuse, Abuse , 1997, Hum. Factors.

[15]  Leonard Adelman,et al.  Examining the effects of cognitive consistency between training and displays , 1998, IEEE Trans. Syst. Man Cybern. Part A.

[16]  John D. Lee,et al.  Trust, self-confidence, and operators' adaptation to automation , 1994, Int. J. Hum. Comput. Stud..

[17]  Gwendolyn E. Campbell,et al.  The Diagnostic Utility of Fuzzy System Modeling for Application in Training Systems , 2000 .

[18]  Jan Noyes,et al.  Civil Aircraft Warning Systems: Future Directions in Information Management and Presentation , 2000 .

[19]  James K. Kuchar,et al.  Prototype Conflict Alerting Logic for Free Flight , 1997 .

[20]  David Woods,et al.  The alarm problem and directed attention in dynamic fault management , 1995 .

[21]  R. Wigton,et al.  Chapter 13 Research in Medical Ethics: The Role of Social Judgment Theory , 1988 .

[22]  Bonnie M. Muir,et al.  Trust in automation. I: Theoretical issues in the study of trust and human intervention in automated systems , 1994 .

[23]  Renwick E. Curry,et al.  Flight-deck automation: promises and problems , 1980 .

[24]  Berndt Brehmer,et al.  Effects of task predictability and cue validity on interpersonal learning of inference tasks involving both linear and nonlinear relations , 1973 .

[25]  James K. Kuchar.  Methodology for Alerting-System Performance Evaluation , 1996 .

[26]  Amy R. Pritchett,et al.  Experimental Studies Of Pilot Performance At Collision Avoidance During Closely Spaced Parallel Approaches , 1997 .

[27]  E. Brunswik,et al.  The Conceptual Framework of Psychology , 1954 .

[28]  M. Doherty,et al.  Effects of cognitive feedback on performance. , 1989 .

[29]  Michael E. Doherty,et al.  Social judgement theory , 1996 .

[30]  Amy R. Pritchett,et al.  Software architecture for a reconfigurable flight simulator. , 2000 .

[31]  K. J. Vicente,et al.  The Ecology of Human-Machine Systems II: Mediating 'Direct Perception' in Complex Work Domains , 1990 .

[32]  K. R. Hammond,et al.  A research paradigm for the study of interpersonal learning. , 1966, Psychological bulletin.

[33]  L. Tucker.  A SUGGESTED ALTERNATIVE FORMULATION IN THE DEVELOPMENTS BY HURSCH, HAMMOND, AND HURSCH, AND BY HAMMOND, HURSCH, AND TODD. , 1964, Psychological review.

[34]  Anil K. Jain,et al.  A Survey of Automated Visual Inspection , 1995, Comput. Vis. Image Underst..

[35]  B. Brehmer,et al.  Cognitive sources of interpersonal conflict: Analysis of interactions between linear and nonlinear cognitive systems☆ , 1973 .

[36]  H. J. Einhorn,et al.  Linear regression and process-tracing models of judgment. , 1979 .

[37]  N. Moray,et al.  Trust in automation. Part II. Experimental studies of trust and human intervention in a process control simulation. , 1996, Ergonomics.

[38]  B. Brehmer.  The psychology of linear judgement models , 1994 .

[39]  Younho Seong,et al.  Judgment and Trust in Conjunction with Automated Decision Aids: A Theoretical Model and Empirical Investigation , 2002 .

[40]  Alex Kirlik,et al.  Modes in Human-Machine Systems: Constructs, Representation, and Classification , 1999 .

[41]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[42]  R. Dawes.  Judgment under uncertainty: The robust beauty of improper linear models in decision making , 1979 .

[43]  Thomas R. Stewart,et al.  Seven components of judgmental forecasting skill: Implications for research and the improvement of forecasts , 1994 .

[44]  Alex Kirlik,et al.  Inferring rule-based strategies in dynamic judgment tasks: toward a noncompensatory formulation of the lens model , 2003, IEEE Trans. Syst. Man Cybern. Part A.

[45]  Peter Freebody,et al.  Teachers’ Predictions of Children’s Early Reading Achievement: An Application of Social Judgment Theory , 1986 .

[46]  Ellen J. Bass,et al.  Measuring the Seat of the Pants: Commercial Airline Pilot Turbulence Assessments in a Full-Motion Simulator , 2002 .

[47]  William M. Jones,et al.  The Effect of Providing Automated Clear Air Turbulence Assessments to Commercial Airline Pilots , 2001 .

[48]  Earl L. Wiener,et al.  Human factors of advanced technology (glass cockpit) transport aircraft , 1989 .

[49]  L. R. Goldberg.  Simple models or simple processes? Some research on clinical judgments. , 1968, The American psychologist.

[50]  Anthony D. Andre,et al.  Displaying Uncertainty in Advanced Navigation Systems , 1998 .

[51]  T. R. Stewart.  A Decomposition of the Correlation Coefficient and its Use in Analyzing Forecasting Skill , 1990 .

[52]  R. Parasuraman.  Vigilance, monitoring, and search. , 1986 .

[53]  Arthur D. Fisk,et al.  Modeling and analysis of a dynamic judgment task using a lens model approach , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[54]  Philip J. Smith,et al.  Brittleness in the design of cooperative problem-solving systems: the effects on user performance , 1997, IEEE Trans. Syst. Man Cybern. Part A.

[55]  Alex Kirlik,et al.  MODES IN HUMAN-MACHINE SYSTEMS: REVIEW, CLASSIFICATION, AND APPLICATION , 1999 .

[56]  K. R. Hammond,et al.  SOME METHODOLOGICAL CONSIDERATIONS IN MULTIPLE-CUE PROBABILITY STUDIES. , 1964, Psychological review.

[57]  D. Corcoran,et al.  Cooperation of listener and computer in a recognition task. II. Effects of computer reliability and "dependent" versus "independent" conditions. , 1972, The Journal of the Acoustical Society of America.

[58]  Gavan Lintern,et al.  Part-Task Training for Tracking and Manual Control , 1985 .

[59]  L J Skitka,et al.  Automation bias: decision making and performance in high-tech cockpits. , 1997, The International journal of aviation psychology.

[60]  Christopher D. Wickens,et al.  A model for types and levels of human interaction with automation , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[61]  R. Jennrich,et al.  Unbalanced repeated-measures models with structured covariance matrices. , 1986, Biometrics.

[62]  Ann M. Bisantz,et al.  Measuring the Fit between Human Judgments and Automated Alerting Algorithms: A Study of Collision Detection , 2003, Hum. Factors.

[63]  James K. Kuchar,et al.  Prototype Conflict Alerting System for Free Flight , 1997 .

[64]  Raja Parasuraman,et al.  Automation- Induced "Complacency": Development of the Complacency-Potential Rating Scale , 1993 .

[65]  N Moray,et al.  Trust, control strategies and allocation of function in human-machine systems. , 1992, Ergonomics.

[66]  Penelope M. Sanderson,et al.  Anesthesia Alarms in Context: An Observational Study , 2001, Hum. Factors.

[67]  Raja Parasuraman,et al.  Performance Consequences of Automation-Induced 'Complacency' , 1993 .

[68]  E. Bass.  Human-automated judgment learning :a research paradigm based on interpersonal learning to investigate human interaction with automated judgments of hazards , 2002 .

[69]  A. Kirlik.  Requirements for psychological models to support design: Towards ecological task analysis , 1991 .

[70]  Anthony J. Masalonis.  Effects of training operators on situation-specific automation reliability , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[71]  Amy R. Pritchett,et al.  Designing Situation Displays to Promote Conformance to Automatic Alerts , 2001 .

[72]  Amy R. Pritchett.  REVIEWING THE ROLE OF COCKPIT ALERTING SYSTEMS , 2001 .

[73]  B. Brehmer.  Social Judgment Theory and the Analysis of Interpersonal Conflict. , 1976 .

[74]  J. Smedslund.  Multiple-probability learning : an inquiry into the origins of perception , 1955 .

[75]  Thomas A. Dingus,et al.  Human Factors Field Evaluation of Automotive Headway Maintenance/Collision Warning Devices , 1997, Hum. Factors.

[76]  A Kirlik,et al.  Modeling Strategic Behavior in Human-Automation Interaction: Why an "Aid" Can (and Should) Go Unused , 1993, Human factors.

[77]  Christopher D. Wickens,et al.  Workload and Reliability of Predictor Displays in Aircraft Traffic Avoidance , 2000 .

[78]  Raja Parasuraman,et al.  Trust in Decision Aids: a Model and Its Training Implications , 1998 .

[79]  Stephen R. Ellis,et al.  Perceived Threat and Avoidance Maneuvers in Response to Cockpit Traffic Displays , 1984 .

[80]  Alex Kirlik,et al.  Modes in human-automation interaction: initial observations about a modeling approach , 1995, 1995 IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century.