Cognitively-engineered multisensor image fusion for military applications

Abstract The fusion of imagery from multiple sensors is a field of research that has been gaining prominence in the scientific community in recent years. The technical aspects of combining multisensory image information have been and are currently being studied extensively. However, the cognitive aspects of multisensor image fusion have not received as much attention. In this study, a concurrent protocol procedure was used to identify how humans fuse information from visible and infrared imagery in low- and high-stress situations. The results of the concurrent protocol were used to develop operator function models, which were then used to develop preliminary design points for fusing multisensor image data. Fused image data were then used in a combat/target identification simulation, and operator performance, accuracy, and speed were compared with results obtained using unfused data. The results show that the model is an accurate depiction of how humans interpret information from multiple disparate sensors in this particular scenario, and that the algorithm design points show promise for assisting fighter pilots in quicker and more accurate target identification.

[1]  Michele Piccini.  Human Factors in the Design of Supervisory Control Systems and Human–Machine Interfaces for Highly Automated Complex Systems , 2002, Cognition, Technology & Work.

[2]  N. Sarter,et al.  Supporting decision-making and action selection under time pressure and uncertainty: The case of inflight icing , 2001 .

[3]  Caroline C. Hayes,et al.  DAISY: a decision support design methodology for complex, experience-centered domains , 2002, IEEE Trans. Syst. Man Cybern. Part A.

[4]  Amar Mitiche,et al.  Multisensor Information Integration for Object Identification , 1993 .

[5]  Marvin S. Cohen,et al.  COGNITIVE ASPECTS OF AUTOMATED TARGET RECOGNITION INTERFACE DESIGN: AN EXPERIMENTAL ANALYSIS , 1997 .

[6]  David Hooper,et al.  Real-time image fusion: a vision aid for helicopter pilotage , 2002, IS&T/SPIE Electronic Imaging.

[7]  Sundaram Narayanan,et al.  Design of model-based interfaces for a real world information system , 2002, IEEE Trans. Syst. Man Cybern. Part A.

[8]  S S Stevens,et al.  HUMAN ENGINEERING FOR AN EFFECTIVE AIR-NAVIGATION AND TRAFFIC-CONTROL SYSTEM, AND APPENDIXES 1 THRU 3 , 1951 .

[9]  Jake K. Aggarwal,et al.  Multisensor Fusion for Computer Vision , 1993, NATO ASI Series.

[10]  James Reason,et al.  Cognitive Aids in Process Environments: Prostheses or Tools? , 1987, Int. J. Man Mach. Stud..

[11]  Alexander Toet,et al.  New false color mapping for image fusion , 1996 .

[12]  Thomas B. Sheridan.  Some musings on four ways humans couple: implications for systems design , 2002, IEEE Trans. Syst. Man Cybern. Part A.

[13]  Nadine B. Sarter,et al.  Supporting Decision Making and Action Selection under Time Pressure and Uncertainty: The Case of In-Flight Icing , 2001, Hum. Factors.

[14]  Susan G. Hutchins,et al.  Principles for Intelligent Decision Aiding , 1996 .

[15]  Balaram Das.  Cognition friendly interaction , 2000 .

[16]  Michael J. Young,et al.  Work-centered Support System technology: a new interface client technology for the battlespace infosphere , 2000, Proceedings of the IEEE 2000 National Aerospace and Electronics Conference. NAECON 2000. Engineering Tomorrow (Cat. No.00CH37093).

[17]  Kevin B. Bennett,et al.  Human Interaction with an "Intelligent" Machine , 1987, Int. J. Man Mach. Stud..

[18]  G. G. Kuperman.  Human system interface (HSI) issues in assisted target recognition (ASTR) , 1997, Proceedings of the IEEE 1997 National Aerospace and Electronics Conference. NAECON 1997.

[19]  Jacques Verly,et al.  Fusion of multisensor passive and active 3D imagery , 2001, SPIE Defense + Commercial Sensing.

[20]  David D. Woods,et al.  Commentary: Cognitive Engineering in Complex and Dynamic Worlds , 1988, Int. J. Man Mach. Stud..

[21]  Gianfranco Lamperti,et al.  AMMETH: a methodology for requirements analysis of advanced human-system interfaces , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[22]  C. W. Swonger,et al.  A prospectus for automatic target recognition , 1989 .

[23]  Christine M. Mitchell,et al.  GT-MSOCC: A Domain for Research on Human - Computer Interaction and Decision Aiding in Supervisory Control Systems , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[24]  David D. Woods,et al.  Cognitive Technologies: The Design of Joint Human-Machine Cognitive Systems , 1986, AI Mag..

[25]  Stacey D. Scott,et al.  Investigating human-computer optimization , 2002, CHI.

[26]  Michael D. McNeese,et al.  A Framework for Cognitive Field Studies , 1999 .

[27]  Tamio Arai,et al.  Fusion of range image and intensity image for 3D shape recognition , 1996, Proceedings of IEEE International Conference on Robotics and Automation.

[28]  A. Willsky,et al.  A multiresolution methodology for signal-level fusion and data assimilation with applications to remote sensing , 1997, Proc. IEEE.

[29]  Heath A. Ruff,et al.  Human-Integrated Supervisory Control of Uninhabited Combat Aerial Vehicles , 2000, J. Robotics Mechatronics.

[30]  John D. Lee,et al.  Augmenting the operator function model with cognitive operations: assessing the cognitive demands of technological innovation in ship navigation , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[31]  John A. Rushing,et al.  Intelligent processing techniques for sensor fusion , 1998, Defense, Security, and Sensing.

[32]  Allen M. Waxman,et al.  Color Night Vision: Opponent Processing in the Fusion of Visible and IR Imagery , 1997, Neural Networks.

[33]  Bonnie M. Muir,et al.  Trust Between Humans and Machines, and the Design of Decision Aids , 1987, Int. J. Man Mach. Stud..

[34]  Ann M. Bisantz,et al.  Assessment of operator trust in and utilization of automated decision-aids under different framing conditions , 2001 .

[35]  Sundaram Narayanan,et al.  Modeling real‐world information seeking in a corporate environment , 1999 .

[36]  Erik Hollnagel,et al.  Cognitive Systems Engineering: New wine in new bottles , 1999, Int. J. Hum. Comput. Stud..

[37]  Todd W. Kustra.  A Methodology to Develop Interactive Decision Support Systems for Complex United States Air Force Logistics Planning , 2000 .

[38]  Steven K. Rogers,et al.  Multisensor Data Fusion Of Laser Radar And Forward Looking Infrared (FLIR) For Target Segmentation And Enhancement , 1987, Other Conferences.

[39]  Tamar Peli,et al.  Feature-level sensor fusion , 1999, Defense, Security, and Sensing.

[40]  Christopher D. Wickens,et al.  A model for types and levels of human interaction with automation , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[41]  Raymond J. Martel.  HCI Architecting for System Reliability , 1996 .

[42]  Erik Hollnagel,et al.  Cognitive Systems Engineering: New Wine in New Bottles , 1983, Int. J. Man Mach. Stud..

[43]  Alexander Toet,et al.  Fusion of visible and thermal imagery improves situational awareness , 1997, Defense, Security, and Sensing.

[44]  Celestine A. Ntuen,et al.  Human Interaction with Complex Systems: Conceptual Principles and Design Practice , 1996 .

[45]  Alexander Toet,et al.  Perceptual evaluation of different image fusion schemes , 2003 .

[46]  Heath A. Ruff,et al.  Human Interaction with Levels of Automation and Decision-Aid Fidelity in the Supervisory Control of Multiple Simulated Unmanned Air Vehicles , 2002, Presence: Teleoperators & Virtual Environments.

[47]  Erik Hollnagel.  Information and Reasoning in Intelligent Decision Support Systems , 1987, Int. J. Man Mach. Stud..

[48]  Mary Fendley,et al.  A Knowledge-Based System To Model Human Supervisory Control In Dynamic Planning , 2004, Int. J. Uncertain. Fuzziness Knowl. Based Syst..

[49]  Shimon Ullman,et al.  Recognizing solid objects by alignment with an image , 1990, International Journal of Computer Vision.