User performance improvement via multimodal interface fusion augmentation

The functional role of a data fusion system is to provide timely and accurate information to the user. At the lowest level, the fusion system filters relevant incoming data and at the highest level, the fusion system presents actionable information to the user. In many cases, the fusion system design is dependent on user capabilities and knowledge. Thus, only presenting the fused data might not enhance operator perjormance, while presenting all the multimodal information might overload the operators cognitive capabilities. In this paper, we use the JDL-User model to design an interface which utilizes image fusion and multimodal characteristics to enhance user performance. The design allows for the user to passively monitor and actively interact with the fusion system with controls for image overlays, queries and audio warnings to support unattended areas, and an eye tracker to improve attention. Implemented in JA VA, we present a tracking display design that utilizes a multimodal interface to (I) take advantage of the human's ability to fuse muliisensory information, (2) facilitate situational awareness, and (3) enhance the users cognitive task analysis.

[1]  G.A. Carpenter,et al.  Information fusion for image analysis: geospatial foundations for higher-level fusion , 2002, Proceedings of the Fifth International Conference on Information Fusion. FUSION 2002. (IEEE Cat.No.02EX5997).

[2]  Karl T. Ulrich,et al.  Product Design and Development , 1995 .

[3]  Martin Buss,et al.  A systems theoretical model for human perception in multimodal presence systems , 2001 .

[4]  Blake Hannaford,et al.  Control law design for haptic interfaces to virtual reality , 2002, IEEE Trans. Control. Syst. Technol..

[5]  K. J. Vicente,et al.  Cognitive Work Analysis: Toward Safe, Productive, and Healthy Computer-Based Work , 1999 .

[6]  Colin Ware,et al.  Information Visualization: Perception for Design , 2000 .

[7]  Austin Henderson,et al.  Interaction design: beyond human-computer interaction , 2002, UBIQ.

[8]  Charles E. Billings,et al.  Aviation Automation: The Search for A Human-centered Approach , 1996 .

[9]  Kathryn B. Laskey,et al.  Multi-entity Bayesian networks for situation assessment , 2002, Proceedings of the Fifth International Conference on Information Fusion. FUSION 2002. (IEEE Cat.No.02EX5997).

[10]  Imad H. Elhajj,et al.  Haptic information in Internet-based teleoperation , 2001 .

[11]  W Ercoline.  The good, the bad, and the ugly of head-up displays. , 2000, IEEE engineering in medicine and biology magazine : the quarterly magazine of the Engineering in Medicine & Biology Society.

[13]  Blake Hannaford,et al.  A probabilistic representation of human workspace for use in the design of human interface mechanisms , 2001 .

[14]  Jens Rasmussen,et al.  Information Processing and Human-Machine Interaction , 1986 .

[15]  Erik Blasch,et al.  JDL level 5 fusion model: user refinement issues and applications in group tracking , 2002, SPIE Defense + Commercial Sensing.

[16]  Erland Jungert,et al.  Towards a query assisted tool for situation assessment , 2002, Proceedings of the Fifth International Conference on Information Fusion. FUSION 2002. (IEEE Cat.No.02EX5997).

[17]  Mario Aguilar,et al.  Fusion of multi-modality volumetric medical imagery , 2002, Proceedings of the Fifth International Conference on Information Fusion. FUSION 2002. (IEEE Cat.No.02EX5997).

[18]  Yvonne Rogers,et al.  Interaction Design: Beyond Human-Computer Interaction , 2002 .

[19]  Jean Roy,et al.  Hierarchical adaptation scheme for multiagent data fusion and resource management in situation analysis , 2001, SPIE Defense + Commercial Sensing.

[20]  C. Anken,et al.  Intelligent systems technology for higher level fusion , 2002, Proceedings of the Fifth International Conference on Information Fusion. FUSION 2002. (IEEE Cat.No.02EX5997).

[21]  J. G. Hollands,et al.  Engineering Psychology and Human Performance , 1984 .

[22]  J. Roy.  From Data Fusion to Situation Analysis , 2001 .

[23]  J. Masters.  Structured Knowledge Source Integration and its applications to information fusion , 2002, Proceedings of the Fifth International Conference on Information Fusion. FUSION 2002. (IEEE Cat.No.02EX5997).

[24]  Yunmo Chung,et al.  ASIC design of color control driver for LCOS (liquid crystal on silicon) micro display , 2001, IEEE Trans. Consumer Electron..

[25]  E. Blasch,et al.  Assembling a distributed fused information-based human-computer cognitive decision making tool , 2000, IEEE Aerospace and Electronic Systems Magazine.