Multisensor fusion and integration: approaches, applications, and future research directions

Multisensor fusion and integration is a rapidly evolving research area and requires interdisciplinary knowledge in control theory, signal processing, artificial intelligence, probability and statistics, etc. The advantages gained through the use of redundant, complementary, or more timely information in a system can provide more reliable and accurate information. This paper provides an overview of current sensor technologies and describes the paradigm of multisensor fusion and integration as well as fusion techniques at different fusion levels. Applications of multisensor fusion in robotics, biomedical system, equipment monitoring, remote sensing, and transportation system are also discussed. Finally, future research directions of multisensor fusion technology including microsensors, smart sensors, and adaptive fusion techniques are presented.

[1]  R. G. Reynolds.  Robust estimation of covariance matrices , 1990 .

[2]  Moshe Kam,et al.  Sensor Fusion for Mobile Robot Navigation , 1997, Proc. IEEE.

[3]  F. Azuaje,et al.  Improving clinical decision support through case-based data fusion , 1999, IEEE Transactions on Biomedical Engineering.

[4]  V. V. S. Sarma,et al.  Multisensor data fusion and decision support for airborne target identification , 1991, IEEE Trans. Syst. Man Cybern..

[5]  Ren C. Luo,et al.  Dynamic multi-sensor data fusion system for intelligent robots , 1988, IEEE J. Robotics Autom..

[6]  D. Fussel,et al.  Detection and isolation of sensor faults on nonlinear processes based on local linear models , 1997, Proceedings of the 1997 American Control Conference (Cat. No.97CH36041).

[7]  Pramod K. Varshney.  Multisensor data fusion , 1997 .

[8]  Mieczyslaw M. Kokar,et al.  Multisensor integration in the tracking of landing aircraft , 1994, Proceedings of 1994 IEEE International Conference on MFI '94. Multisensor Fusion and Integration for Intelligent Systems.

[9]  Bruno Siciliano,et al.  Control in robotics: open problems and future directions , 1998, Proceedings of the 1998 IEEE International Conference on Control Applications (Cat. No.98CH36104).

[10]  Lawrence A. Klein,et al.  Sensor and Data Fusion Concepts and Applications , 1993 .

[11]  V. Cappellini,et al.  An application of data fusion to landcover classification of remote sensed imagery: a neural network approach , 1994, Proceedings of 1994 IEEE International Conference on MFI '94. Multisensor Fusion and Integration for Intelligent Systems.

[12]  P. G. O'Reilly.  Local sensor fault detection using Bayesian decision theory , 1998 .

[13]  B. Solaiman,et al.  Information fusion, application to data and model fusion for ultrasound image segmentation , 1999, IEEE Transactions on Biomedical Engineering.

[14]  Ren C. Luo,et al.  Multilayered fuzzy behavior fusion for real-time reactive control of systems with multiple sensors , 1996, IEEE Trans. Ind. Electron..

[15]  P. Valin,et al.  Position and attribute fusion of radar, ESM, IFF and Datalink for AAW missions of the Canadian Patrol Frigate , 1996, 1996 IEEE/SICE/RSJ International Conference on Multisensor Fusion and Integration for Intelligent Systems (Cat. No.96TH8242).

[16]  Theresa M. Mullin,et al.  Cognitive biases and time stress in team decision making , 1997, IEEE Trans. Syst. Man Cybern. Part A.

[17]  Ren C. Luo,et al.  Target tracking by grey prediction theory and look-ahead fuzzy logic control , 1999, Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C).

[18]  Victor R. Lesser,et al.  Cooperative Multiagent Systems: A Personal View of the State of the Art , 1999, IEEE Trans. Knowl. Data Eng..

[19]  Stan Matwin,et al.  Intelligent data fusion for environmental monitoring , 1995, 1995 International Geoscience and Remote Sensing Symposium, IGARSS '95. Quantitative Remote Sensing for Science and Applications.

[20]  K. H. Kim,et al.  Review of multisensor data fusion architectures and techniques , 1993, Proceedings of 8th IEEE International Symposium on Intelligent Control.

[21]  G. Passariello,et al.  Multisensor fusion for atrial and ventricular activity detection in coronary care monitoring , 1999, IEEE Transactions on Biomedical Engineering.

[22]  Gaurav S. Sukhatme,et al.  Sensor fault detection and identification in a mobile robot , 1998, Proceedings. 1998 IEEE/RSJ International Conference on Intelligent Robots and Systems. Innovations in Theory, Practice and Applications (Cat. No.98CH36190).

[23]  James A. Stover,et al.  A fuzzy-logic architecture for autonomous multisensor data fusion , 1996, IEEE Trans. Ind. Electron..

[24]  T. Takenaka,et al.  The development of Honda humanoid robot , 1998, Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146).

[25]  Ren C. Luo,et al.  Multisensor integration and fusion in intelligent systems , 1989, IEEE Trans. Syst. Man Cybern..

[26]  P. Bélanger.  Estimation of noise covariance matrices for a linear time-varying stochastic process , 1972, Autom..

[27]  R. R. Jr. Carson,et al.  Fusion of IFF and radar data , 1997, 16th DASC. AIAA/IEEE Digital Avionics Systems Conference. Reflections to the Future. Proceedings.

[28]  Hugh F. Durrant-Whyte,et al.  A Fully Decentralized Multi-Sensor System For Tracking and Surveillance , 1993, Int. J. Robotics Res..

[29]  R. Mehra.  On the identification of variances and adaptive Kalman filtering , 1970 .

[30]  Guy Littlefair,et al.  Fusion of integrated multisensor data for tool wear monitoring , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[31]  D. L. Hall,et al.  Mathematical Techniques in Multisensor Data Fusion , 1992 .

[32]  James Llinas,et al.  An introduction to multisensor data fusion , 1997, Proc. IEEE.

[33]  Felix Schmid,et al.  Fault detection and isolation in multisensor train navigation systems , 1998 .

[34]  Ren C. Luo,et al.  A tutorial on multisensor integration and fusion , 1990, [Proceedings] IECON '90: 16th Annual Conference of IEEE Industrial Electronics Society.

[35]  Michael J. Grimble,et al.  Integrated control and fault diagnosis design: a polynomial approach , 1996 .

[36]  Ren C. Luo,et al.  Future trends in multisensor integration and fusion , 1994, Proceedings of 1994 IEEE International Symposium on Industrial Electronics (ISIE'94).

[37]  Chingiz Hajiyev,et al.  Fault detection in flight control systems via innovation sequence of Kalman filter , 1998 .

[38]  Stelios C. A. Thomopoulos.  Sensor integration and data fusion , 1990, J. Field Robotics.

[39]  D. J. Peters,et al.  Fusion of IFF and radar data , 1996, Proceeding of 1st Australian Data Fusion Symposium.

[40]  T. I. Liu,et al.  On-line sensing of drill wear using neural network approach , 1993, IEEE International Conference on Neural Networks.

[41]  Alain Hillion,et al.  An information fusion method for multispectral image classification postprocessing , 1998, IEEE Trans. Geosci. Remote. Sens..

[42]  Felix Schmid,et al.  Application of sensor fusion to railway systems , 1996, 1996 IEEE/SICE/RSJ International Conference on Multisensor Fusion and Integration for Intelligent Systems (Cat. No.96TH8242).

[43]  Lang Hong.  Adaptive data fusion , 1991, Conference Proceedings 1991 IEEE International Conference on Systems, Man, and Cybernetics.

[44]  Ren C. Luo,et al.  Multilevel Multiagent Based Team Decision Fusion for Autonomous Tracking System , 1999 .

[45]  James Llinas,et al.  Multisensor Data Fusion , 1990 .

[46]  T. W. Long,et al.  Sensor fusion and failure detection using virtual sensors , 1999, Proceedings of the 1999 American Control Conference (Cat. No. 99CH36251).

[47]  P. Belanger,et al.  Identification of optimum filter steady-state gain for systems with unknown noise covariances , 1973 .

[48]  Ren C. Luo,et al.  A generalized look-ahead method for adaptive multiple sequential data fusion and decision making , 1999, Proceedings. 1999 IEEE/SICE/RSJ. International Conference on Multisensor Fusion and Integration for Intelligent Systems. MFI'99 (Cat. No.99TH8480).

[49]  K. Kobayashi,et al.  Accurate navigation via differential GPS and vehicle local sensors , 1994, Proceedings of 1994 IEEE International Conference on MFI '94. Multisensor Fusion and Integration for Intelligent Systems.

[50]  M. Fernandez,et al.  A failure detection and isolation algorithm for a decentralised multisensor system , 1994, Proceedings of 1994 IEEE International Conference on MFI '94. Multisensor Fusion and Integration for Intelligent Systems.

[51]  Chee-Yee Chong,et al.  Fusion technologies for drug interdiction , 1994, Proceedings of 1994 IEEE International Conference on MFI '94. Multisensor Fusion and Integration for Intelligent Systems.

[52]  Martin Gade,et al.  Monitoring algae blooms in the Baltic Sea: a multi sensor approach , 1999, Oceans '99. MTS/IEEE. Riding the Crest into the 21st Century. Conference and Exhibition. Conference Proceedings (IEEE Cat. No.99CH37008).

[53]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[54]  Nikolaos Papanikolopoulos,et al.  Vision sensing for intelligent vehicle and highway systems , 1994, Proceedings of 1994 IEEE International Conference on MFI '94. Multisensor Fusion and Integration for Intelligent Systems.

[55]  F. Busson,et al.  Multiple sensor fault detection in heat exchanger systems , 1998, Proceedings of the 1998 IEEE International Conference on Control Applications (Cat. No.98CH36104).

[56]  E. T. Powner,et al.  Intelligent sensors: the way forward to decentralised intelligence and control , 1998 .

[57]  Sumit Roy,et al.  Decentralized structures for parallel Kalman filtering , 1988 .

[58]  Cheng-Jen Lin,et al.  An intelligent sensor fusion system for tool monitoring on a machining centre , 1996 .

[59]  Belur V. Dasarathy,et al.  Decision fusion , 1994 .

[60]  K. Partington,et al.  Data fusion for use of passive microwave data in operational sea-ice monitoring , 1999, IEEE 1999 International Geoscience and Remote Sensing Symposium. IGARSS'99 (Cat. No.99CH36293).

[61]  Ren C. Luo,et al.  Sensor technologies and microsensor issues for mechatronics systems , 1996 .

[62]  G. Passariello,et al.  Intelligent patient monitoring and management systems: a review , 1993, IEEE Engineering in Medicine and Biology Magazine.

[63]  MengChu Zhou,et al.  Real-time tool wear identification using sensor integration with neural network , 1994, Proceedings of 1994 33rd IEEE Conference on Decision and Control.

[64]  Junichi Sato,et al.  Automatic operation of a combined harvester in a rice field , 1996, 1996 IEEE/SICE/RSJ International Conference on Multisensor Fusion and Integration for Intelligent Systems (Cat. No.96TH8242).

[65]  R. Krishnamurthy,et al.  Monitoring of tool status using intelligent acoustic emission sensing and decision based neural network , 1995, Proceedings of IEEE/IAS International Conference on Industrial Automation and Control.

[66]  Arthur Filippidis,et al.  Multisensor data fusion for surface land-mine detection , 2000, IEEE Trans. Syst. Man Cybern. Part C.

[67]  B. R. Bracio,et al.  Sensor fusion in biomedical systems , 1997, Proceedings of the 19th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 'Magnificent Milestones and Emerging Opportunities in Medical Engineering' (Cat. No.97CH36136).

[68]  Arthur C. Sanderson,et al.  Minimal representation multisensor fusion using differential evolution , 1999, IEEE Trans. Syst. Man Cybern. Part A.

[69]  Belur V. Dasarathy,et al.  Sensor fusion potential exploitation-innovative architectures and illustrative applications , 1997, Proc. IEEE.

[70]  Bassel Solaiman,et al.  Multisensor data fusion using fuzzy concepts: application to land-cover classification using ERS-1/JERS-1 SAR composites , 1999, IEEE Trans. Geosci. Remote. Sens..

[71]  D. L. Hall,et al.  A new approach to the challenge of machinery prognostics , 1995 .