Multisensor & spectral image fusion & mining: from neural systems to applications

We have continued development of a system for multisensor image fusion and interactive mining based on neural models of color vision processing, learning and pattern recognition. We pioneered this work while at MIT Lincoln Laboratory, initially for color fused night vision (low-light visible and uncooled thermal imagery) and later extended it to multispectral IR and 3D ladder. We also developed a proof-of-concept system for EO, IR, SAR fusion and mining. Over the last year we have generalized this approach and developed a user-friendly system integrated into a COTS exploitation environment known as ERDAS Imagine. In this paper, we have summarized the approach and the neural networks used, and demonstrate fusion and interactive mining (i.e., target learning and search) of low-light visible/SWIR/MWIR/LWIR night imagery, and IKONOS multispectral and high-resolution panchromatic imagery. In addition, we had demonstrated how target learning and search can be enabled over extended operating conditions by allowing training over multiple scenes. This has been illustrated for the detection of small boats in coastal waters using fused visible/MWIR/LWIR imagery.

[1]  Jacques Verly,et al.  A Prototype System for 3D Color Fusion and Mining of Multisensor/Spectral Imagery , 2001 .

[2]  Stephen Grossberg,et al.  Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps , 1992, IEEE Trans. Neural Networks.

[3]  Nikos K Logothetis,et al.  The color-opponent and broad-band channels of the primate visual system , 1990, Trends in Neurosciences.

[4]  G.A. Carpenter,et al.  Information fusion for image analysis: geospatial foundations for higher-level fusion , 2002, Proceedings of the Fifth International Conference on Information Fusion. FUSION 2002. (IEEE Cat.No.02EX5997).

[5]  Leo Maurice Hurvich,et al.  Color vision , 1981 .

[6]  D. Levine.  Introduction to Neural and Cognitive Modeling , 2018 .

[7]  Jacques Verly,et al.  Fusion of Visible, Infrared and 3D LADAR Imagery , 2001 .

[8]  S Grossberg,et al.  Neural dynamics of perceptual grouping: Textures, boundaries, and emergent segmentations , 1985, Perception & psychophysics.

[9]  Alan N. Gove,et al.  Neural processing of targets in visible, multispectral IR and SAR imagery , 1995, Neural Networks.

[10]  Alexander Toet.  Detection of dim point targets in cluttered maritime backgrounds through multisensor image fusion , 2002, SPIE Defense + Commercial Sensing.

[11]  J. Mollon.  Color vision. , 1982, Annual review of psychology.

[12]  E. Newman,et al.  The infrared 'vision' of snakes , 1982 .

[13]  Stephen Grossberg,et al.  Fuzzy ART: Fast stable learning and categorization of analog patterns by an adaptive resonance system , 1991, Neural Networks.

[14]  Allen M. Waxman,et al.  Fusion of 2- /3- /4-sensor imagery for visualization, target learning, and search , 2000, Defense, Security, and Sensing.

[15]  Ennio Mingolla,et al.  Neural dynamics of perceptual grouping: Textures, boundaries, and emergent segmentations , 1985 .

[16]  Stephen Grossberg,et al.  Nonlinear neural networks: Principles, mechanisms, and architectures , 1988, Neural Networks.

[17]  J. Daugman.  Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[18]  Jacques Verly,et al.  Multi-sensor 3D image fusion and interactive search , 2000, Proceedings of the Third International Conference on Information Fusion.

[19]  Allen M. Waxman,et al.  Fused multi-sensor image mining for feature foundation data , 2000, Proceedings of the Third International Conference on Information Fusion.

[20]  Fang Liu,et al.  Multi- sensor Image Fusion for Efficient Color 3D Site Visualization and Rapid Analysis Utilizing Search Acceleration Algorithms , 2000 .