Fusion artmap: neural networks for multi-sensor fusion and classification

Real-time neural networks for multi-sensor pattern classification are developed. These networks classify objects using information from multiple sensors of different modalities, views or scales. Each sensor is assigned an individual Adaptive Resonance Theory (ART) classifier whose compressed output serves as input to a global classifier. When global predictive errors occur, individual sensor inouts are relassified to improve system performance. Deciding which classifier to modify is known as the credit assignment problem. Solving this problem is a fundamental goal of the Fusion ARTMAP design. Two solutions to the credit assignment problem are investigated. Parallel match tracking assigns blame for global errors to the classifier with least confidence in its prediction by raising the vigilance of all ART modules in parallel. The Fusion ARTMAP network has a symmetric organization such that each channel can be dynamically configured to serve as either a data input or a teaching input. Simulations using an artificial Quadruped Mammal database show that Fusion ARTMAP requires 2/3 as many connections as does the standard method, vector concatenation. The second approach assigns credit by resetting all classifiers which mismatch the global recognition code. The resulting system integrates multiple fuzzy ARTMAP modules, slow learning, and ART-EMAP evidence accumulation techniques. Performance is illustrated by an extension of the circle-in-square benchmark. When compared to the concatenation approach, slow learning improves performance from 64% to 80% and reduces connectivity from 620 to 416 connections. The ART-EMAP method further improves performance to 85.6% without changing connectivity. Two multi-sensor pattern recognition applications are also presented. The first uses ART-EMAP to recognize 3-D objects sampled by multiple cameras. With four cameras, the network achieves 100% performance compared to 95% with the concatenation approach. A second application illustrates a case where the concatenation approach may be a better choice than a modular network. When categorizing multispectral satellite images, a single-channel fuzzy ARTMAP neural network achieves near optimum (89%) performance in comparison to thirteen other classification methods while achieving a 6:1 code compression ratio. The choice between a modular approach to multi-sensor fusion and vector concatenation is discussed.