Physics-based integration of multiple sensing modalities for scene interpretation

The fusion of multiple imaging modalities offers many advantages over the analysis, separately, of the individual sensory modalities. In this paper we present a unique approach to the integrated analysis of disparate sources of imagery for object recognition. The approach is based on physics-based modeling of the image generation mechanisms. Such models make possible features that are physically meaningful and have an improved capacity to differentiate between multiple classes of objects. We illustrate the use of physics-based approach to develop multisensory vision systems for different object recognition application domains. The paper discusses the integration of different suites of sensors, the integration of image-derived information with model-derived information and the physics-based simulation of multisensory imagery.

[1]  Glenn Healey.  Using color to segment images of 3-D scenes , 1991, Defense, Security, and Sensing.

[2]  Jake K. Aggarwal,et al.  Volume/surface octrees for the representation of three-dimensional objects , 1986, Comput. Vis. Graph. Image Process..

[3]  Jake K. Aggarwal,et al.  Thermal and visual information fusion for outdoor scene perception , 1988, Proceedings. 1988 IEEE International Conference on Robotics and Automation.

[4]  N. Nandhakumar,et al.  Quasi-invariant behavior of thermophysical features for interpretation of multisensor imagery , 1996 .

[5]  Peter H. Hartline.  The Optic Tectum of Reptiles: Neurophysiological Studies , 1984 .

[6]  N. Nandhakumar,et al.  Thermophysical affine invariants from IR imagery for object recognition , 1995, Proceedings of the Workshop on Physics-Based Modeling in Computer Vision.

[7]  Jake K. Aggarwal,et al.  Integrated modelling of thermal and visual image generation , 1989, Proceedings CVPR '89: IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[8]  A. Hurlbert.  Physics-based vision: Principles and practice, color, edited by Lawrence B. Wolff, Steven A. Shafer, and Glenn E. Healey, A. K. Peters, Ltd., Wellesley, MA, 1992, hardbound, $59.95 , 1993 .

[9]  E. Newman,et al.  The infrared 'vision' of snakes , 1982 .

[10]  Jake K. Aggarwal,et al.  Multisensory Computer Vision , 1992, Adv. Comput..

[11]  Lawrence B. Wolff,et al.  Scene understanding from propagation and consistency of polarization-based constraints , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[12]  E. Kennaugh,et al.  Transient and impulse response approximations , 1965 .

[13]  Deepak Kapur,et al.  Using Elimination Methods to Compute Thermophysical Algebraic Invariants from Infrared Imagery , 1996, AAAI/IAAI, Vol. 2.

[14]  D. Hilbert,et al.  Theory of algebraic invariants , 1993 .

[15]  N. Nandhakumar,et al.  Robust thermophysics-based interpretation of radiometrically uncalibrated IR images for ATR and site change detection , 1997, IEEE Trans. Image Process..

[16]  David A. Forsyth,et al.  Invariant Descriptors for 3D Object Recognition and Pose , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  N. Nandhakumar,et al.  MULTISENSOR INTEGRATION - EXPERIMENTS IN INTEGRATING THERMAL AND VISUAL SENSORS. , 1987 .

[18]  A. Ishimaru,et al.  Application of the composite roughness model to high‐frequency bottom backscattering , 1986 .

[19]  M S Loop,et al.  Merging of modalities in the optic tectum: infrared and visual integration in rattlesnakes. , 1978, Science.

[20]  N. Nandhakumar,et al.  Features for detecting obscured objects in ultra-wideband (UWB) SAR imagery using a phenomenological approach , 1996, Pattern Recognit..

[21]  Glenn Healey,et al.  Using illumination invariant descriptors for recognition , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[22]  N. Nandhakumar,et al.  A physics-based approach for detecting man-made objects in ultra-wideband SAR imagery , 1995, Proceedings of the Workshop on Physics-Based Modeling in Computer Vision.

[23]  Lawrence B. Wolff,et al.  Polarization-Based Material Classification from Specular Reflection , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Jake K. Aggarwal,et al.  Integrated Analysis of Thermal and Visual Images for Scene Interpretation , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Takeo Kanade,et al.  Image Segmentation And Reflection Analysis Through Color , 1988, Defense, Security, and Sensing.

[26]  N. Nandhakumar,et al.  Modeling nonhomogeneous 3-D objects for thermal and visual image synthesis , 1991, Defense, Security, and Sensing.

[27]  G. Healey,et al.  Using Illumination Invariant Color Histogram Descriptors for Recognit ion , 1994 .

[28]  Y. N. Lakshman,et al.  Computing invariants using elimination methods , 1995, Proceedings of International Symposium on Computer Vision - ISCV.

[29]  W. B. Sawyer,et al.  High-frequency acoustic backscattering from a coarse shell ocean bottom , 1989 .

[30]  Rui J. P. de Figueiredo,et al.  Fusion of radar and optical sensors for space robotic vision , 1988, Proceedings. 1988 IEEE International Conference on Robotics and Automation.