Visible/Infrared Combined 3D Reconstruction Scheme Based on Nonrigid Registration of Multi-Modality Images With Mixed Features

When traditional 3D reconstruction techniques were used to reconstruct a scene with hidden and disguised heat source targets, the reconstructed scene could not contain these targets; thus, we could not recognize them. The disadvantages of such problems are particularly acute in the military and remote-sensing areas. For this application problem, the authors proposed a visible/infrared combined 3D reconstruction scheme. The 3D scene containing thermal radiation information could be reconstructed by fusing the data from RGB optical and infrared images combined with computer vision passive optical 3D scene reconstruction technique. Meanwhile, the authors proposed the nonrigid registration of multi-modality images with mixed features to solve the problem that the registration algorithm, which is widely used in traditional passive reconstruction technology, cannot accurately match visible and infrared images. Registration accuracy was improved by approximately 40%, experimental results showed that the visible/infrared combined 3D reconstruction scenes retain the visual reality of the traditional 3D reconstruction, and the hidden targets are highlighted in the scenes, which is conducive to the detection and recognition of interesting targets.

[1]  Zhuowen Tu,et al.  Regularized vector field learning with sparse approximation for mismatch removal , 2013, Pattern Recognit..

[2]  George Vosselman,et al.  Knowledge based reconstruction of building models from terrestrial laser scanning data , 2009 .

[3]  Alan L. Yuille,et al.  Semi-Supervised Sparse Representation Based Classification for Face Recognition With Insufficient Labeled Samples , 2016, IEEE Transactions on Image Processing.

[4]  Olivier D. Faugeras,et al.  What can be seen in three dimensions with an uncalibrated stereo rig , 1992, ECCV.

[5]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[6]  A. U.S.,et al.  Recovering Surface Shape and Orientation from Texture , 2002 .

[7]  Junjun Jiang,et al.  Guided Locality Preserving Feature Matching for Remote Sensing Image Registration , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[8]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[9]  Junjun Jiang,et al.  Locality Preserving Matching , 2018, International Journal of Computer Vision.

[10]  Michael Breuß,et al.  Perspective Shape from Shading with Non-Lambertian Reflectance , 2008, DAGM-Symposium.

[11]  Junjun Jiang,et al.  FusionGAN: A generative adversarial network for infrared and visible image fusion , 2019, Inf. Fusion.

[12]  Quan Z. Sheng,et al.  Nonrigid Point Set Registration With Robust Transformation Learning Under Manifold Regularization , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[13]  Tao Lu,et al.  Multi-Memory Convolutional Neural Network for Video Super-Resolution , 2019, IEEE Transactions on Image Processing.

[14]  Alan L. Yuille,et al.  Non-Rigid Point Set Registration by Preserving Global and Local Structures , 2016, IEEE Transactions on Image Processing.

[15]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Bahram Javidi,et al.  Recent Advances in the Capture and Display of Macroscopic and Microscopic 3-D Scenes by Integral Imaging , 2017, Proceedings of the IEEE.

[17]  Alessandro Marro,et al.  Three-Dimensional Printing and Medical Imaging: A Review of the Methods and Applications. , 2016, Current problems in diagnostic radiology.

[18]  David A. Forsyth,et al.  Shape from Texture without Boundaries , 2002, International Journal of Computer Vision.

[19]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[20]  Richard I. Hartley,et al.  Kruppa's Equations Derived from the Fundamental Matrix , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Filiberto Pla,et al.  Multidimensional Optical Sensing and Imaging System (MOSIS): From Macroscales to Microscales , 2017, Proceedings of the IEEE.

[22]  Judith Sausse,et al.  Advances in 3-D infrared remote sensing gas monitoring. Application to an urban atmospheric environment , 2016 .

[23]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[24]  Junjun Jiang,et al.  Robust Feature Matching for Remote Sensing Image Registration via Locally Linear Transforming , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[25]  Jan-Michael Frahm,et al.  Detailed Real-Time Urban 3D Reconstruction from Video , 2007, International Journal of Computer Vision.

[26]  Takeo Kanade,et al.  Shape and motion from image streams under orthography: a factorization method , 1992, International Journal of Computer Vision.

[27]  Sim Heng Ong,et al.  A robust global and local mixture distance based non-rigid point set registration , 2015, Pattern Recognit..

[28]  Bahram Javidi,et al.  Advances in three-dimensional integral imaging: sensing, display, and applications [Invited]. , 2013, Applied optics.

[29]  Yan Ke,et al.  PCA-SIFT: a more distinctive representation for local image descriptors , 2004, CVPR 2004.

[30]  Zhuowen Tu,et al.  Robust Point Matching via Vector Field Consensus , 2014, IEEE Transactions on Image Processing.

[31]  Jiayi Ma,et al.  Infrared and visible image fusion methods and applications: A survey , 2018, Inf. Fusion.

[32]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera , 2011, UIST.

[33]  Charles A. Micchelli,et al.  On Learning Vector-Valued Functions , 2005, Neural Computation.

[34]  Yansheng Li,et al.  Feature guided Gaussian mixture model with semi-supervised EM and local geometric constraint for retinal image registration , 2017, Inf. Sci..

[35]  Ji Zhao,et al.  Non-rigid visible and infrared face registration via regularized Gaussian fields criterion , 2015, Pattern Recognit..

[36]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.