Augmenting full colour-fused multi-band night vision imagery with synthetic imagery in real-time

We present the design and first field trial results of an all-day all-weather enhanced and synthetic-fused multi-band colour night vision surveillance and observation system. The system augments a fused and dynamic three-band natural-colour night vision image with synthetic 3D imagery in real-time. The night vision sensor suite consists of three cameras, sensitive in, respectively, the visual (400–700 nm), the near-infrared (NIR, 700–1000 nm) and the long-wave infrared (LWIR, 8–14 µm) bands of the electromagnetic spectrum. The optical axes of the three cameras are aligned. Image quality of the fused sensor signals is enhanced in real-time through dynamic noise reduction, super resolution and local adaptive contrast enhancement. The quality of the LWIR image is enhanced through scene-based non-uniformity correction. The visual and NIR signals are used to represent the fused multi-band night vision image in natural daytime colours, using the Colour-the-Night colour remapping technique. Colour remapping can also be deployed to enhance the visibility of thermal targets that are camouflaged in the visual and NIR range of the spectrum. The dynamic false-colour night-time images are augmented with corresponding synthetic 3D scene views, generated in real-time using a geometric 3D scene model in combination with position and orientation information supplied by the Global Positioning System and inertial sensors of the system. Initial field trials show that this system provides enhanced situational information in various low-visibility conditions.

[1]  Allen M. Waxman,et al.  Fusion of multi-sensor imagery for night vision: color visualization, target learning and search , 2000, Proceedings of the Third International Conference on Information Fusion.

[2]  Paul Milgram,et al.  Viewpoint Animation With a Dynamic Tether for Supporting Navigation in a Virtual Environment , 2009, Hum. Factors.

[3]  Klamer Schutte,et al.  Signal conditioning algorithms for enhanced tactical sensor imagery , 2003, SPIE Defense + Commercial Sensing.

[4]  Ke Wang,et al.  Applying daytime colors to night-time imagery with an efficient color transfer method , 2007, SPIE Defense + Commercial Sensing.

[5]  Klamer Schutte.  Multiscale adaptive gain control of IR images , 1997, Defense, Security, and Sensing.

[6]  Gang Liu,et al.  Color fusion of SAR and FLIR images using a natural color transfer technique , 2005 .

[7]  Olivier R. Joubert,et al.  How long to get to the “gist” of real-world natural scenes? , 2005 .

[8]  J. K. Kraker,et al.  Re-usable behavior specifications for tactical doctrine , 2009 .

[9]  Harvey S. Smallman,et al.  The Use of 2D and 3D Displays for Shape-Understanding versus Relative-Position Tasks , 2001, Hum. Factors.

[10]  Vassilis Tsagaris,et al.  Fusion of visible and infrared imagery for night color vision , 2005, Displays.

[11]  Weiqi Jin,et al.  Color fusion schemes for low-light CCD and infrared images of different properties , 2002, SPIE/COS Photonics Asia.

[12]  Christopher D. Wickens,et al.  Frames of Reference for the Display of Battlefield Information: Judgment-Display Dependencies , 2000, Hum. Factors.

[13]  Yufeng Zheng,et al.  Coloring night-vision imagery with statistical properties of natural colors by using image segmentation and histogram matching , 2005, IS&T/SPIE Electronic Imaging.

[14]  Alexander Toet,et al.  Natural colour mapping for multiband nightvision imagery , 2003, Inf. Fusion.

[15]  Alexander Toet,et al.  Method for applying daytime colors to nighttime imagery in realtime , 2008, SPIE Defense + Commercial Sensing.

[16]  Michel-Ange Amorim,et al.  Cognitive Repositioning inside a Desktop VE: The Constraints Introduced by First-versus Third-Person Imagery and Mental Representation Richness , 2000, Presence: Teleoperators & Virtual Environments.

[17]  Alan N. Gove,et al.  Color night vision: fusion of intensified visible and thermal IR imagery , 1995, Defense, Security, and Sensing.

[18]  D. Ruderman,et al.  Statistics of cone responses to natural images: implications for visual coding , 1998 .

[19]  Joseph P. Racamato,et al.  Solid-State Color Night Vision: Fusion of Low-Light Visible and Thermal Infrared Imagery , 1998 .

[20]  Melvin R. Kruer,et al.  Real-time color fusion of E/O sensors with PC-based COTS hardware , 2000, Defense, Security, and Sensing.

[21]  Ryota Kanai,et al.  Recognising the forest, but not the trees: An effect of colour on scene perception and recognition , 2008, Consciousness and Cognition.

[22]  Zhanfeng Yue,et al.  Real-time EO/IR sensor fusion on a portable computer and head-mounted display , 2007, SPIE Defense + Commercial Sensing.

[23]  Christopher D. Wickens,et al.  Two- and Three-Dimensional Displays for Aviation: A Theoretical and Empirical Comparison , 1993 .

[24]  Alexander Toet,et al.  Fast natural color mapping for night-time imagery , 2010, Inf. Fusion.

[25]  A. Oliva,et al.  Diagnostic colours contribute to the early stages of scene categorization: Behavioural and neurophysiological evidence , 2005 .

[26]  Justin G. Hollands,et al.  Viewpoint Tethering in Complex Terrain Navigation and Awareness , 2005 .

[27]  J. G. Hollands,et al.  Engineering Psychology and Human Performance , 1984 .

[28]  Junsheng Shi,et al.  Objective evaluation of color fusion of visual and IR imagery by measuring image contrast , 2005, SPIE/COS Photonics Asia.

[29]  W. K. Krebs,et al.  Scene Recognition with Infrared, Low-Light, and Sensor-Fused Imagery , 1999 .

[30]  Wendell R. Watkins,et al.  Targets and backgrounds VI : characterization, visualization, and the detection process : 24-26 April 2000, Orlando, USA , 2000 .

[31]  Alexander Toet,et al.  Color the night: applying daytime colors to nighttime imagery , 2003, SPIE Defense + Commercial Sensing.

[32]  Weiqi Jin,et al.  Real-time color transfer system for low-light level visible and infrared images in YUV color space , 2007, SPIE Defense + Commercial Sensing.