Combining Multi-Frame Images for Enhancement Using Self-Delaying Dynamic Networks

This paper presents the use of a newly created network structure known as a Self-Delaying Dynamic Network (SDN). The SDNs were created to process data which varies with time. They feature an inbuilt timing structure which allows them to pass different items of data at different rates. This allows the network to store data from one time period until it can be used with related data in a later time period. These SDNs are non-recurrent temporal neural networks which store input data and they feature dynamic logic based connections between layers. An application is shown to create a high resolution image from a set of time stepped input frames. Several low resolution images and one high resolution image of a number of scenes were presented to the SDN during training by a Genetic Algorithm (GA). The trained SDN was then used to enhance a number of unseen noisy image sets. The images formed by the SDN are superior in several ways to the images produced using bi-cubic interpolation.

[1]  Jean-Cédric Chappelier,et al.  RST : a Spatiotemporal Neural Network , 2007 .

[2]  Nirmal K. Bose,et al.  Recursive reconstruction of high resolution image from noisy undersampled multiframes , 1990, IEEE Trans. Acoust. Speech Signal Process..

[3]  Xavier Otazu,et al.  Multiresolution-based image fusion with additive wavelet decomposition , 1999, IEEE Trans. Geosci. Remote. Sens..

[4]  Gonzalo Pajares Martinsanz,et al.  A wavelet-based image fusion tutorial , 2004 .

[5]  Robert L. Stevenson,et al.  Extraction of high-resolution frames from video sequences , 1996, IEEE Trans. Image Process..

[6]  Jean-Cédric Chappelier,et al.  Time in neural networks , 1994, SGAR.

[7]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[8]  Allen M. Waxman,et al.  Color Night Vision: Opponent Processing in the Fusion of Visible and IR Imagery , 1997, Neural Networks.

[9]  Gemma Piella,et al.  A general framework for multiresolution image fusion: from pixels to regions , 2003, Inf. Fusion.

[10]  Jürgen Schmidhuber,et al.  LSTM recurrent networks learn simple context-free and context-sensitive languages , 2001, IEEE Trans. Neural Networks.

[11]  Paul Rodríguez,et al.  Simple Recurrent Networks Learn Context-Free and Context-Sensitive Languages by Counting , 2001, Neural Computation.

[12]  E. Salari,et al.  Integrated recurrent neural network for image resolution enhancement from multiple image frames , 2003 .

[13]  DeLiang Wang,et al.  On Temporal Generalization of Simple Recurrent Networks , 1996, Neural Networks.

[14]  J. Flusser,et al.  Image Fusion via Probabilistic Deconvolution , 2004 .

[15]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[16]  A. Garzelli.  WAVELET-BASED FUSION OF OPTICAL AND SAR IMAGE DATA OVER URBAN AREA , 2002 .

[17]  Yun Zhang,et al.  Multi-resolution and multi-spectral image fusion for urban object extraction , 2004 .

[18]  B. S. Manjunath,et al.  Multi-sensor image fusion using the wavelet transform , 1994, Proceedings of 1st International Conference on Image Processing.

[19]  Wl Lau,et al.  Comparison of image data fusion techniques using entropy and INI , 2016 .

[20]  R. Ivry,et al.  The neural representation of time , 2004, Current Opinion in Neurobiology.

[21]  Jos Nijhuis,et al.  Neural network approaches to capture temporal information , 2001 .

[22]  Andreas V. M. Herz,et al.  How Is Time Represented in the Brain , 2006 .

[23]  Michal Irani,et al.  Super resolution from image sequences , 1990, [1990] Proceedings. 10th International Conference on Pattern Recognition.

[24]  S. Sides,et al.  Comparison of three different methods to merge multiresolution and multispectral data: Landsat TM and SPOT panchromatic , 1991 .

[25]  J. Schott,et al.  Resolution enhancement of multispectral image data to improve classification accuracy , 1993 .

[26]  R. Gregory.  Eye and Brain: The Psychology of Seeing , 1969 .

[27]  R. Haber,et al.  Visual Perception , 2018, Encyclopedia of Database Systems.

[28]  Roger Y. Tsai,et al.  Multiframe image restoration and registration , 1984 .

[29]  Peter Caccetta,et al.  Image Fusion with Conditional Probability Networks for Monitoring the Salinization of Farmland , 1998, Digit. Signal Process..