Eyes on the Target: Super-Resolution and License-Plate Recognition in Low-Quality Surveillance Videos

Low-quality surveillance cameras throughout the cities could provide important cues to identify a suspect, for example, in a crime scene. However, the license-plate recognition is especially difficult under poor image resolutions. In this vein, super-resolution (SR) can be an inexpensive solution, via software, to overcome this limitation. Consecutive frames in a video may contain different information that could be integrated into a single image, richer in details. In this paper, we design and develop a novel, free and open-source framework underpinned by SR and automatic license-plate recognition (ALPR) techniques to identify license-plate characters in low-quality real-world traffic videos, captured by cameras not designed specifically for the ALPR task, aiding forensic analysts in understanding an event of interest. The framework handles the necessary conditions to identify a target license plate, using a novel methodology to locate, track, align, super-resolve, and recognize its alphanumerics. The user receives as outputs the rectified and super-resolved license-plate, richer in detail, and also the sequence of license-plates characters that have been automatically recognized in the super-resolved image. Additionally, we also design and develop a novel SR method that projects the license-plates separately onto the rectified grid, and then fill in the missing pixels using inpainting techniques. We compare the different algorithms in the framework (five for tracking, three for registration, seven for reconstruction, two for post-processing, and two for the recognition step), and present discussions on the pros and cons of each choice. Our experiments show that SR can indeed increase the number of correctly recognized characters posing the framework as an important step toward providing forensic experts and practitioners with a solution for the license-plate recognition problem under difficult acquisition conditions.

[1]  Roger Y. Tsai,et al.  Multiframe image restoration and registration , 1984 .

[2]  Gunnar Farnebäck,et al.  Two-Frame Motion Estimation Based on Polynomial Expansion , 2003, SCIA.

[3]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[4]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[5]  M. Ng,et al.  Superresolution image reconstruction using fast inpainting algorithms , 2007 .

[6]  Guna Seetharaman,et al.  Application Of Papoulis-Gerchberg Method In Image Super-Resolution and Inpainting , 2009, Comput. J..

[7]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Ying Liu,et al.  A License Plate Super-resolution Reconstruction Algorithm Based on Manifold Learning , 2014, 2014 IEEE 17th International Conference on Computational Science and Engineering.

[9]  Avinash C. Kak,et al.  Principles of computerized tomographic imaging , 2001, Classics in applied mathematics.

[10]  A. N. Rajagopalan,et al.  Superresolution of License Plates in Real Traffic Videos , 2007, IEEE Transactions on Intelligent Transportation Systems.

[11]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[12]  Wael Badawy,et al.  Automatic License Plate Recognition (ALPR): A State-of-the-Art Review , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[13]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[14]  Yi Wang,et al.  GPU-CPU implementation for super-resolution mosaicking of Unmanned Aircraft System (UAS) surveillance video , 2010, 2010 IEEE Southwest Symposium on Image Analysis & Interpretation (SSIAI).

[15]  Michael Elad,et al.  Fast and robust multiframe super resolution , 2004, IEEE Transactions on Image Processing.

[16]  Ying-Hong Liang,et al.  A rapid inclined distortion rectification method for vehicle licenses plate images , 2008, 2008 International Conference on Machine Learning and Cybernetics.

[17]  Sabine Süsstrunk,et al.  Superresolution images reconstructed from aliased images , 2003, Visual Communications and Image Processing.

[18]  R. Gerchberg.  Super-resolution through Error Energy Reduction , 1974 .

[19]  Hanseok Ko,et al.  Resolution enhancement of ROI from surveillance video using Bernstein interpolation , 2011, 2011 8th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS).

[20]  Gary R. Bradski,et al.  ORB: An efficient alternative to SIFT or SURF , 2011, 2011 International Conference on Computer Vision.

[21]  J.-Y. Bouguet,et al.  Pyramidal implementation of the lucas kanade feature tracker , 1999 .

[22]  David G. Lowe,et al.  Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration , 2009, VISAPP.

[23]  Guillermo Sapiro,et al.  Navier-stokes, fluid dynamics, and image and video inpainting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[24]  R. Smith,et al.  An Overview of the Tesseract OCR Engine , 2007, Ninth International Conference on Document Analysis and Recognition (ICDAR 2007).

[25]  Stephen Lin,et al.  Super resolution using edge prior and single image detail synthesis , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[26]  Moon Gi Kang,et al.  Super-resolution image reconstruction: a technical overview , 2003, IEEE Signal Process. Mag..

[27]  Klamer Schutte,et al.  Robust Fusion of Irregularly Sampled Data Using Adaptive Normalized Convolution , 2006, EURASIP J. Adv. Signal Process..

[28]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[29]  Alexandru Telea,et al.  An Image Inpainting Technique Based on the Fast Marching Method , 2004, J. Graphics, GPU, & Game Tools.

[30]  Hiroshi Murase,et al.  Robust Face Super-Resolution Using Free-Form Deformations for Low-Quality Surveillance Video , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[31]  Thomas M. Breuel,et al.  The OCRopus open source OCR system , 2008, Electronic Imaging.

[32]  Haidawati Nasir,et al.  Singular value decomposition based fusion for super-resolution image reconstruction , 2011, ICSIPA.

[33]  H Stark,et al.  High-resolution image recovery from image-plane arrays, using convex projections. , 1989, Journal of the Optical Society of America. A, Optics and image science.

[34]  Michal Irani,et al.  Super resolution from image sequences , 1990, [1990] Proceedings. 10th International Conference on Pattern Recognition.

[35]  E. Meijering,et al.  A chronology of interpolation: from ancient astronomy to modern signal and image processing , 2002, Proc. IEEE.

[36]  E. Meijering.  A chronology of interpolation: from ancient astronomy to modern signal and image processing , 2002, Proc. IEEE.

[37]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Xuelong Li,et al.  Learning Multiple Linear Mappings for Efficient Single Image Super-Resolution , 2015, IEEE Transactions on Image Processing.

[39]  A. Çapar,et al.  License Plate Recognition From Still Images and Video Sequences: A Survey , 2008, IEEE Transactions on Intelligent Transportation Systems.

[40]  William T. Freeman,et al.  Example-Based Super-Resolution , 2002, IEEE Computer Graphics and Applications.

[41]  Weisi Lin,et al.  Visual Object Tracking Based on Backward Model Validation , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[42]  N. Otsu.  A threshold selection method from gray level histograms , 1979 .

[43]  Shmuel Peleg,et al.  Robust super-resolution , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[44]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.

[45]  N. K. Bose,et al.  High resolution image formation from low resolution frames using Delaunay triangulation , 2002, IEEE Trans. Image Process..

[46]  Kees Joost Batenburg,et al.  Super-Resolution of License Plate Images using Algebraic Reconstruction Technique , 2013 .

[47]  Dmitriy Vatolin,et al.  Fast video super-resolution via classification , 2008, 2008 15th IEEE International Conference on Image Processing.

[48]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[49]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[50]  C. Manjula,et al.  SUPER RESOLUTION BASED INPAINTING , 2015 .

[51]  Anderson Rocha,et al.  Fast and Effective Geometric K-Nearest Neighbors Multi-frame Super-Resolution , 2015, 2015 28th SIBGRAPI Conference on Graphics, Patterns and Images.

[52]  Kai-Kuang Ma,et al.  A survey on super-resolution imaging , 2011, Signal Image Video Process..

[53]  A. Papoulis.  A new algorithm in spectral analysis and band-limited extrapolation. , 1975 .

[54]  Peyman Milanfar,et al.  RAISR: Rapid and Accurate Image Super Resolution , 2016, IEEE Transactions on Computational Imaging.

[55]  Ming-Chao Chiang,et al.  Efficient image warping and super-resolution , 1996, Proceedings Third IEEE Workshop on Applications of Computer Vision. WACV'96.

[56]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[57]  A. Murat Tekalp,et al.  Super resolution recovery for multi-camera surveillance imaging , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[58]  Peter Cheeseman,et al.  Super-Resolved Surface Reconstruction from Multiple Images , 1996 .

[59]  Thomas B. Moeslund,et al.  Super-resolution: a comprehensive survey , 2014, Machine Vision and Applications.

[60]  Joel H. Ferziger,et al.  Computational methods for fluid dynamics , 1996 .

[61]  Li Xue,et al.  A New Fast Algorithm to Rectify Tilt Image of Vehicle License Plates , 2006, 2007 Chinese Control Conference.

[62]  Soumitra Kar,et al.  Using Radon Transform to Recognize Skewed Images of Vehicular License Plates , 2016, Computer.

[63]  Daniel Howard,et al.  The Novel Stochastic Bernstein Method of Functional Approximation , 2006, First NASA/ESA Conference on Adaptive Hardware and Systems (AHS'06).

[64]  Xiang Zhu,et al.  Fast super-resolution for license plate image reconstruction , 2008, 2008 19th International Conference on Pattern Recognition.

[65]  Sei-Wang Chen,et al.  Automatic license plate recognition , 2004, IEEE Transactions on Intelligent Transportation Systems.