Robust Automatic Recognition of Chinese License Plates in Natural Scenes

Automatic license plate recognition has a wide range of applications in intelligent transportation systems and is of great significance. However, most of the current work on license plate recognition focuses on the images on the front of license plates. license plate recognition in natural scenes and arbitrary perspective is still a huge challenge. To solve this problem, this work mainly studies the detection and recognition of inclined Chinese license plates in natural scenes. We propose a robust method that can detect and correct multiple license plates with severe distortion or skewing in one image and input them into the license plate recognition module to obtain the final result. Different from the existing methods of license plate detection and recognition, our method performs affine transformation during license plate detection to rectify the distorted license plate image. It can not only avoid the accumulation of intermediate errors but also improve the accuracy of recognition. As an additional contribution, we put forward a challenging Chinese license plate recognition data set, including images obtained from different scenes under a variety of weather conditions. Through a large number of comparative experiments, we have proved the effectiveness of our proposed method.

[1]  Afshin Dehghan,et al.  License Plate Detection and Recognition Using Deeply Learned Convolutional Neural Networks , 2017, ArXiv.

[2]  Nidhi Chandrakar,et al.  Study and comparison of various image edge detection techniques , 2012 .

[3]  Xiang Bai,et al.  An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[5]  Kai Wang,et al.  End-to-end scene text recognition , 2011, 2011 International Conference on Computer Vision.

[6]  Lianwen Jin,et al.  A New CNN-Based Method for Multi-Directional Car License Plate Detection , 2018, IEEE Transactions on Intelligent Transportation Systems.

[7]  Changming Sun,et al.  An End-to-End TextSpotter with Explicit Alignment and Attention , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[8]  Sei-Wang Chen,et al.  Automatic license plate recognition , 2004, IEEE Transactions on Intelligent Transportation Systems.

[9]  Rahim Panahi,et al.  Accurate Detection and Recognition of Dirty Vehicle Plate Numbers for High-Speed Applications , 2017, IEEE Transactions on Intelligent Transportation Systems.

[10]  Max Q.-H. Meng,et al.  A novel license plate location method based on wavelet transform and EMD analysis , 2015, Pattern Recognit..

[11]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[12]  Kai Wang,et al.  Word Spotting in the Wild , 2010, ECCV.

[13]  Hartmut Neven,et al.  PhotoOCR: Reading Text in Uncontrolled Conditions , 2013, 2013 IEEE International Conference on Computer Vision.

[14]  Yousri Kessentini,et al.  A two-stage deep neural network for multi-norm license plate detection and recognition , 2019, Expert Syst. Appl..

[15]  Huiyun Mao,et al.  Robust License Plate Recognition With Shared Adversarial Training Network , 2020, IEEE Access.

[16]  Xiang Bai,et al.  ASTER: An Attentional Scene Text Recognizer with Flexible Rectification , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Andrew Zisserman,et al.  Deep Structured Output Learning for Unconstrained Text Recognition , 2014, ICLR.

[18]  Shijian Lu,et al.  ESIR: End-To-End Scene Text Recognition via Iterative Image Rectification , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Pan He,et al.  Reading Scene Text in Deep Convolutional Sequences , 2015, AAAI.

[20]  Shijian Lu,et al.  Accurate recognition of words in scenes without character segmentation using recurrent neural network , 2017, Pattern Recognit..

[21]  Orhan Bulan,et al.  Segmentation- and Annotation-Free License Plate Recognition With Deep Localization and Failure Identification , 2017, IEEE Transactions on Intelligent Transportation Systems.

[22]  Tao Wang,et al.  End-to-end text recognition with convolutional neural networks , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[23]  Xiang Bai,et al.  Robust Scene Text Recognition with Automatic Rectification , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Shijian Lu,et al.  Accurate Scene Text Recognition Based on Recurrent Neural Network , 2014, ACCV.

[25]  Jürgen Schmidhuber,et al.  Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks , 2006, ICML.

[26]  Sang-Woong Lee,et al.  Multinational License Plate Recognition Using Generalized Character Sequence Detection , 2020, IEEE Access.

[27]  Zheng-Jun Zha,et al.  Towards Human-Level License Plate Recognition , 2018, ECCV.

[28]  Liusheng Huang,et al.  Towards End-to-End License Plate Detection and Recognition: A Large Dataset and Baseline , 2018, ECCV.

[29]  Jiri Matas,et al.  Real-Time Lexicon-Free Scene Text Localization and Recognition , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Simon Osindero,et al.  Recursive Recurrent Nets with Attention Modeling for OCR in the Wild , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Siome Goldenstein,et al.  Eyes on the Target: Super-Resolution and License-Plate Recognition in Low-Quality Surveillance Videos , 2017, IEEE Access.

[32]  Christopher D. Manning,et al.  Effective Approaches to Attention-based Neural Machine Translation , 2015, EMNLP.

[33]  Jiřı́ Matas,et al.  Real-time scene text localization and recognition , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[35]  William R. Schwartz,et al.  A Robust Real-Time Automatic License Plate Recognition Based on the YOLO Detector , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[36]  Christoph Meinel,et al.  SEE: Towards Semi-Supervised End-to-End Scene Text Recognition , 2017, AAAI.

[37]  Jiri Matas,et al.  Deep TextSpotter: An End-to-End Trainable Scene Text Localization and Recognition Framework , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[38]  Chunhua Shen,et al.  Reading Car License Plates Using Deep Convolutional Neural Networks and LSTMs , 2016, ArXiv.

[39]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  R. Maini.  Study and Comparison of Various Image Edge Detection Techniques , 2004 .

[41]  David Menotti,et al.  Benchmark for license plate character segmentation , 2016, J. Electronic Imaging.

[42]  Shijian Lu,et al.  Multilingual scene character recognition with co-occurrence of histogram of oriented gradients , 2016, Pattern Recognit..

[43]  Quoc V. Le,et al.  EfficientDet: Scalable and Efficient Object Detection , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[45]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Hendry,et al.  Automatic License Plate Recognition via sliding-window darknet-YOLO deep learning , 2019, Image Vis. Comput..

[47]  Chunhua Shen,et al.  Toward End-to-End Car License Plate Detection and Recognition With Deep Neural Networks , 2019, IEEE Transactions on Intelligent Transportation Systems.

[48]  Nikos Komodakis,et al.  A Robust and Efficient Approach to License Plate Detection , 2017, IEEE Transactions on Image Processing.

[49]  Wei Shi,et al.  Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification , 2016, ACL.

[50]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[51]  Wenyu Liu,et al.  Strokelets: A Learned Multi-scale Representation for Scene Text Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[52]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[53]  Wei Liu,et al.  Char-Net: A Character-Aware Neural Network for Distorted Scene Text Recognition , 2018, AAAI.

[54]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[55]  Jun Yang,et al.  A Light CNN for End-to-End Car License Plates Detection and Recognition , 2019, IEEE Access.

[56]  Muhammad Kamal Hossen,et al.  License plate detection and character recognition system for commercial vehicles based on morphological approach and template matching , 2016, 2016 3rd International Conference on Electrical Engineering and Information Communication Technology (ICEEICT).

[57]  Wang Weihong,et al.  Research on License Plate Recognition Algorithms Based on Deep Learning in Complex Environment , 2020, IEEE Access.

[58]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[59]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[60]  Enrico Magli,et al.  Robust license plate recognition using neural networks trained on synthetic images , 2019, Pattern Recognit..

[61]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.