Automated Visual Defect Classification for Flat Steel Surface: A Survey

For a typical surface automated visual inspection (AVI) instrument of planar materials, defect classification is an indispensable part after defect detection, which acts as a crucial precondition for achieving the online quality inspection of end products. In the industrial environment of manufacturing flat steels, this task is awfully difficult due to diverse defect appearances, ambiguous intraclass, and interclass distances. This article attempts to present a focused but systematic review of the traditional and emerging automated computer-vision-based defect classification methods by investigating approximately 140 studies on three specific flat steel products of con-casting slabs, hot-rolled steel strips, and cold-rolled steel strips. According to the natural image processing procedure of defect recognition, the diverse approaches are grouped into five successive parts: image acquisition, image preprocessing, feature extraction, feature selection, and defect classifier. Recent literature has been reviewed from an industrial goal-oriented perspective to provide some guidelines for future studies and recommend suitable methods for boosting the surface quality inspection level of AVI instruments.

[1]  Ke Xu,et al.  Application of Shearlet transform to classification of surface defects for metals , 2015, Image Vis. Comput..

[2]  Sang Woo Kim,et al.  Algorithm for detecting seam cracks in steel plates using a Gabor filter combination method. , 2014, Applied optics.

[3]  Qian Huang,et al.  Improving Automatic Detection of Defects in Castings by Applying Wavelet Technique , 2006, IEEE Transactions on Industrial Electronics.

[4]  Szu-Hao Huang,et al.  Automated visual inspection in the semiconductor industry: A survey , 2015, Comput. Ind..

[5]  Hong Zheng,et al.  Automatic inspection of metallic surface defects using genetic algorithms , 2002 .

[6]  Xu Ke,et al.  Surface defect classification of steels with a new semi-supervised learning method , 2019, Optics and Lasers in Engineering.

[7]  Liang-Tien Chia,et al.  Convolutional networks for voting-based anomaly classification in metal surface inspection , 2017, 2017 IEEE International Conference on Industrial Technology (ICIT).

[8]  Yichuang Sun,et al.  Automated Visual Defect Detection for Flat Steel Surface: A Survey , 2020, IEEE Transactions on Instrumentation and Measurement.

[9]  Kay Chen Tan,et al.  A Generic Deep-Learning-Based Approach for Automated Surface Inspection , 2018, IEEE Transactions on Cybernetics.

[10]  Mohammad Reza Yazdchi,et al.  Detection and Classification of Surface Defects of Cold Rolling Mill Steel Using Morphology and Neural Network , 2008, 2008 International Conference on Computational Intelligence for Modelling Control & Automation.

[11]  Fatimah Khalid,et al.  Surface Defects Classification of Hot-Rolled Steel Strips Using Multi-directional Shearlet Features , 2018, Arabian Journal for Science and Engineering.

[12]  Xi Yang,et al.  Surface Defect Detection via Entity Sparsity Pursuit With Intrinsic Priors , 2020, IEEE Transactions on Industrial Informatics.

[13]  Yigang He,et al.  A cost-effective and automatic surface defect inspection system for hot-rolled flat steel , 2016 .

[14]  Yan Yunhui.  Surface Defect Recognition for Steel Strips by Combining Multiple Classifiers , 2012 .

[15]  Ke Xu,et al.  A Bran-new Feature Extraction Method and its application to Surface Defect Recognition of Hot Rolled Strips , 2007, 2007 IEEE International Conference on Automation and Logistics.

[16]  Jie Zhao,et al.  Steel surface defects recognition based on multi-type statistical features and enhanced twin support vector machine , 2017 .

[17]  Xiaoping Liu,et al.  Multi-class classification method for strip steel surface defects based on support vector machine with adjustable hyper-sphere , 2018, Journal of Iron and Steel Research International.

[18]  Gyogwon Koo,et al.  Automated defect inspection system for metal surfaces based on deep learning and data augmentation , 2020 .

[19]  Anders Landström,et al.  Morphology-Based Crack Detection for Steel Slabs , 2012, IEEE Journal of Selected Topics in Signal Processing.

[20]  Ge-Wen Kang,et al.  Surface defects inspection of cold rolled strips based on neural network , 2005, 2005 International Conference on Machine Learning and Cybernetics.

[21]  D Jeulin,et al.  Texture classification by statistical learning from morphological image processing: application to metallic surfaces , 2010, Journal of microscopy.

[22]  Feng Xuehui,et al.  The Strip Steel Surface Defects Classification Method Based on Weak Classifier Adaptive Enhancement , 2011, 2011 Third International Conference on Measuring Technology and Mechatronics Automation.

[23]  Łukasz Rauch,et al.  Use of Artificial Intelligence in Classification of Mill Scale Defects , 2015 .

[24]  Florent Dupont,et al.  Optimization of the recognition of defects in flat steel products with the cost matrices theory , 1997 .

[25]  Matti Pietikäinen,et al.  Deep Learning for Generic Object Detection: A Survey , 2018, International Journal of Computer Vision.

[26]  De Xu,et al.  Automatic Metallic Surface Defect Detection and Recognition with Convolutional Neural Networks , 2018, Applied Sciences.

[27]  Franz Pernkopf,et al.  Detection of surface defects on raw steel blocks using Bayesian network classifiers , 2004, Pattern Analysis and Applications.

[28]  A. O. Martins Luiz,et al.  Automatic detection of surface defects on rolled steel using Computer Vision and Artificial Neural Networks , 2010, IECON 2010 - 36th Annual Conference on IEEE Industrial Electronics Society.

[29]  Pavel Hora,et al.  Surface defect classification and detection on extruded aluminum profiles using convolutional neural networks , 2020, International Journal of Material Forming.

[30]  Stefano Roccella,et al.  Visual-Based Defect Detection and Classification Approaches for Industrial Applications—A SURVEY , 2020, Sensors.

[31]  Paul W. Fieguth,et al.  A review on computer vision based defect detection and condition assessment of concrete and asphalt civil infrastructure , 2015, Adv. Eng. Informatics.

[32]  Liming Liu,et al.  Multi-class classification method using twin support vector machines with multi-information for steel surface defects , 2018 .

[33]  Ari Ide-Ektessabi,et al.  A Contrast Adjustment Thresholding Method for Surface Defect Detection Based on Mesoscopy , 2015, IEEE Transactions on Industrial Informatics.

[34]  Chang-hou Lu,et al.  A local annular contrast based real-time inspection algorithm for steel bar surface defects , 2012 .

[35]  Sang Woo Kim,et al.  Detection of Scarfing Faults on the Edges of Slabs , 2014 .

[36]  Ke Xu,et al.  Online Surface Defect Identification of Cold Rolled Strips Based on Local Binary Pattern and Extreme Learning Machine , 2018 .

[37]  Anirban Mukherjee,et al.  Automatic Defect Detection on Hot-Rolled Flat Steel Products , 2013, IEEE Transactions on Instrumentation and Measurement.

[38]  Xiang-yang Zhao,et al.  An Improved BP Algorithm and Its Application in Classification of Surface Defects of Steel Plate , 2007 .

[39]  Maoxiang Chu,et al.  Strip Steel Surface Defect Recognition Based on Novel Feature Extraction and Enhanced Least Squares Twin Support Vector Machine , 2014 .

[40]  Xiaoping Liu,et al.  The Strip Steel Surface Defect Recognition Based on Multiple Support Vector Hyper‐Sphere with Feature and Sample Weights , 2016 .

[41]  De Xu,et al.  Weak scratch detection and defect classification methods for a large-aperture optical element , 2017 .

[42]  Jing Li,et al.  Research Progress of Visual Inspection Technology of Steel Products—A Review , 2018, Applied Sciences.

[43]  Boštjan Likar,et al.  Real-Time Rotation Estimation Using Histograms of Oriented Gradients , 2014, PloS one.

[44]  Mauricio Orozco-Alzate,et al.  Incremental learning of concept drift in Multiple Instance Learning for industrial visual inspection , 2019, Comput. Ind..

[45]  Li Yi,et al.  An evolutionary classifier for steel surface defects with small sample set , 2017, EURASIP J. Image Video Process..

[46]  Mao-xiang Chu,et al.  Multi-class classification for steel surface defects based on machine learning with quantile hyper-spheres , 2017 .

[47]  Fereidoon Moghadas Nejad,et al.  Image Based Techniques for Crack Detection, Classification and Quantification in Asphalt Pavement: A Review , 2017 .

[48]  Maoxiang Chu,et al.  Multi-class Classification Methods of Enhanced LS-TWSVM for Strip Steel Surface Defects , 2014 .

[49]  Ke Xu,et al.  An Algorithm for Surface Defect Identification of Steel Plates Based on Genetic Algorithm and Extreme Learning Machine , 2017 .

[50]  Hui Lin,et al.  Defect Image Sample Generation With GAN for Improving Defect Recognition , 2020, IEEE Transactions on Automation Science and Engineering.

[51]  Ke Xu,et al.  Application of RNAMlet to surface defect identification of steels , 2018, Optics and Lasers in Engineering.

[52]  Muhammed Fatih Talu,et al.  Fabric defect detection systems and methods—A systematic literature review , 2016 .

[53]  Ming-Kuei Hu,et al.  Visual pattern recognition by moment invariants , 1962, IRE Trans. Inf. Theory.

[54]  Kewen Xia,et al.  Dimension Reduction and Defect Recognition of Strip Surface Defects Based on Intelligent Information Processing , 2018 .

[55]  Dusmanta Kumar Mohanta,et al.  Review of vision-based steel surface inspection systems , 2014, EURASIP Journal on Image and Video Processing.

[56]  Tianlong Zhang,et al.  Classification of steel samples by laser-induced breakdown spectroscopy and random forest , 2016 .

[57]  Daniel F. Garcia,et al.  A Non-Invasive Technique for Online Defect Detection on Steel Strip Surfaces , 2016 .

[58]  Marco Vannucci,et al.  A fuzzy inference system applied to defect detection in flat steel production , 2010, International Conference on Fuzzy Systems.

[59]  Feng Xiang,et al.  Modeling and recognition of steel-plate surface defects based on a new backward boosting algorithm , 2018 .

[60]  Di He,et al.  Design of multi-scale receptive field convolutional neural network for surface inspection of hot rolled steels , 2019, Image Vis. Comput..

[61]  Augustus Odena,et al.  Semi-Supervised Learning with Generative Adversarial Networks , 2016, ArXiv.

[62]  Ying Tian,et al.  Steel Surface Defect Detection Using a New Haar–Weibull-Variance Model in Unsupervised Manner , 2017, IEEE Transactions on Instrumentation and Measurement.

[63]  Jon Atli Benediktsson,et al.  Hyperspectral Image Classification via Multiple-Feature-Based Adaptive Sparse Representation , 2017, IEEE Transactions on Instrumentation and Measurement.

[64]  Tod S. Levitt,et al.  A Real-Time Automated Visual Inspection System for Hot Steel Slabs , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[65]  Yunhui Yan,et al.  Vision-based automatic detection of steel surface defects in the cold rolling process: considering the influence of industrial liquids and surface textures , 2017 .

[66]  Paolo Valigi,et al.  Magnetic imaging and machine vision NDT for the on-line inspection of stainless steel strips , 2013 .

[67]  Yu He,et al.  Semi-supervised defect classification of steel surface based on multi-training and generative adversarial network , 2019, Optics and Lasers in Engineering.

[68]  Xuewu Zhang,et al.  Surface Defect Target Identification on Copper Strip Based on Adaptive Genetic Algorithm and Feature Saliency , 2013 .

[69]  Nicu Sebe,et al.  Optimal graph learning with partial tags and multiple features for image and video annotation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[70]  Dennis M. Healy,et al.  Wavelet transform domain filters: a spatially selective noise filtration technique , 1994, IEEE Trans. Image Process..

[71]  Franz Pernkopf,et al.  Image Acquisition Techniques for Automatic Visual Inspection of Metallic Surfaces , 2003 .

[72]  Li Chen,et al.  Surface Inspection System of Steel Strip Based on Machine Vision , 2009, 2009 First International Workshop on Database Technology and Applications.

[73]  Zhang Xuewu,et al.  A vision inspection system for the surface defects of strongly reflected metal based on multi-class SVM , 2011 .

[74]  David Berthelot,et al.  MixMatch: A Holistic Approach to Semi-Supervised Learning , 2019, NeurIPS.

[75]  Ke Xu,et al.  Improved contourlet transform construction and its application to surface defect recognition of metals , 2020, Multidimens. Syst. Signal Process..

[76]  Jie Zhao,et al.  The cold rolling strip surface defect on-line inspection system based on machine vision , 2010, 2010 Second Pacific-Asia Conference on Circuits, Communications and System.

[77]  Youping Chen,et al.  Learning a Class-specific and Shared Dictionary for Classifying Surface Defects of Steel Sheet , 2017 .

[78]  Jon Atli Benediktsson,et al.  Spectral–Spatial Hyperspectral Image Classification With Edge-Preserving Filtering , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[79]  Jun Li,et al.  Recent Advances on Spectral–Spatial Hyperspectral Image Classification: An Overview and New Guidelines , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[80]  José R. Perán,et al.  Automated visual classification of frequent defects in flat steel coils , 2011 .

[81]  Yunhui Yan,et al.  Automatic Recognition of Surface Defects on Hot-Rolled Steel Strip Using Scattering Convolution Network , 2014 .

[82]  Xia Kewen,et al.  Defect Classification Recognition on Strip Steel Surface Using Second-Order Cone Programming-Relevance Vector Machine Algorithm , 2016 .

[83]  S. R. Aghdam,et al.  A fast method of steel surface defect detection using decision trees applied to LBP based features , 2012, 2012 7th IEEE Conference on Industrial Electronics and Applications (ICIEA).

[84]  Ke Xu,et al.  An Improved MB-LBP Defect Recognition Approach for the Surface of Steel Plates , 2019, Applied Sciences.

[85]  Yung-Nien Sun,et al.  A computer vision system for automatic steel surface inspection , 2010, 2010 5th IEEE Conference on Industrial Electronics and Applications.

[86]  Xiang Zhang,et al.  Automatic classification of defects on the product surface in grinding and polishing , 2006 .

[87]  Ralph Seulin,et al.  Machine vision system for surface inspection on brushed industrial parts , 2004, IS&T/SPIE Electronic Imaging.

[88]  Xiaoming Liu,et al.  Surface defect identification of aluminium strips with non-subsampled shearlet transform , 2020 .

[89]  Sang Jun Lee,et al.  Steel-surface defect detection using a switching-lighting scheme. , 2016, Applied optics.

[90]  N. H. C. Yung,et al.  Automated fabric defect detection - A review , 2011, Image Vis. Comput..

[91]  Xianghua Xie,et al.  A Review of Recent Advances in Surface Defect Detection using Texture analysis Techniques , 2008 .

[92]  Jon Atli Benediktsson,et al.  Spectral–Spatial Hyperspectral Image Classification via Multiscale Adaptive Sparse Representation , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[93]  Ling Li,et al.  Distributed defect recognition on steel surfaces using an improved random forest algorithm with optimal multi-feature-set fusion , 2018, Multimedia Tools and Applications.

[94]  Siti Zaiton Mohd Hashim,et al.  AUTOMATED VISION INSPECTION OF TIMBER SURFACE DEFECT: A REVIEW , 2015 .

[96]  Shervan Fekri Ershad,et al.  Multi-Resolution and Noise-Resistant Surface Defect Detection Approach Using New Version of Local Binary Patterns , 2017, Appl. Artif. Intell..

[97]  Weikuan Jia,et al.  Deep-Learning-Based Small Surface Defect Detection via an Exaggerated Local Variation-Based Generative Adversarial Network , 2020, IEEE Transactions on Industrial Informatics.

[98]  Ruiyu Liang,et al.  A vision inspection system for the surface defects of strongly reflected metal based on multi-class SVM , 2011, Expert Syst. Appl..

[99]  Kechen Song,et al.  Micro Surface Defect Detection Method for Silicon Steel Strip Based on Saliency Convex Active Contour Model , 2013 .

[100]  Kechen Song,et al.  Surface Defect Detection Method Using Saliency Linear Scanning Morphology for Silicon Steel Strip under Oil Pollution Interference , 2014 .

[101]  S. Sadri,et al.  Detection of steel defect using the image processing algorithms , 2008, 2008 IEEE International Multitopic Conference.

[102]  Li Liu,et al.  Surface Defect Classification for Hot-Rolled Steel Strips by Selectively Dominant Local Binary Patterns , 2019, IEEE Access.

[103]  Qingyong Li,et al.  A Real-Time Visual Inspection System for Discrete Surface Defects of Rail Heads , 2012, IEEE Transactions on Instrumentation and Measurement.

[104]  Sang-Woo Kim,et al.  Detection of Periodic Defects Using Dual-Light Switching Lighting Method on the Surface of Thick Plates , 2015 .

[105]  Yunhui Yan,et al.  Steel Surface Defect Classification Based on Discriminant Manifold Regularized Local Descriptor , 2018, IEEE Access.

[106]  Pavel S. Kostenetskiy,et al.  Real-time system for automatic cold strip surface defect detection , 2019, FME Transactions.

[107]  Jürgen Schmidhuber,et al.  Multi-scale pyramidal pooling network for generic steel defect classification , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[108]  Anil K. Jain,et al.  A Survey of Automated Visual Inspection , 1995, Comput. Vis. Image Underst..

[109]  Yunhui Yan,et al.  A noise robust method based on completed local binary patterns for hot-rolled steel strip surface defects , 2013 .

[110]  Yang Fan,et al.  Strip Steel Surface Defects Recognition Based on SOCP Optimized Multiple Kernel RVM , 2018 .

[111]  Youping Chen,et al.  Classification of surface defects on steel sheet using convolutional neural networks , 2017 .

[112]  Yao Feng,et al.  A multi-class classifier based on support vector hyper-spheres for steel plate surface defects , 2019, Chemometrics and Intelligent Laboratory Systems.

[113]  Maoxiang Chu,et al.  Steel surface defect classification using multiple hyper-spheres support vector machine with additional information , 2018 .

[114]  Mehran Yazdi,et al.  Steel Surface Defect Detection Using Texture Segmentation Based on Multifractal Dimension , 2009, 2009 International Conference on Digital Image Processing.

[115]  I. Kunttu,et al.  Shape-based retrieval of industrial surface defects using angular radius Fourier descriptor , 2007 .

[116]  C.S. Lee,et al.  Feature extraction algorithm based on adaptive wavelet packet for surface defect classification , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[117]  Tony Lindeberg,et al.  An automatic assessment scheme for steel quality inspection , 2000, Machine Vision and Applications.

[118]  Ke Xu,et al.  Surface Detection of Continuous Casting Slabs Based on Curvelet Transform and Kernel Locality Preserving Projections , 2013 .

[119]  Xiaoming Liu,et al.  Real-Time Steel Inspection System Based on Support Vector Machine and Multiple Kernel Learning , 2011 .

[120]  Ajay Kumar,et al.  Computer-Vision-Based Fabric Defect Detection: A Survey , 2008, IEEE Transactions on Industrial Electronics.

[121]  Howard Huang,et al.  Process knowledge based multi-class support vector classification (PK-MSVM) approach for surface defects in hot rolling , 2011, Expert Syst. Appl..

[122]  Huiqian Wang,et al.  Defect detection in slab surface: a novel dual Charge-coupled Device imaging-based fuzzy connectedness strategy. , 2014, The Review of scientific instruments.

[123]  Michael Ying Yang,et al.  A deep-learning-based approach for fast and robust steel surface defects classification , 2019, Optics and Lasers in Engineering.

[124]  Yigang He,et al.  Generalized Completed Local Binary Patterns for Time-Efficient Steel Surface Defect Classification , 2019, IEEE Transactions on Instrumentation and Measurement.

[125]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[126]  Maoxiang Chu,et al.  Invariant Feature Extraction Method Based on Smoothed Local Binary Pattern for Strip Steel Surface Defect , 2015 .

[127]  Di He,et al.  Defect detection of hot rolled steels with a new object detection framework called classification priority network , 2019, Comput. Ind. Eng..

[128]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[129]  Hui Lin,et al.  Classification of Hyperspectral Images by Gabor Filtering Based Deep Network , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[130]  Jon Atli Benediktsson,et al.  Deep Learning for Hyperspectral Image Classification: An Overview , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[131]  Jürgen Schmidhuber,et al.  Steel defect classification with Max-Pooling Convolutional Neural Networks , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[132]  Yuanxiang Li,et al.  Classification of defects in steel strip surface based on multiclass support vector machine , 2014, Multimedia Tools and Applications.

[133]  Liqiang Nie,et al.  Genetic algorithm and mathematical morphology based binarization method for strip steel defect image with non-uniform illumination , 2016, J. Vis. Commun. Image Represent..

[134]  Ke Xu,et al.  Application of multi-scale feature extraction to surface defect classification of hot-rolled steels , 2013, International Journal of Minerals, Metallurgy, and Materials.

[135]  Charles A. Harlow,et al.  Automated Visual Inspection: A Survey , 1982, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[136]  Xiu-yong Wu,et al.  Application of Undecimated Wavelet Transform to Surface Defect Detection of Hot Rolled Steel Plates , 2008, 2008 Congress on Image and Signal Processing.

[137]  Liqiang Nie,et al.  Surface defect classification in large-scale strip steel image collection via hybrid chromosome genetic algorithm , 2016, Neurocomputing.