Generating Defective Epoxy Drop Images for Die Attachment in Integrated Circuit Manufacturing via Enhanced Loss Function CycleGAN

In integrated circuit manufacturing, defects in epoxy drops for die attachments are required to be identified during production. Modern identification techniques based on vision-based deep neural networks require the availability of a very large number of defect and non-defect epoxy drop images. In practice, however, very few defective epoxy drop images are available. This paper presents a generative adversarial network solution to generate synthesized defective epoxy drop images as a data augmentation approach so that vision-based deep neural networks can be trained or tested using such images. More specifically, the so-called CycleGAN variation of the generative adversarial network is used by enhancing its cycle consistency loss function with two other loss functions consisting of learned perceptual image patch similarity (LPIPS) and a structural similarity index metric (SSIM). The results obtained indicate that when using the enhanced loss function, the quality of synthesized defective epoxy drop images are improved by 59%, 12%, and 131% for the metrics of the peak signal-to-noise ratio (PSNR), universal image quality index (UQI), and visual information fidelity (VIF), respectively, compared to the CycleGAN standard loss function. A typical image classifier is used to show the improvement in the identification outcome when using the synthesized images generated by the developed data augmentation approach.

[1]  Marco F. Huber,et al.  Defect Transfer GAN: Diverse Defect Synthesis for Data Augmentation , 2023, BMVC.

[2]  Juan José Aguilar Martín,et al.  Intraclass Image Augmentation for Defect Detection Using Generative Adversarial Neural Networks , 2023, Sensors.

[3]  Long Wen,et al.  A New Cycle-consistent Adversarial Networks With Attention Mechanism for Surface Defect Classification With Small Samples , 2022, IEEE Transactions on Industrial Informatics.

[4]  Zhenyu Liu,et al.  Mask2Defect: A Prior Knowledge-Based Data Augmentation Method for Metal Surface Defect Inspection , 2022, IEEE Transactions on Industrial Informatics.

[5]  Nian Cai,et al.  Multi-scale GAN with transformer for surface defect inspection of IC metal packages , 2022, Expert Syst. Appl..

[6]  Xinggang Wang,et al.  Region- and Strength-Controllable GAN for Defect Generation and Segmentation in Industrial Images , 2022, IEEE Transactions on Industrial Informatics.

[7]  Yang Xu,et al.  Unsupervised deep learning method for bridge condition assessment based on intra-and inter-class probabilistic correlations of quasi-static responses , 2022, Structural Health Monitoring.

[8]  Zhaohui Jiang,et al.  A novel MAS-GAN-based data synthesis method for object surface defect detection , 2022, Neurocomputing.

[9]  Ali Asghar Heidari,et al.  Generative Adversarial Networks in Medical Image augmentation: A review , 2022, Comput. Biol. Medicine.

[10]  Omar Sultan Al-Kadi,et al.  The effect of loss function on conditional generative adversarial networks , 2022, J. King Saud Univ. Comput. Inf. Sci..

[11]  Xinggang Wang,et al.  Defect attention template generation cycleGAN for weakly supervised surface defect segmentation , 2021, Pattern Recognit..

[12]  Hui Li,et al.  A self‐sparse generative adversarial network for autonomous early‐stage design of architectural sketches , 2021, Comput. Aided Civ. Infrastructure Eng..

[13]  Nian Cai,et al.  IC solder joint inspection via generator-adversarial-network based template , 2021, Machine Vision and Applications.

[14]  P. Yan,et al.  A Method for Classification of Surface Defect on Metal Workpieces Based on Twin Attention Mechanism Generative Adversarial Network , 2021, IEEE Sensors Journal.

[15]  Chao-Ton Su,et al.  CNNs Combined With a Conditional GAN for Mura Defect Classification in TFT-LCDs , 2021, IEEE Transactions on Semiconductor Manufacturing.

[16]  Shijian Lu,et al.  Defect-GAN: High-Fidelity Defect Synthesis for Automated Defect Inspection , 2021, 2021 IEEE Winter Conference on Applications of Computer Vision (WACV).

[17]  D. Perng,et al.  Detecting and Measuring Defects in Wafer Die Using GAN and YOLOv3 , 2020, Applied Sciences.

[18]  Girish Kumar,et al.  Synthetic data augmentation for surface defect detection and classification using deep learning , 2020, Journal of Intelligent Manufacturing.

[19]  Bengong Zhang,et al.  Defect detection of fabrics With Generative Adversarial Network Based flaws modeling , 2020, 2020 Chinese Automation Congress (CAC).

[20]  Seon Joo Kim,et al.  Investigating Loss Functions for Extreme Super-Resolution , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[21]  Hui Lin,et al.  Defect Image Sample Generation With GAN for Improving Defect Recognition , 2020, IEEE Transactions on Automation Science and Engineering.

[22]  Yu He,et al.  Semi-supervised defect classification of steel surface based on multi-training and generative adversarial network , 2019, Optics and Lasers in Engineering.

[23]  Taghi M. Khoshgoftaar,et al.  A survey on Image Data Augmentation for Deep Learning , 2019, Journal of Big Data.

[24]  Amos J. Storkey,et al.  Augmenting Image Classifiers Using Data Augmentation Generative Adversarial Networks , 2018, ICANN.

[25]  J. S. Hu,et al.  Industrial Anomaly Detection and One-class Classification using Generative Adversarial Networks , 2018, 2018 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM).

[26]  Saeid Nahavandi,et al.  SSIMLayer: Towards Robust Deep Representation Learning via Nonlinear Structural Similarity , 2018, 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC).

[27]  Zengfu Wang,et al.  A Generative Adversarial Network Based Framework for Unsupervised Visual Surface Inspection , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[28]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  Jan Kautz,et al.  Loss Functions for Image Restoration With Neural Networks , 2017, IEEE Transactions on Computational Imaging.

[30]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Chuan Li,et al.  Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks , 2016, ECCV.

[33]  J. DiCarlo,et al.  Using goal-driven deep learning models to understand sensory cortex , 2016, Nature Neuroscience.

[34]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Aaron C. Courville,et al.  Generative Adversarial Nets , 2014, NIPS.

[36]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[37]  Yi Li,et al.  Recent advances of conductive adhesives as a lead-free alternative in electronic packaging: Materials, processing, reliability and applications , 2006 .

[38]  Alan C. Bovik,et al.  Image information and visual quality , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[39]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[40]  A. Bovik,et al.  A universal image quality index , 2002, IEEE Signal Processing Letters.

[41]  Xinyu Li,et al.  A New Contrastive GAN With Data Augmentation for Surface Defect Recognition Under Limited Data , 2023, IEEE Transactions on Instrumentation and Measurement.

[42]  Jaewon Kim,et al.  GAN-based Defect Image Generation for Imbalanced Defect Classification of OLED panels , 2022, EGSR.

[43]  Guifang Shao,et al.  DuCaGAN: Unified Dual Capsule Generative Adversarial Network for Unsupervised Image-to-Image Translation , 2020, IEEE Access.

[44]  Moni Naor,et al.  Computer Vision – ECCV 2016 , 2016, Lecture Notes in Computer Science.