A deep learning approach for complex microstructure inference

Automated, reliable, and objective microstructure inference from micrographs is essential for a comprehensive understanding of process-microstructure-property relations and tailored materials development. However, such inference, with the increasing complexity of microstructures, requires advanced segmentation methodologies. While deep learning offers new opportunities, an intuition about the required data quality/quantity and a methodological guideline for microstructure quantification is still missing. This, along with deep learning’s seemingly intransparent decision-making process, hampers its breakthrough in this field. We apply a multidisciplinary deep learning approach, devoting equal attention to specimen preparation and imaging, and train distinct U-Net architectures with 30–50 micrographs of different imaging modalities and electron backscatter diffraction-informed annotations. On the challenging task of lath-bainite segmentation in complex-phase steel, we achieve accuracies of 90% rivaling expert segmentations. Further, we discuss the impact of image context, pre-training with domain-extrinsic data, and data augmentation. Network visualization techniques demonstrate plausible model decisions based on grain boundary morphology.

[1]  J. Naylor The influence of the lath morphology on the yield stress and transition temperature of martensitic- bainitic steels , 1979 .

[2]  Jan Flusser,et al.  Image registration methods: a survey , 2003, Image Vis. Comput..

[3]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[4]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[5]  S. Zając,et al.  Characterisation and Quantification of Complex Bainitic Microstructures in High and Ultra-High Strength Linepipe Steels , 2005 .

[6]  S. Wright,et al.  EBSD Image Quality Mapping , 2005, Microscopy and Microanalysis.

[7]  Carlos Ortiz-de-Solorzano,et al.  Consistent and Elastic Registration of Histological Sections Using Vector-Spline Regularization , 2006, CVAMIA.

[8]  Xiaoxu Huang,et al.  Effect of block size on the strength of lath martensite in low carbon steels , 2006 .

[9]  S. Zaefferer,et al.  EBSD as a tool to identify and quantify bainite and ferrite in low‐alloyed Al‐TRIP steels , 2008, Journal of microscopy.

[10]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[11]  Haibo He,et al.  Learning from Imbalanced Data , 2009, IEEE Transactions on Knowledge and Data Engineering.

[12]  Cristian Sminchisescu,et al.  CPMC: Automatic Object Segmentation Using Constrained Parametric Min-Cuts , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[14]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[15]  Shengen Yan,et al.  Deep Image: Scaling up Image Recognition , 2015, ArXiv.

[16]  Raquel Urtasun,et al.  Understanding the Effective Receptive Field in Deep Convolutional Neural Networks , 2016, NIPS.

[17]  Christoph H. Lampert,et al.  Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation , 2016, ECCV.

[18]  S. Zaefferer,et al.  Damage Initiation in Dual-Phase Steels: Influence of Crystallographic and Morphological Parameters , 2016 .

[19]  Christopher Joseph Pal,et al.  The Importance of Skip Connections in Biomedical Image Segmentation , 2016, LABELS/DLMIA@MICCAI.

[20]  Vladlen Koltun,et al.  Multi-Scale Context Aggregation by Dilated Convolutions , 2015, ICLR.

[21]  E. Holm,et al.  ERRATUM Erratum to : UHCSDB : UltraHigh Carbon Steel Micrograph DataBase Tools for Exploring Large Heterogeneous Microstructure Datasets , 2017 .

[22]  Stefano Ermon,et al.  Label-Free Supervision of Neural Networks with Physics and Domain Knowledge , 2016, AAAI.

[23]  Brian L. DeCost,et al.  UHCSDB: UltraHigh Carbon Steel Micrograph DataBase , 2017, Integrating Materials and Manufacturing Innovation.

[24]  F. Mücklich,et al.  A Correlative Approach to Capture and Quantify Substructures by Means of Image Registration , 2017 .

[25]  Bolei Zhou,et al.  Network Dissection: Quantifying Interpretability of Deep Visual Representations , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Sergio Guadarrama,et al.  The Devil is in the Decoder , 2017, BMVC.

[27]  H. Noguchi,et al.  Bone-like crack resistance in hierarchical metastable nanolaminate steels , 2017, Science.

[28]  Ben Glocker,et al.  DLTK: State of the Art Reference Implementations for Deep Learning on Medical Images , 2017, ArXiv.

[29]  Mario Fritz,et al.  Advanced Steel Microstructural Classification by Deep Learning Methods , 2017, Scientific Reports.

[30]  Masahiko Demura,et al.  Pattern recognition with machine learning on optical microscopy images of typical metallurgical microstructures , 2018, Scientific Reports.

[31]  Shang-Hong Lai,et al.  AugGAN: Cross Domain Adaptation with GAN-Based Data Augmentation , 2018, ECCV.

[32]  Ion Stoica,et al.  Tune: A Research Platform for Distributed Model Selection and Training , 2018, ArXiv.

[33]  Quanshi Zhang,et al.  Visual interpretability for deep learning: a survey , 2018, Frontiers of Information Technology & Electronic Engineering.

[34]  Seetha Hari,et al.  Learning From Imbalanced Data , 2019, Advances in Computer and Electrical Engineering.

[35]  Thomas Brox,et al.  U-Net: deep learning for cell counting, detection, and morphometry , 2018, Nature Methods.

[36]  Stefano Cozzini,et al.  The first annotated set of scanning electron microscopy images for nanoscience , 2018, Scientific Data.

[37]  J. Yang,et al.  Phase quantification in low carbon Nb-Mo bearing steel by electron backscatter diffraction technique coupled with kernel average misorientation , 2018 .

[38]  Aleksander Madry,et al.  How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift) , 2018, NeurIPS.

[39]  A. Ramazani,et al.  Quantification of complex-phase steel microstructure by using combined EBSD and EPMA measurements , 2018, Materials Characterization.

[40]  Wenyu Liu,et al.  Weakly-Supervised Semantic Segmentation Network with Deep Seeded Region Growing , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[41]  Lars Petersson,et al.  Effective Use of Synthetic Data for Urban Scene Semantic Segmentation , 2018, ECCV.

[42]  Swami Sankaranarayanan,et al.  Learning from Synthetic Data: Addressing Domain Shift for Semantic Segmentation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[43]  Patrick Pérez,et al.  ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Elizabeth A. Holm,et al.  High Throughput Quantitative Metallography for Complex Microstructures Using Deep Learning: A Case Study in Ultrahigh Carbon Steel , 2018, Microscopy and Microanalysis.

[45]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[46]  Sungroh Yoon,et al.  FickleNet: Weakly and Semi-Supervised Semantic Image Segmentation Using Stochastic Inference , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Ashley D. Spear,et al.  Predicting Microstructure-Sensitive Fatigue-Crack Path in 3D Using a Machine Learning Framework , 2019, JOM.

[48]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[49]  Thomas Straub,et al.  Automated Quantitative Analyses of Fatigue-Induced Surface Damage by Deep Learning , 2020, Materials.

[50]  Zhengyang Wang,et al.  Pixel Transposed Convolutional Networks , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[51]  Alexandr A. Kalinin,et al.  Albumentations: fast and flexible image augmentations , 2018, Inf..

[52]  Thorsten Staudt,et al.  Classification of Bainitic Structures Using Textural Parameters and Machine Learning Techniques , 2020 .

[53]  Carl F. Sabottke,et al.  The Effect of Image Resolution on Deep Learning in Radiography. , 2020, Radiology. Artificial intelligence.

[54]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[55]  A. Yuille,et al.  Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation , 2020, ECCV.

[56]  Qiang Wang,et al.  Tracking-by-Fusion via Gaussian Process Regression Extended to Transfer Learning , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[57]  Ganapathy Krishnamurthi,et al.  Demystifying Brain Tumor Segmentation Networks: Interpretability and Uncertainty Analysis , 2020, Frontiers in Computational Neuroscience.

[58]  Martin Müller,et al.  Segmentation of Lath-Like Structures via Localized Identification of Directionality in a Complex-Phase Steel , 2020, Metallography, Microstructure, and Analysis.

[59]  Willem Waegeman,et al.  Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods , 2019, Machine Learning.