The Adoption of Image-Driven Machine Learning for Microstructure Characterization and Materials Design: A Perspective

The recent surge in the adoption of machine learning techniques for materials design, discovery, and characterization has resulted in increased interest in and application of image-driven machine learning (IDML) approaches. In this work, we review the application of IDML to the field of materials characterization. A hierarchy of six action steps is defined that compartmentalizes a problem statement into well-defined modules. The studies reviewed in this work are analyzed through the decisions adopted in them at each of these steps. Such a review permits a granular assessment of the field, for example, the impact of IDML on materials characterization at the nanoscale, the number of images in a typical dataset required to train a semantic segmentation model on electron microscopy images, the prevalence of transfer learning in the domain, etc. Finally, we discuss the importance of interpretability and explainability, and provide an overview of two emerging techniques in the field: semantic segmentation and generative adversarial networks.

[1]  H. Sebastian Seung,et al.  Trainable Weka Segmentation: a machine learning tool for microscopy pixel classification , 2017, Bioinform..

[2]  Quanshi Zhang,et al.  Visual interpretability for deep learning: a survey , 2018, Frontiers of Information Technology & Electronic Engineering.

[3]  Joshua B. Tenenbaum,et al.  Human-level concept learning through probabilistic program induction , 2015, Science.

[4]  Sidnei Paciornik,et al.  Semantic segmentation of the micro-structure of strain-hardening cement-based composites (SHCC) by applying deep learning on micro-computed tomography scans , 2020 .

[5]  Stylianos Tsopanidis,et al.  Toward quantitative fractography using convolutional neural networks , 2019, Engineering Fracture Mechanics.

[6]  Fang Liu,et al.  Microstructure Instance Segmentation from Aluminum Alloy Metallographic Image Using Different Loss Functions , 2020, Symmetry.

[7]  Shashi Phoha,et al.  Application of supervised machine learning for defect detection during metallic powder bed fusion additive manufacturing using high resolution imaging. , 2018 .

[8]  K. Müller,et al.  Towards exact molecular dynamics simulations with machine-learned force fields , 2018, Nature Communications.

[9]  J. Saal,et al.  Element redistributions during early stages of oxidation in a Ni38Cr22Fe20Mn10Co10 multi-principal element alloy , 2021 .

[10]  Prasanna Balaprakash,et al.  Phase Segmentation in Atom-Probe Tomography Using Deep Learning-Based Edge Detection , 2019, Scientific Reports.

[11]  Elizabeth A. Holm,et al.  A computer vision approach for automated analysis and classification of microstructural image data , 2015 .

[12]  Jian Ma,et al.  Rotation equivariant and invariant neural networks for microscopy image analysis , 2019, Bioinform..

[13]  Michael I. Jordan,et al.  Advances in Neural Information Processing Systems 30 , 1995 .

[14]  Hang Z. Yu,et al.  Quantitative microstructure analysis for solid-state metal additive manufacturing via deep learning , 2020, Journal of Materials Research.

[15]  Anne E. McMills.  Tech , 2021, The Assistant Lighting Designer's Toolkit.

[16]  F. C. Thompson,et al.  A History of Metallography , 1961, Nature.

[17]  Wei-keng Liao,et al.  Deep learning approaches for mining structure-property linkages in high contrast composites from simulation datasets , 2018 .

[18]  Rama Vasudevan,et al.  Deep Learning of Atomically Resolved Scanning Transmission Electron Microscopy Images: Chemical Identification and Tracking Local Transformations. , 2017, ACS nano.

[19]  Wenxiang Xu,et al.  Hierarchical n-point polytope functions for quantitative representation of complex heterogeneous materials and microstructural evolution , 2019, Acta Materialia.

[20]  A. Choudhary,et al.  Perspective: Materials informatics and big data: Realization of the “fourth paradigm” of science in materials science , 2016 .

[21]  Chiho Kim,et al.  Machine learning in materials informatics: recent applications and prospects , 2017, npj Computational Materials.

[22]  Aydogan Ozcan,et al.  Resolution enhancement in scanning electron microscopy using deep learning , 2019, Scientific Reports.

[23]  Salvatore Torquato,et al.  Microstructure of two‐phase random media. I. The n‐point probability functions , 1982 .

[24]  Elizabeth A. Holm,et al.  High Throughput Quantitative Metallography for Complex Microstructures Using Deep Learning: A Case Study in Ultrahigh Carbon Steel , 2018, Microscopy and Microanalysis.

[25]  Matthias Weber,et al.  Machine Learning Techniques for the Segmentation of Tomographic Image Data of Functional Materials , 2019, Front. Mater..

[26]  P. Murray,et al.  New methods for automatic quantification of microstructural features using digital image processing , 2018 .

[27]  Wei Chen,et al.  A Machine Learning-Based Design Representation Method for Designing Heterogeneous Microstructures , 2014, DAC 2014.

[28]  Levent Sagun,et al.  Energy landscapes for machine learning. , 2017, Physical chemistry chemical physics : PCCP.

[29]  Min Song,et al.  An approach for the aging process optimization of Al–Zn–Mg–Cu series alloys , 2009 .

[30]  Jiaying Gao,et al.  Stochastic nonlinear analysis of unidirectional fiber composites using image-based microstructural uncertainty quantification , 2021 .

[31]  David L. McDowell,et al.  The materials innovation ecosystem: A key enabler for the Materials Genome Initiative , 2016 .

[32]  S. Choi,et al.  Integrated application of semantic segmentation-assisted deep learning to quantitative multi-phased microstructural analysis in composite materials: Case study of cathode composite materials of solid oxide fuel cells , 2020 .

[33]  Badri Narayanan,et al.  Machine learning enabled autonomous microstructural characterization in 3D samples , 2020, npj Computational Materials.

[34]  Jesse Ward,et al.  Use of neural networks to analyze pulse shape data in low-background detectors , 2018, Journal of Radioanalytical and Nuclear Chemistry.

[35]  D. Alloyeau,et al.  A deep learning approach for determining the chiral indices of carbon nanotubes from high-resolution transmission electron microscopy images , 2020, Carbon.

[36]  Peter W. Voorhees,et al.  Optimizing convolutional neural networks to perform semantic segmentation on large materials imaging datasets: X-ray tomography and serial sectioning , 2020, Materials Characterization.

[37]  D. Morgan,et al.  Opportunities and Challenges for Machine Learning in Materials Science , 2020, Annual Review of Materials Research.

[38]  Sergei V. Kalinin,et al.  Big-deep-smart data in imaging for guiding materials design. , 2015, Nature materials.

[39]  A. McCallum,et al.  Materials Synthesis Insights from Scientific Literature via Text Extraction and Machine Learning , 2017 .

[40]  Dali Wang,et al.  Toward Large-Scale Image Segmentation on Summit , 2020, ICPP.

[41]  Yuksel C. Yabansu,et al.  Establishing structure-property localization linkages for elastic deformation of three-dimensional high contrast composites using deep learning approaches , 2019, Acta Materialia.

[42]  Kevin Kaufmann,et al.  Crystal symmetry determination in electron diffraction using machine learning , 2020, Science.

[43]  Yi Liu,et al.  High-throughput experiments facilitate materials innovation: A review , 2019, Science China Technological Sciences.

[44]  Danna Zhou,et al.  d. , 1840, Microbial pathogenesis.

[45]  Anubhav Jain,et al.  Harnessing the Materials Project for machine-learning and accelerated discovery , 2018, MRS Bulletin.

[46]  Claudia Draxl,et al.  NOMAD: The FAIR concept for big data-driven materials science , 2018, MRS Bulletin.

[47]  Sergei V. Kalinin,et al.  Towards data-driven next-generation transmission electron microscopy , 2020, Nature Materials.

[48]  Tolga Tasdizen,et al.  Decoding crystallography from high-resolution electron imaging and diffraction datasets with deep learning , 2019, Science Advances.

[49]  Robert Hull,et al.  Adaptive characterization of microstructure dataset using a two stage machine learning approach , 2020 .

[50]  Wing Kam Liu,et al.  Computational microstructure characterization and reconstruction: Review of the state-of-the-art techniques , 2018, Progress in Materials Science.

[51]  Simone Bianco,et al.  Active deep learning reduces annotation burden in automatic cell segmentation , 2017, bioRxiv.

[52]  Elizabeth Kautz,et al.  Rapid and Flexible Semantic Segmentation of Electron Microscopy Data Using Few-Shot Machine Learning , 2021 .

[53]  Nicholas Lubbers,et al.  Inferring low-dimensional microstructure representations using convolutional neural networks , 2016, Physical review. E.

[54]  장윤희,et al.  Y. , 2003, Industrial and Labor Relations Terms.

[55]  Hidenori Terasaki,et al.  Microstructural diagram for steel based on crystallography with machine learning , 2019, Computational Materials Science.

[56]  Sergei V. Kalinin,et al.  Materials informatics: From the atomic-level to the continuum , 2019, Acta Materialia.

[57]  Ole Winther,et al.  A Deep Learning Approach to Identify Local Structures in Atomic‐Resolution Transmission Electron Microscopy Images , 2018, Advanced Theory and Simulations.

[58]  W. Park,et al.  Classification of crystal structure using a convolutional neural network , 2017, IUCrJ.

[59]  Fan Zhang,et al.  Slice-to-voxel stochastic reconstructions on porous media with hybrid deep generative model , 2021 .

[60]  Juhan Nam,et al.  Multimodal Deep Learning , 2011, ICML.

[61]  Cormac Toher,et al.  Data-driven design of inorganic materials with the Automatic Flow Framework for Materials Discovery , 2018, MRS Bulletin.

[62]  Erin Antono,et al.  Building Data-driven Models with Microstructural Images: Generalization and Interpretability , 2017, ArXiv.

[63]  Antonio-José Almeida,et al.  NAT , 2019, Springer Reference Medizin.

[64]  Ying Shirley Meng,et al.  Three-dimensional nanoscale characterisation of materials by atom probe tomography , 2018 .

[65]  Wei Chen,et al.  Descriptor-based methodology for statistical characterization and 3D reconstruction of microstructural materials , 2014 .

[66]  Brian Hutchinson,et al.  Deep Learning for Semantic Segmentation of Defects in Advanced STEM Images of Steels , 2019, Scientific Reports.

[67]  Wuyin Jin,et al.  Quantitative Metallographic Analysis of GCr15 Microstructure Using Mask R-CNN , 2020 .

[68]  Alberto Santamaria-Pang,et al.  Towards Emergent Language Symbolic Semantic Segmentation and Model Interpretability , 2020, MICCAI.

[69]  J. Behler,et al.  Construction of high-dimensional neural network potentials using environment-dependent atom pairs. , 2012, The Journal of chemical physics.

[70]  B. Yener,et al.  Image-driven discriminative and generative machine learning algorithms for establishing microstructure-processing relationships , 2020, ArXiv.

[71]  Daniel Lewis,et al.  An image-driven machine learning approach to kinetic modeling of a discontinuous precipitation reaction , 2019, Materials Characterization.

[72]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[73]  Elizabeth Kautz,et al.  A machine learning approach to thermal conductivity modeling: A case study on irradiated uranium-molybdenum nuclear fuels , 2018, Computational Materials Science.

[74]  P. Voorhees,et al.  Quantitative serial sectioning analysis , 2001, Journal of microscopy.

[75]  William J. Emery,et al.  Active Learning Methods for Remote Sensing Image Classification , 2009, IEEE Transactions on Geoscience and Remote Sensing.

[76]  Baskar Ganapathysubramanian,et al.  Interpretable deep learning for guided microstructure-property explorations in photovoltaics , 2018, npj Computational Materials.

[77]  M. Nardelli,et al.  An efficient and accurate framework for calculating lattice thermal conductivity of solids: AFLOW—AAPL Automatic Anharmonic Phonon Library , 2017, npj Computational Materials.

[78]  P. Alam.  ‘A’ , 2021, Composites Engineering: An A–Z Guide.

[79]  Ye Wei,et al.  3D nanostructural characterisation of grain boundaries in atom probe data utilising machine learning methods , 2019, PloS one.

[80]  Noah H. Paulson,et al.  Reduced-order structure-property linkages for polycrystalline microstructures based on 2-point statistics , 2017 .

[81]  P. Alam.  ‘L’ , 2021, Composites Engineering: An A–Z Guide.

[82]  Surya R. Kalidindi,et al.  Microstructure informatics using higher-order statistics and efficient data-mining protocols , 2011 .

[83]  Huan Wang.  Material image segmentation based on feature similarity and non-spherical clustering , 2019, IOP Conference Series: Materials Science and Engineering.

[84]  Jack Beuth,et al.  Using machine learning to identify in-situ melt pool signatures indicative of flaw formation in a laser powder bed fusion additive manufacturing process , 2019, Additive Manufacturing.

[85]  Mario Fritz,et al.  Advanced Steel Microstructural Classification by Deep Learning Methods , 2017, Scientific Reports.

[86]  Noam Bernstein,et al.  Machine learning unifies the modeling of materials and molecules , 2017, Science Advances.

[87]  Wanbo Liu,et al.  Deep Learning-Based Image Segmentation for Al-La Alloy Microscopic Images , 2018, Symmetry.

[88]  G. Requena,et al.  Semantic segmentation of synchrotron tomography of multiphase Al-Si alloys using a convolutional neural network with a pixel-wise weighted loss function , 2019, Scientific Reports.

[89]  Ruimao Zhang,et al.  Cost-Effective Active Learning for Deep Image Classification , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[90]  S. Kalidindi,et al.  Novel microstructure quantification framework for databasing, visualization, and analysis of microstructure data , 2013, Integrating Materials and Manufacturing Innovation.

[91]  Yanan Zhu,et al.  A deep convolutional neural network approach to single-particle recognition in cryo-electron microscopy , 2016, BMC Bioinformatics.

[92]  Daniel W. Davies,et al.  Machine learning for molecular and materials science , 2018, Nature.

[93]  Yue Liu,et al.  Materials discovery and design using machine learning , 2017 .

[94]  Lehan Yao,et al.  Machine Learning to Reveal Nanoparticle Dynamics from Liquid-Phase TEM Videos , 2020, ACS central science.

[95]  P. Alam.  ‘S’ , 2021, Composites Engineering: An A–Z Guide.

[96]  Elizabeth A. Holm,et al.  Computer Vision and Machine Learning for Autonomous Characterization of AM Powder Feedstocks , 2016, JOM.

[97]  Atsuto Seko,et al.  Progress in nanoinformatics and informational materials science , 2018, MRS Bulletin.

[98]  Zhongxiao Peng,et al.  Machine-learning assisted laser powder bed fusion process optimization for AlSi10Mg: New microstructure description indices and fracture mechanisms , 2020 .

[99]  AutoDetect-mNP: An Unsupervised Machine Learning Algorithm for Automated Analysis of Transmission Electron Microscope Images of Metal Nanoparticles , 2021, JACS Au.

[100]  P. Alam.  ‘T’ , 2021, Composites Engineering: An A–Z Guide.

[101]  Bülent Yener,et al.  Image driven machine learning methods for microstructure recognition , 2016 .

[102]  A. Raman,et al.  Elucidating the Behavior of Nanophotonic Structures through Explainable Machine Learning Algorithms , 2020, ACS Photonics.

[103]  G. Bruno,et al.  Advanced Deep Learning‐Based 3D Microstructural Characterization of Multiphase Metal Matrix Composites , 2020, Advanced Engineering Materials.

[104]  H. Abrams.  Grain size measurement by the intercept method , 1971 .

[105]  Sergei V. Kalinin,et al.  Correlated Materials Characterization via Multimodal Chemical and Functional Imaging. , 2018, ACS nano.

[106]  Rahul Rao,et al.  Autonomy in materials research: a case study in carbon nanotube growth , 2016 .

[107]  Gongping Yang,et al.  Hierarchical level features based trainable segmentation for electron microscopy images , 2013, BioMedical Engineering OnLine.