A Survey on Evaluation Metrics for Synthetic Material Micro-Structure Images from Generative Models

The evaluation of synthetic micro-structure images is an emerging problem as machine learning and materials science research have evolved together. Typical state of the art methods in evaluating synthetic images from generative models have relied on the Fr\'echet Inception Distance. However, this and other similar methods, are limited in the materials domain due to both the unique features that characterize physically accurate micro-structures and limited dataset sizes. In this study we evaluate a variety of methods on scanning electron microscope (SEM) images of graphene-reinforced polyurethane foams. The primary objective of this paper is to report our findings with regards to the shortcomings of existing methods so as to encourage the machine learning community to consider enhancements in metrics for assessing quality of synthetic images in the material science domain.

[1]  B. Sumpter,et al.  Deep Generative Models for Materials Discovery and Machine Learning-Accelerated Innovation , 2022, Frontiers in Materials.

[2]  Litao Chen,et al.  Generative models for inverse design of inorganic solid materials , 2021, Journal of Materials Informatics.

[3]  Luis Caicedo Torres,et al.  A Survey on Optimal Transport for Machine Learning: Theory and Applications , 2021, ArXiv.

[4]  Ali Borji,et al.  Pros and Cons of GAN Evaluation Measures: New Developments , 2021, Comput. Vis. Image Underst..

[5]  Eero P. Simoncelli,et al.  Comparison of Full-Reference Image Quality Models for Optimization of Image Processing Systems , 2021, International Journal of Computer Vision.

[6]  Song Han,et al.  Differentiable Augmentation for Data-Efficient GAN Training , 2020, NeurIPS.

[7]  Tero Karras,et al.  Training Generative Adversarial Networks with Limited Data , 2020, NeurIPS.

[8]  Seong Joon Oh,et al.  Reliable Fidelity and Diversity Metrics for Generative Models , 2020, ICML.

[9]  Thomas Lukasiewicz,et al.  ManiGAN: Text-Guided Image Manipulation , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Brian Gallagher,et al.  Predicting compressive strength of consolidated molecular solids using computer vision and deep learning , 2019, Materials & Design.

[11]  Jaakko Lehtinen,et al.  Improved Precision and Recall Metric for Assessing Generative Models , 2019, NeurIPS.

[12]  Michael S. Bernstein,et al.  HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models , 2019, NeurIPS.

[13]  Andreas Rössler,et al.  FaceForensics++: Learning to Detect Manipulated Facial Images , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[14]  Baskar Ganapathysubramanian,et al.  Physics-aware Deep Generative Models for Creating Synthetic Microstructures , 2018, ArXiv.

[15]  Mike Marsh,et al.  Dragonfly as a Platform for Easy Image-based Deep Learning Applications , 2018, Microscopy and Microanalysis.

[16]  Olivier Bachem,et al.  Assessing Generative Models via Precision and Recall , 2018, NeurIPS.

[17]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[18]  Arthur Gretton,et al.  Demystifying MMD GANs , 2018, ICLR.

[19]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[20]  Serge J. Belongie,et al.  Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[21]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[22]  Alan C. Bovik,et al.  Blind/Referenceless Image Spatial Quality Evaluator , 2011, 2011 Conference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers (ASILOMAR).

[23]  Alemayehu S. Admasu,et al.  Deep Learning Image-based Design of Graphene-reinforced Polyurethane Foams , 2021 .

[24]  Paolo Soda,et al.  Evaluating GANs in Medical Imaging , 2021, DGM4MICCAI/DALI@MICCAI.