Artificial-intelligence-driven scanning probe microscopy

Scanning probe microscopy (SPM) has revolutionized the fields of materials, nano-science, chemistry, and biology, by enabling mapping of surface properties and surface manipulation with atomic precision. However, these achievements require constant human supervision; fully automated SPM has not been accomplished yet. Here we demonstrate an artificial intelligence framework based on machine learning for autonomous SPM operation (DeepSPM). DeepSPM includes an algorithmic search of good sample regions, a convolutional neural network to assess the quality of acquired images, and a deep reinforcement learning agent to reliably condition the state of the probe. DeepSPM is able to acquire and classify data continuously in multi-day scanning tunneling microscopy experiments, managing the probe quality in response to varying experimental conditions. Our approach paves the way for advanced methods hardly feasible by human operation (e.g., large dataset acquisition and SPM-based nanolithography). DeepSPM can be generalized to most SPM techniques, with the source code publicly available. Enabling atomic-precision mapping and manipulation of surfaces, scanning probe microscopy requires constant human supervision to assess image quality and probe conditions. Here, the authors demonstrate DeepSPM, a machine learning approach allowing to acquire and classify data autonomously in multi-day Scanning Tunnelling Microscopy experiments.

[1]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[2]  G. Binnig,et al.  Scanning tunneling microscopy-from birth to adolescence , 1987 .

[3]  Robert Tibshirani,et al.  An Introduction to the Bootstrap , 1994 .

[4]  M F Crommie,et al.  Confinement of Electrons to Quantum Corrals on a Metal Surface , 1993, Science.

[5]  J. Villarrubia.  Algorithms for Scanned Probe Microscope Image Simulation, Surface Reconstruction, and Tip Estimation , 1997, Journal of research of the National Institute of Standards and Technology.

[6]  A. Rinzler,et al.  Electronic structure of atomically resolved carbon nanotubes , 1998, Nature.

[7]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[8]  Peter Liljeroth,et al.  Amplifying the Pacific Climate System Response to a Small 11-Year Solar Cycle Forcing , 2009, Science.

[9]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[10]  M. Rao,et al.  Three decades of scanning tunnelling microscopy that changed the course of surface science , 2011 .

[11]  D. Sánchez-Portal,et al.  Atomic-scale engineering of electrodes for single-molecule contacts. , 2011, Nature nanotechnology.

[12]  F. Giessibl,et al.  Revealing the Angular Symmetry of Chemical Bonds by Atomic Force Microscopy , 2012, Science.

[13]  L. Kantorovich,et al.  Erratum: Precise Orientation of a Single C-60 Molecule on the Tip of a Scanning Probe Microscope (vol 108, 268302, 2012) , 2012 .

[14]  Leo Gross,et al.  Imaging the charge distribution within a single molecule. , 2012, Nature nanotechnology.

[15]  L. Kantorovich,et al.  Precise orientation of a single C60 molecule on the tip of a scanning probe microscope. , 2012, Physical review letters.

[16]  J. Barth,et al.  Assessment of Scanning Tunneling Spectroscopy Modes Inspecting Electron Confinement in Surface-Confined Supramolecular Networks , 2013, Scientific Reports.

[17]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[18]  FIM tips in SPM: Apex orientation and temperature considerations on atom transfer and diffusion , 2013, 1308.0714.

[19]  Johannes V. Barth,et al.  CORRIGENDUM: Assessment of Scanning Tunneling Spectroscopy Modes Inspecting Electron Confinement in Surface-Confined Supramolecular Networks , 2013, Scientific Reports.

[20]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[21]  J. Barth,et al.  Porphyrins at interfaces. , 2015, Nature chemistry.

[22]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[23]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[24]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[25]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[26]  Pieter Abbeel,et al.  Benchmarking Deep Reinforcement Learning for Continuous Control , 2016, ICML.

[27]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[28]  Arindam Mandal,et al.  Multi-Task Learning and Weighted Cross-Entropy for DNN-Based Keyword Spotting , 2016, INTERSPEECH.

[29]  Li Wang,et al.  Conformation-based signal transfer and processing at the single-molecule level. , 2017, Nature nanotechnology.

[30]  Leo Gross,et al.  Generation, manipulation and characterization of molecules by atomic force microscopy , 2017 .

[31]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[32]  K. M. Bastiaans,et al.  Robust procedure for creating and characterizing the atomic structure of scanning tunneling microscope tips , 2017, Beilstein journal of nanotechnology.

[33]  C. Gerber,et al.  Imaging modes of atomic force microscopy for application in molecular and cell biology. , 2017, Nature nanotechnology.

[34]  Mohammad Rashidi,et al.  Autonomous Scanning Probe Microscopy in Situ Tip Conditioning through Machine Learning. , 2018, ACS nano.

[35]  A. Extance.  How atomic imaging is being pushed to its limit , 2018, Nature.

[36]  Leonidas J. Guibas,et al.  Taskonomy: Disentangling Task Transfer Learning , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[37]  K. Rusimova,et al.  Regulating the femtosecond excited-state lifetime of a single molecule , 2018, Science.

[38]  M. Rashidi,et al.  Lithography for robust and editable atomic-scale silicon devices and memories , 2018, Nature Communications.

[39]  Loic A. Royer,et al.  Content-aware image restoration: pushing the limits of fluorescence microscopy , 2018, Nature Methods.

[40]  Jing Wang,et al.  Direct imaging of surface states hidden in the third layer of Si (111)-7 × 7 surface bypz-wave tip , 2018, Applied Physics Letters.

[41]  Boyuan Huang,et al.  Artificial Intelligent Atomic Force Microscope Enabled by Machine Learning , 2018, 1807.09985.

[42]  A. Extance.  How atomic imaging is being pushed to its limit. , 2018 .

[43]  Zhenghao Li,et al.  An artificial intelligence atomic force microscope enabled by machine learning. , 2018, Nanoscale.

[44]  O. Gordon,et al.  Scanning Probe State Recognition With Multi-Class Neural Network Ensembles , 2019, Review of Scientific Instruments.

[45]  O. Gordon,et al.  Embedding human heuristics in machine-learning-enabled probe microscopy , 2019, Mach. Learn. Sci. Technol..

[46]  Ernst Meyer,et al.  Scanning Probe Microscopy: The Lab on a Tip , 2021 .