Toward Benchmarking in Catalysis Science: Best Practices, Challenges, and Opportunities

Benchmarking is a community-based and (preferably) community-driven activity involving consensus-based decisions on how to make reproducible, fair, and relevant assessments. In catalysis science, important catalyst performance metrics include activity, selectivity, and the deactivation profile, which enable comparisons between new and standard catalysts. Benchmarking also requires careful documentation, archiving, and sharing of methods and measurements, to ensure that the full value of research data can be realized. Beyond these goals, benchmarking presents unique opportunities to advance and accelerate understanding of complex reaction systems by combining and comparing experimental information from multiple, in situ and operando techniques with theoretical insights derived from calculations characterizing model systems. This Perspective describes the origins and uses of benchmarking and its applications in computational catalysis, heterogeneous catalysis, molecular catalysis, and electrocatalysis. It a...

[1]  L. Cavallo,et al.  Selectivity in propene polymerization with metallocene catalysts. , 2000, Chemical reviews.

[2]  J. Wagner,et al.  Catalysts under Controlled Atmospheres in the Transmission Electron Microscope , 2014 .

[3]  Michel Dupuis,et al.  Homogeneous Ni catalysts for H2 oxidation and production: an assessment of theoretical methods, from density functional theory to post Hartree-Fock correlated wave-function theory. , 2010, The journal of physical chemistry. A.

[4]  G. Ertl,et al.  Handbook of Heterogeneous Catalysis , 1997 .

[5]  Jan M. L. Martin,et al.  “Turning Over” Definitions in Catalytic Cycles , 2012 .

[6]  I. Cockburn,et al.  The Economics of Reproducibility in Preclinical Research , 2015, PLoS biology.

[7]  Charles C. L. McCrory,et al.  Benchmarking heterogeneous electrocatalysts for the oxygen evolution reaction. , 2013, Journal of the American Chemical Society.

[8]  Donald G Truhlar,et al.  Tests of the RPBE, revPBE, tau-HCTHhyb, omegaB97X-D, and MOHLYP density functional approximations and 29 others against representative databases for diverse bond energies and barrier heights in catalysis. , 2010, The Journal of chemical physics.

[9]  Charles C. L. McCrory,et al.  Benchmarking hydrogen evolving reaction and oxygen evolving reaction electrocatalysts for solar water splitting devices. , 2015, Journal of the American Chemical Society.

[10]  Vincent Artero,et al.  Toward the Rational Benchmarking of Homogeneous H2-Evolving Catalysts. , 2014, Energy & environmental science.

[11]  Donald G Truhlar,et al.  Do Practical Standard Coupled Cluster Calculations Agree Better than Kohn-Sham Calculations with Currently Available Functionals When Compared to the Best Available Experimental Data for Dissociation Energies of Bonds to 3d Transition Metals? , 2015, Journal of chemical theory and computation.

[12]  J. Savéant,et al.  Multielectron, Multistep Molecular Catalysis of Electrochemical Reactions: Benchmarking of Homogeneous Catalysts , 2014 .

[13]  Donna G Blackmond.  Reaction progress kinetic analysis: a powerful methodology for mechanistic studies of complex catalytic reactions. , 2005, Angewandte Chemie.

[14]  B. Goldsmith,et al.  Water-catalyzed activation of H2O2 by methyltrioxorhenium: a combined computational-experimental study. , 2013, Inorganic chemistry.

[15]  M. Boudart,et al.  Experimental criterion for the absence of artifacts in the measurement of rates of heterogeneous catalytic reactions , 1982 .

[16]  Susan Elliott Sim,et al.  Using benchmarking to advance research: a challenge to software engineering , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[17]  Hylke B. J. Koers,et al.  Bringing Digital Science Deep Inside the Scientific Article: the Elsevier Article of the Future Project , 2014 .

[18]  Huimin Zhao,et al.  Preparing Your Manuscript for Submission to ACS Catalysis , 2014 .

[19]  Jonathan V Sweedler,et al.  Striving for Reproducible Science. , 2015, Analytical chemistry.

[20]  G. Kroes,et al.  Toward a Database of Chemically Accurate Barrier Heights for Reactions of Molecules with Metal Surfaces. , 2015, The journal of physical chemistry letters.

[21]  David L. Donoho,et al.  WaveLab and Reproducible Research , 1995 .

[22]  S. Grimme,et al.  "Mindless" DFT Benchmarking. , 2009, Journal of chemical theory and computation.

[23]  I. Chorkendorff,et al.  Benchmarking the Stability of Oxygen Evolution Reaction Catalysts: The Importance of Monitoring Mass Losses , 2014 .

[24]  M. Vandichel,et al.  Origin of highly active metal-organic framework catalysts: defects? Defects! , 2016, Dalton transactions.

[25]  G. Lente.  Comment on "'turning over' definitions in catalytic cycles" , 2013 .

[26]  J. Savéant,et al.  Benchmarking of homogeneous electrocatalysts: overpotential, turnover frequency, limiting turnover number. , 2015, Journal of the American Chemical Society.

[27]  C. Corminboeuf,et al.  Comprehensive Benchmarking of a Density-Dependent Dispersion Correction. , 2011, Journal of chemical theory and computation.

[28]  F. Prinz,et al.  Believe it or not: how much can we rely on published data on potential drug targets? , 2011, Nature Reviews Drug Discovery.

[29]  Stefan Seritan,et al.  Rate-Enhancing Roles of Water Molecules in Methyltrioxorhenium-Catalyzed Olefin Epoxidation by Hydrogen Peroxide. , 2015, Journal of the American Chemical Society.

[30]  W. Schuhmann,et al.  Experimental Aspects in Benchmarking of the Electrocatalytic Activity , 2015 .

[31]  Christopher B. Murray,et al.  Control of Metal Nanocrystal Size Reveals Metal-Support Interface Role for Ceria Catalysts , 2013, Science.

[32]  Heather A. Piwowar,et al.  Sharing Detailed Research Data Is Associated with Increased Citation Rate , 2007, PloS one.

[33]  L. Ghiringhelli,et al.  Not so loosely bound rare gas atoms: finite-temperature vibrational fingerprints of neutral gold-cluster complexes , 2013 .

[34]  Thomas Bligaard,et al.  A benchmark database for adsorption bond energies to transition metal surfaces and comparison to selected DFT functionals , 2015 .

[35]  David L Donoho,et al.  An invitation to reproducible computational research. , 2010, Biostatistics.

[36]  Doreen Mollenhauer,et al.  A balanced procedure for the treatment of cluster–ligand interactions on gold phosphine systems in catalysis , 2014, J. Comput. Chem..

[37]  Charles T. Campbell,et al.  Finding the Rate-Determining Step in a Mechanism: Comparing DeDonder Relations with the “Degree of Rate Control” , 2001 .

[38]  Matthias Schwab,et al.  Making scientific computations reproducible , 2000, Comput. Sci. Eng..

[39]  B. Newell,et al.  Priming Intelligent Behavior: An Elusive Phenomenon , 2013, PloS one.

[40]  Walter Leitner,et al.  Continuous flow organometallic catalysis: new wind in old sails. , 2011, Chemical communications.

[41]  R. Neumann,et al.  Fine-tuning and recycling of homogeneous tungstate and polytungstate epoxidation catalysts , 2008 .

[42]  J. Nørskov,et al.  Towards the computational design of solid catalysts. , 2009, Nature chemistry.

[43]  Jennifer C Molloy,et al.  The Open Knowledge Foundation: Open Data Means Better Science , 2011, PLoS biology.

[44]  J. Hrbek,et al.  Activity of CeOx and TiOx Nanoparticles Grown on Au(111) in the Water-Gas Shift Reaction , 2007, Science.

[45]  M. Sanford,et al.  Quantitative Assay for the Direct Comparison of Platinum Catalysts in Benzene H/D Exchange , 2009 .

[46]  S. Levchenko,et al.  AuN clusters (N = 1-6) supported on MgO(100) surfaces: Effect of exact exchange and dispersion interactions on adhesion energies , 2012 .

[47]  John R. Kitchin,et al.  Examples of Effective Data Sharing in Scientific Publishing , 2015 .

[48]  Julien Farlin,et al.  How benchmarking in science can lead to a reversal of priorities. , 2015, Environmental science & technology.

[49]  W. Conner,et al.  Analysis of Catalyst Surface Structure by Physical Sorption , 2013 .

[50]  D. Dixon,et al.  Chemical accuracy in ab initio thermochemistry and spectroscopy: current strategies and future challenges , 2012, Theoretical Chemistry Accounts.

[51]  Philippe Sautet,et al.  Introducing structural sensitivity into adsorption-energy scaling relations by means of coordination numbers. , 2015, Nature chemistry.

[52]  Mark E. Davis,et al.  Fundamentals of Chemical Reaction Engineering , 2002 .

[53]  Jan H. Jensen,et al.  Predicting accurate absolute binding energies in aqueous solution: thermodynamic considerations for electronic structure methods. , 2015, Physical chemistry chemical physics : PCCP.

[54]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[55]  Christopher W. Jones.  On the Stability and Recyclability of Supported Metal–Ligand Complex Catalysts: Myths, Misconceptions and Critical Research Needs , 2010 .

[56]  Matthias Scheffler,et al.  On the accuracy of density-functional theory exchange-correlation functionals for H bonds in small water clusters: benchmarks approaching the complete basis set limit. , 2007, The Journal of chemical physics.

[57]  A. Tkatchenko,et al.  Dispersion Interactions with Density-Functional Theory: Benchmarking Semiempirical and Interatomic Pairwise Corrected Density Functionals. , 2011, Journal of chemical theory and computation.

[58]  B. Gates,et al.  Temperature-Programmed Desorption of Hydrogen from Platinum Particles on γ-Al2O3: Evidence of Platinum-Catalyzed Dehydroxylation of γ-Al2O3 , 1999 .

[59]  D. Singleton,et al.  A Case Study of the Mechanism of Alcohol-Mediated Morita Baylis–Hillman Reactions. The Importance of Experimental Observations , 2015, Journal of the American Chemical Society.

[60]  Benedikt Fecher,et al.  What Drives Academic Data Sharing? , 2014, PloS one.

[61]  E. Carter,et al.  Quantum Chemical Benchmarking, Validation, and Prediction of Acidity Constants for Substituted Pyridinium Ions and Pyridinyl Radicals. , 2012, Journal of chemical theory and computation.

[62]  P. Chirik.  Editorial: Introducing Tutorials , 2015 .

[63]  Thomas Bligaard,et al.  Density functional theory in surface chemistry and catalysis , 2011, Proceedings of the National Academy of Sciences.

[64]  David C. Forbes,et al.  Recent Advances in Asymmetric Catalytic Metal Carbene Transformations. , 1998, Chemical reviews.

[65]  D. Su,et al.  Electron microscopy of solid catalysts--transforming from a challenge to a toolbox. , 2015, Chemical reviews.

[66]  Donald G Truhlar,et al.  Benchmark Energetic Data in a Model System for Grubbs II Metathesis Catalysis and Their Use for the Development, Assessment, and Validation of Electronic Structure Methods. , 2009, Journal of chemical theory and computation.

[67]  W. Leitner,et al.  Can Contemporary Density Functional Theory Predict Energy Spans in Molecular Catalysis Accurately Enough To Be Applicable for in Silico Catalyst Design? A Computational/Experimental Case Study for the Ruthenium-Catalyzed Hydrogenation of Olefins. , 2016, Journal of the American Chemical Society.

[68]  Matthias Scheffler,et al.  Stability and metastability of clusters in a reactive atmosphere: theoretical evidence for unexpected stoichiometries of MgMOx. , 2013, Physical review letters.

[69]  H. Gasteiger,et al.  Activity benchmarks and requirements for Pt, Pt-alloy, and non-Pt oxygen reduction catalysts for PEMFCs , 2005 .

[70]  W. Kaminsky.  Highly active metallocene catalysts for olefin polymerization , 1998 .

[71]  J. Ioannidis.  Why Most Published Research Findings Are False , 2005, PLoS medicine.

[72]  Andrew G. Leach,et al.  A Standard Set of Pericyclic Reactions of Hydrocarbons for the Benchmarking of Computational Methods: The Performance of ab Initio, Density Functional, CASSCF, CASPT2, and CBS-QB3 Methods for the Prediction of Activation Barriers, Reaction Energetics, and Transition State Geometries , 2003 .

[73]  J. Nørskov,et al.  Using scaling relations to understand trends in the catalytic activity of transition metals , 2008, Journal of physics. Condensed matter : an Institute of Physics journal.

[74]  C. Begley,et al.  Drug development: Raise standards for preclinical cancer research , 2012, Nature.

[75]  A. Andreasen,et al.  Degree of rate control: how much the energies of intermediates and transition states control rates. , 2009, Journal of the American Chemical Society.