Reproducible Econometric Simulations

Abstract Reproducibility of economic research has attracted considerable attention in recent years. So far, the discussion has focused mainly on reproducibility of empirical analyses. This paper addresses a further aspect of reproducibility, the reproducibility of computational experiments. More specifically, we contribute to the emerging literature on reproducibility in economics along three lines: (i) we document how simulations of various types are an integral part of publications in modern econometrics, (ii) we provide some general guidelines about how to set up reproducible simulation experiments, and, finally, (iii) we provide a case study from time series econometrics that illustrates the main issues arising in connection with reproducibility, emphasizing the use of modular tools.

[1]  Matthias Sutter,et al.  Working Papers in Economics and Statistics Gender differences in competition emerge early in life , 2010 .

[2]  Matthias Sutter,et al.  Social Preferences in Childhood and Adolescence: A Large-Scale Experiment , 2010, SSRN Electronic Journal.

[3]  H. Vinod,et al.  Verifying the Solution from a Nonlinear Solver: A Case Study , 2004 .

[4]  Teresa D. Harrison,et al.  Do Economics Journal Archives Promote Replicable Research? , 2006 .

[5]  Rodney Sparapani,et al.  Random Number Generation and Monte Carlo Methods (2nd edition) , 2004 .

[6]  Matthias Sutter,et al.  Working Papers in Economics and Statistics Group polarization in the team dictator game reconsidered , 2007 .

[7]  M. Gächter,et al.  Convergence of the Health Status at the Local Level: Empirical Evidence from Austria , 2010 .

[8]  John M. Chambers,et al.  Software for data analysis , 2008 .

[9]  Thomas J. Santner,et al.  The Design and Analysis of Computer Experiments , 2003, Springer Series in Statistics.

[10]  Engelbert Theurl,et al.  Working Papers in Economics and Statistics Stronger Sex but Earlier Death: a Multi-level Socioeconomic Analysis of Gender Differences in Mortality in Austria Stronger Sex but Earlier Death: a Multi-level Socioeconomic Analysis of Gender Differences in Mortality in Austria , 2022 .

[11]  Josef Baumgartner,et al.  Milking The Prices: The Role of Asymmetries in the Price Transmission Mechanism for Milk Products in Austria , 2010 .

[12]  Walter Krämer,et al.  The CUSUM test for OLS residuals , 1992 .

[13]  P. Raschky,et al.  Uncertainty of Governmental Relief and the Crowding out of Flood Insurance , 2013 .

[14]  John M. Mulvey,et al.  On Reporting Computational Experiments with Mathematical Software , 1979, TOMS.

[15]  Takuji Nishimura,et al.  Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator , 1998, TOMC.

[16]  Matthias Sutter,et al.  Teams Make You Smarter: Learning and Knowledge Transfer in Auctions and Markets by Teams and Individuals , 2010, SSRN Electronic Journal.

[17]  Michael Kirchler,et al.  Trading strategies and trading profits in experimental asset markets with cumulative information , 2010 .

[18]  Engelbert Theurl,et al.  Socioeconomic Environment and Mortality: A two-level Decomposition by Sex and Cause of Death , 2010 .

[19]  P. Hellekalek Good random number generators are (not so) easy to find , 1998 .

[20]  M. Gächter,et al.  Retaining the thin blue line: What shapes workers' intentions not to quit the current work environment , 2013 .

[21]  B. Hansen Approximate Asymptotic P Values for Structural-Change Tests , 1997 .

[22]  Pierre L’Ecuyer,et al.  Random Number Generation , 2008, Encyclopedia of Algorithms.

[23]  Stefan Lang,et al.  Working Papers in Economics and Statistics Applications of Multilevel Structured Additive Regression Models to Insurance Data Applications of Multilevel Structured Additive Regression Models to Insurance Data , 2022 .

[24]  Gabor Grothendieck,et al.  Lattice: Multivariate Data Visualization with R , 2008 .

[25]  Stefan Lang,et al.  Working Papers in Economics and Statistics Modeling House Prices Using Multilevel Structured Additive Regression Modeling House Prices Using Multilevel Structured Additive Regression , 2022 .

[26]  Matthias Sutter,et al.  Social preferences during childhood and the role of gender and age -- An experiment in Austria and Sweden , 2011 .

[27]  Jesús Crespo-Cuaresma,et al.  Business cycle convergence in EMU : A first look at the second moment * , 2010 .

[28]  Craig B. Borkowf,et al.  Random Number Generation and Monte Carlo Methods , 2000, Technometrics.

[29]  Matthias Sutter,et al.  Equality, Equity and Incentives: An Experiment , 2013, SSRN Electronic Journal.

[30]  Matthias Sutter,et al.  Strategic Sophistication of Adolescents: Evidence from Experimental Normal-Form Games , 2010, SSRN Electronic Journal.

[31]  Achim Zeileis,et al.  On reproducible econometric research , 2009 .

[32]  James G. MacKinnon,et al.  Numerical Distribution Functions of Likelihood Ratio Tests for Cointegration , 1996 .

[33]  Michael Pinedo,et al.  GUIDELINES FOR REPORTING COMPUTATIONAL RESULTS IN IIE TRANSACTIONS , 1993 .

[34]  Stephen G. Nash,et al.  Guidelines for reporting results of computational experiments. Report of the ad hoc committee , 1991, Math. Program..

[35]  Bruce E. Hansen,et al.  Testing for parameter instability in linear models , 1992 .

[36]  Achim Zeileis,et al.  Strucchange: An R package for testing for structural change in linear regression models , 2002 .

[37]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[38]  J. Nyblom Testing for the Constancy of Parameters over Time , 1989 .

[39]  Deepayan Sarkar,et al.  Lattice: Multivariate Data Visualization with R , 2008 .

[40]  Ruth M. Pfeiffer,et al.  Working Papers in Economics and Statistics Comparing Penalized Splines and Fractional Polynomials for Flexible Modelling of the Effects of Continuous Predictor Variables , 2010 .

[41]  Matthias Sutter,et al.  Gender, Competition and the Efficiency of Policy Interventions , 2010, SSRN Electronic Journal.

[42]  J. Gentle Random number generation and Monte Carlo methods , 1998 .

[43]  Mauricio G. C. Resende,et al.  Designing and reporting on computational experiments with heuristic methods , 1995, J. Heuristics.

[44]  M. Sutter,et al.  Strategic Sophistication of Individuals and Teams in Experimental Normal-Form Games , 2010, SSRN Electronic Journal.

[45]  P. Raschky,et al.  Working Papers in Economics and Statistics Uncertainty of Governmental Relief and the Crowding out of Insurance Uncertainty of Governmental Relief and the Crowding out of Insurance , 2022 .

[46]  Achim Zeileis,et al.  A new method for detecting differential item functioning in the Rasch model , 2011 .

[47]  George L. Nemhauser,et al.  Handbooks in operations research and management science , 1989 .

[48]  G. Civille,et al.  Guidelines for Reporting Results , 2015 .

[49]  Francesco Feri,et al.  Psychological Pressure in Competitive Environments: Evidence from A Randomized Natural Experiment: Comment , 2011 .

[50]  David C. Hoaglin,et al.  The Reporting of Computation-Based Results in Statistics , 1975 .

[51]  David Huffman,et al.  University of Innsbruck Working Papers in Economics and Statistics Group Membership , Competition , and Altruistic versus Antisocial Punishment : Evidence from Randomly Assigned Army Groups , 2010 .

[52]  P. L’Ecuyer Random Number Generation , 2012 .

[53]  W. Greene,et al.  The role of data/code archives in the future of economic research , 2008 .

[54]  Frank Schorfheide,et al.  Estimation with overidentifying inequality moment conditions , 2009 .

[55]  Murali Haran,et al.  Markov chain Monte Carlo: Can we trust the third significant figure? , 2007, math/0703746.

[56]  M. Sutter,et al.  Working Papers in Economics and Statistics Guilt from Promise-breaking and Trust in Markets for Expert Services -theory and Experiment Guilt from Promise-breaking and Trust in Markets for Expert Services – Theory and Experiment * , 2022 .

[57]  Matthias Sutter,et al.  University of Innsbruck Working Papers in Economics and Statistics Household Decision Making in Rural China : Using Experiments to Estimate the Influences of Spouses , 2010 .

[58]  Crespo Cuaresma,et al.  Octavio Fernandez-Amador Business cycle convergence in EMU : A second look at the second moment , 2010 .

[59]  Matthias Sutter,et al.  Psychological Pressure in Competitive Environments: New Evidence from Randomized Natural Experiments , 2012, Manag. Sci..

[60]  Brian D. Ripley,et al.  Thoughts on pseudorandom number generators , 1990 .

[61]  M. Hanke,et al.  Football championships and jersey sponsors’ stock prices: an empirical investigation , 2013 .