The Role of Data and Program Code Archives in the Future of Economic Research

This essay examines the role of data and program-code archives in making economic research "replicable." Replication of published results is recognized as an essential part of the scientific method. Yet, historically, both the "demand for" and "supply of" replicable results in economics has been minimal. "Respect for the scientific method" is not sufficient to motivate either economists or editors of professional journals to ensure the replicability of published results. We enumerate the costs and benefits of mandatory data and code archives, and argue that the benefits far exceed the costs. Progress has been made since the gloomy assessment of Dewald, Thursby and Anderson some twenty years ago in the American Economic Review, but much remains to be done before empirical economics ceases to be a "dismal science" when judged by the replicability of its published results.

[1]  Brandon Valeriano,et al.  Barriers to replication in systematic empirical research on world politics , 2003 .

[2]  George Tauchen.  Remarks on my Term at JBES , 1993 .

[3]  Henry E. Kilpatrick.  Some useful methods for measuring the benefits of social science research , 1998 .

[4]  John E. Hunter,et al.  CUMULATIVE RESEARCH KNOWLEDGE AND SOCIAL POLICY FORMULATION: The Critical Role of Meta-Analysis , 1996 .

[5]  D. Hamermesh.  Some thoughts on replications and reviews , 1997 .

[6]  Steven D. Levitt,et al.  Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime: Comment , 2002 .

[7]  G. King,et al.  The Future of Replication , 2003 .

[8]  David L. Donoho,et al.  WaveLab and Reproducible Research , 1995 .

[9]  James B. Boyer,et al.  Editorial statement , 2003, Nature.

[10]  D. Levy,et al.  Econometrics and the Truth-Seeking Assumption: Ethics and Research Independence , 2001 .

[11]  Karsten Boye Rasmussen,et al.  The Data Documentation Initiative , 2004 .

[12]  S. Feigenbaum,et al.  THE TECHNOLOGICAL OBSOLESCENCE OF SCIENTIFIC FRAUD , 1996 .

[13]  Giuseppe Bruno,et al.  A Comparative Study Of Alternative Econometric Packages: An Application To Italian Deposit Interest Rates , 2000 .

[14]  William L. Goffe.  The Internet and the American Economic Association , 2004 .

[15]  David M. Drukker,et al.  “Replicating the results in ‘on efficient estimation with panel data: an empirical comparison of instrumental variables estimators’ ” , 2003 .

[16]  M. Feldstein,et al.  Social Security and Private Saving: Reply , 1982, Journal of Political Economy.

[17]  Min Qi,et al.  Nonlinear Predictability of Stock Returns Using Financial and Economic Variables , 1999 .

[18]  S. Fienberg,et al.  Sharing research data , 1985 .

[19]  R. Mittelstaedt,et al.  Econometric Replication : Lessons from the Experimental Sciences , 2003 .

[20]  Teresa D. Harrison,et al.  Lessons from the JMCB Archive , 2006 .

[21]  Micah Altman,et al.  A Digital Library for the Dissemination and Replication of Quantitative Social Science Research , 2001 .

[22]  R. Engle.  GARCH 101: The Use of ARCH/GARCH Models in Applied Econometrics , 2001 .

[23]  Achim Zeileis,et al.  Validating multiple structural change models : A case study , 2005 .

[24]  A. Zellner,et al.  COMPUTATIONAL ACCURACY AND ESTIMATION OF SIMULTANEOUS EQUATION ECONOMETRIC MODELS1 , 1966 .

[25]  Stephanie W. Haas,et al.  The Role of Metadata in the Statistical Knowledge Network , 2004 .

[26]  M. Gray,et al.  The Impact of Children on the Lifetime Earnings of Australian Women: Evidence from the 1990s , 2001 .

[27]  Micah Altman,et al.  Numerical Issues in Statistical Computing for the Social Scientist , 2003 .

[28]  Ross McKitrick,et al.  Corrections to the Mann et. al. (1998) Proxy Data Base and Northern Hemispheric Average Temperature Series , 2003 .

[29]  Chris Brooks,et al.  Benchmarks and the accuracy of GARCH model estimation , 2001 .

[30]  H. Vinod,et al.  Verifying the Solution from a Nonlinear Solver: A Case Study , 2004 .

[31]  Malcolm K. Hughes,et al.  corrigendum: Global-scale temperature patterns and climate forcing over the past six centuries , 2004 .

[32]  G. Calzolari,et al.  ALTERNATIVE ESTIMATORS OF FIML COVARIANCE MATRIX: A , 1988 .

[33]  S. Feigenbaum,et al.  The market for (Ir)reproducible econometrics , 1993 .

[34]  A. Diamond.  What is a Citation Worth , 1986 .

[35]  G W Timm,et al.  Editorial Comment , 2001, International Journal of Impotence Research.

[36]  Micah Altman,et al.  Numerical Issues in Statistical Computing for the Social Scientist , 2003 .

[37]  Jerry G. Thursby,et al.  Replication in Empirical Economics: The Journal of Money, Credit and Banking Project , 1986 .

[38]  Thomas Mayer,et al.  Economics as a Hard Science: Realistic Goal or Wishful Thinking? , 1980 .

[39]  Giorgio Calzolari,et al.  Alternative Estimators of FIML Covariance Matrix: A Monte Carlo Stud y , 1988 .

[40]  Andreas Blume,et al.  Learning to communicate in cheap-talk games , 2004, Games Econ. Behav..

[41]  P. Perron,et al.  Computation and Analysis of Multiple Structural-Change Models , 1998 .

[42]  Ross McKitrick,et al.  Hockey sticks, principal components, and spurious significance , 2005 .

[43]  Ross McKitrick,et al.  The M&M Critique of the MBH98 Northern Hemisphere Climate Index: Update and Implications , 2005 .

[44]  Hrishikesh D. Vinod,et al.  Care and feeding of reproducible econometrics , 2001 .

[45]  J. Sobel,et al.  STRATEGIC INFORMATION TRANSMISSION , 1982 .

[46]  Gary King,et al.  Symposium on Replication in International Studies Research , 2003 .

[47]  Andrew B. Whinston,et al.  A Sender-Receiver Framework for Knowledge Transfer , 2005, MIS Q..

[48]  Malcolm K. Hughes,et al.  Global-scale temperature patterns and climate forcing over the past six centuries , 1998, Nature.

[49]  Gabriele Fiorentini,et al.  Analytic Derivatives and the Computation of Garch Estimates , 1996 .

[50]  S. Sklivas,et al.  Why econometricians don't replicate (although they do reproduce) , 1991 .

[51]  D. Giles,et al.  Taxes and the Canadian Underground Economy , 2007 .

[52]  Samer Faraj,et al.  Why Should I Share? Examining Social Capital and Knowledge Contribution in Electronic Networks of Practice , 2005, MIS Q..

[53]  Charles G. Renfro,et al.  Benchmarks and software standards: A case study of GARCH procedures , 1998 .

[54]  B. Hansen,et al.  Testing for two-regime threshold cointegration in vector error-correction models , 2002 .

[55]  M. Hughes,et al.  Global-scale temperature patterns and climate forcing over the past six centuries , 1998 .

[56]  M. Hughes,et al.  Northern hemisphere temperatures during the past millennium: Inferences, uncertainties, and limitations , 1999 .

[57]  E. Feige.  The Consequences of Journal Editorial Policies and a Suggestion for Revision , 1975, Journal of Political Economy.

[58]  William H. Greene.  Reply to "A comparative study of alternative econometric packages with an application to Italian deposit interest rates", by G. Bruno and R. De Bonis , 2004 .

[59]  Trevor Breusch.  The Canadian Underground Economy: An Examination of Giles and Tedds , 2006 .

[60]  Steven D. Levitt,et al.  Using Electoral Cycles in Police Hiring to Estimate the Effect of Policeon Crime , 1995 .

[61]  Richard G. Anderson,et al.  Replication and scientific standards in applied economics a decade after the Journal of Money, Credit and Banking project , 1994 .

[62]  T. Stanley,et al.  Wheat from Chaff: Meta-Analysis as Quantitative Literature Review: Response , 2001 .

[63]  T. Breusch,et al.  New estimates of mothers forgone earnings using HILDA data , 2004 .

[64]  W. Huffman,et al.  Setting Efficient Incentives for Agricultural Research: Lessons from Principal‐Agent Theory , 2000 .

[65]  John E. Hunter,et al.  CUMULATIVE RESEARCH KNOWLEDGE AND SOCIAL POLICY FORMULATION : The Critical Role of Meta-Analysis , 1996 .

[66]  J. Neuliep,et al.  Replication Research in the Social Sciences , 1991 .

[67]  Bruce D. McCullough,et al.  The Numerical Reliability of Econometric Software , 1999 .

[68]  FarajSamer,et al.  Why should i share? examining social capital and knowledge contribution in electronic networks of practice , 2005 .

[69]  Houston H. Stokes,et al.  On the advantage of using two or more econometric software systems to solve the same problem , 2004 .

[70]  Atreyi Kankanhalli,et al.  Contributing Knowledge to Electronic Knowledge Repositories: An Empirical Investigation , 2005, MIS Q..