Replications in Economics: A Progress Report

This study reports on various aspects of replication research in economics. It includes (i) a brief history of data sharing and replication; (ii) the results of the authors’ survey administered to the editors of all 333 “Economics” journals listed in Web of Science in December 2013; (iii) an analysis of 155 replication studies that have been published in peer-reviewed economics journals from 1977-2014; (iv) a discussion of the future of replication research in economics, and (v) observations on how replications can be better integrated into research efforts to address problems associated with publication bias and other Type I error phenomena.

[1]  A. Dawson.  The performance of three wage equations in postwar Britain , 1983 .

[2]  L. Robbins,et al.  An Essay on the Nature and Significance of Economic Science. , 1934 .

[3]  K. Basu,et al.  Is literacy shared within households? Theory and evidence for Bangladesh , 2001 .

[4]  R. Palmer-jones,et al.  Literacy Sharing, Assortative Mating, or What? Labour Market Advantages and Proximate Illiteracy Revisited , 2008 .

[5]  R. Palmer-jones,et al.  TV, female empowerment and demographic change in rural India , 2014 .

[6]  M. Paldam.  Regression Costs Fall, Mining Ratios Rise, Publication Bias Looms, and Techniques Get Fancier: Reflections on Some Trends in Empirical Macroeconomics , 2013 .

[7]  D. Winkler.  Educational Achievement and School Peer Group Composition , 1975 .

[8]  David Card,et al.  Time-Series Minimum-Wage Studies: A Meta-analysis , 1995 .

[9]  Gilda Kelner,et al.  Começa tudo outra vez... , 2003 .

[10]  Laura Camfield,et al.  Three ‘Rs’ of Econometrics: Repetition, Reproduction and Replication , 2013 .

[11]  E. Fong,et al.  Coercive Citation in Academic Publishing , 2012, Science.

[12]  Phillip Glandon.  Report on the American Economic Review Data Availability Compliance Project , 2010 .

[13]  M. Donnellan,et al.  Does Cleanliness Influence Moral Judgments , 2014 .

[14]  Reflections of a first-time replicator , 2007 .

[15]  John A. List,et al.  One swallow doesn’t make a summer: new evidence on anchoring effects , 2014 .

[16]  E. Oster,et al.  The Power of TV: Cable Television and Women's Status in India , 2007 .

[17]  Mina Bissell,et al.  Reproducibility: The risks of the replication drive , 2013, Nature.

[18]  C. Hoxby.  Does Competition Among Public Schools Benefit Students and Taxpayers? , 1994 .

[19]  Steven D. Levitt,et al.  Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime: Comment , 2002 .

[20]  Gordon Tullock.  Publication Decisions and Tests of Significance—A Comment , 1959 .

[21]  Michael A. Clemens,et al.  The Meaning of Failed Replications: A Review and Proposal , 2015, SSRN Electronic Journal.

[22]  John Gibson,et al.  Which Journal Rankings Best Explain Academic Salaries? Evidence from the University of California , 2012 .

[23]  Neil Malhotra,et al.  Publication bias in the social sciences: Unlocking the file drawer , 2014, Science.

[24]  T. Kuhn,et al.  The Structure of Scientific Revolutions , 1963 .

[25]  Bruce D. McCullough,et al.  Open Access Economics Journals and the Market for Reproducible Economic Research , 2009 .

[26]  Beyond Race Cards in America's Pastime: An Appreciative Reply to Findlay and Santos , 2012 .

[27]  Outliers and the Halloween Effect: Comment on Maberly and Pierce , 2010 .

[28]  Jesse M. Rothstein.  Does Competition Among Public Schools Benefit Students and Taxpayers? A Comment on Hoxby (2000) , 2004 .

[29]  W. Greene,et al.  The role of data/code archives in the future of economic research , 2008 .

[30]  L. Camfield,et al.  Things you Wanted to Know about Bias in Evaluations but Never Dared to Think , 2014 .

[31]  J. Ioannidis,et al.  What's to Know About the Credibility of Empirical Economics? , 2013 .

[32]  J. Ioannidis.  Why Most Published Research Findings Are False , 2019, CHANCE.

[33]  Leonard E. Burman,et al.  A Call for Replication Studies , 2010 .

[34]  James A. Robinson,et al.  Hither Thou Shalt Come, But No Further: Reply to "the Colonial Origins of Comparative Development: An Empirical Investigation: Comment" , 2011 .

[35]  Roger D Peng,et al.  Reproducible research and Biostatistics. , 2009, Biostatistics.

[36]  E. Feige.  The Consequences of Journal Editorial Policies and a Suggestion for Revision , 1975, Journal of Political Economy.

[37]  R. Rosenthal.  The file drawer problem and tolerance for null results , 1979 .

[38]  P. Gordon,et al.  Does Economic Performance Correlate with Big Government , 2004 .

[39]  Robert A. Moffitt.  Report of the Editor: American Economic Review , 2005 .

[40]  Maren Duvendack,et al.  Replication of quantitative work in development studies: Experiences and suggestions , 2013 .

[41]  Anton Nekrutenko,et al.  Ten Simple Rules for Reproducible Computational Research , 2013, PLoS Comput. Biol..

[42]  K. Basu,et al.  Is Literacy Shared within Households? Theory and Evidence from Bangladesh , 2001 .

[43]  David M. Drukker,et al.  “Replicating the results in ‘on efficient estimation with panel data: an empirical comparison of instrumental variables estimators’ ” , 2003 .

[44]  James A. Robinson,et al.  The Colonial Origins of Comparative Development : An Empirical Investigation : Comment † By , 2012 .

[45]  N. Kerr.  HARKing: Hypothesizing After the Results are Known , 1998, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[46]  Badi H. Baltagi,et al.  On Efficient Estimation with Panel Data: An Empirical Comparison of Instrumental Variables Estimators , 1990 .

[47]  Teresa D. Harrison,et al.  Lessons from the JMCB Archive , 2006 .

[48]  Jerry G. Thursby,et al.  Replication in Empirical Economics: The Journal of Money, Credit and Banking Project , 1986 .

[49]  Jan H. Höffler.  Teaching Replication in Quantitative Empirical Economics , 2014 .

[50]  B. Jasny,et al.  Again, and Again, and Again … , 2011 .

[51]  F. Siskind.  MINIMUM WAGE LEGISLATION IN THE UNITED STATES: COMMENT , 1977 .

[52]  J. Haan,et al.  Growth accelerations and regime changes: A correction , 2007 .

[53]  Hashem Pesaran,et al.  Introducing a replication section , 2003 .

[54]  Michael C. Lovell,et al.  Econometric Software Accidents , 1994 .

[55]  Stephen B. Jarrell,et al.  Meta-Regression Analysis: A Quantitative Method of Literature Surveys , 1989 .

[56]  James A. Robinson,et al.  The Colonial Origins of Comparative Development: An Empirical Investigation , 2000 .

[57]  S. Liebowitz,et al.  Willful Blindness: The Inefficient Reward Structure in Academic Research , 2013 .

[58]  O. Morgenstern,et al.  On the Accuracy of Economic Observations. , 1950 .

[59]  S. Sklivas,et al.  Why econometricians don't replicate (although they do reproduce) , 1991 .

[60]  John M. Fitzgerald,et al.  Trust, Reciprocity, and Social History: A Re-examination , 2000 .

[61]  Jesse Rothstein.  Does Competition Among Public Schools Benefit Students and Taxpayers? Comment , 2007 .

[62]  D. W. Findlay,et al.  Race, Ethnicity, and Baseball Card Prices: A Replication, Correction, and Extension of Hewitt, Muñoz, Oliver, and Regoli , 2012 .

[63]  Daniel Kahneman,et al.  A new etiquette for replication. , 2014 .

[64]  S. Feigenbaum,et al.  The market for (Ir)reproducible econometrics , 1993 .

[65]  Simone Schnall,et al.  With a Clean Conscience , 2008, Psychological science.

[66]  Tommy Stanley,et al.  Meta-Regression Analysis in Economics and Business , 2012 .

[67]  Simone Schnall,et al.  Clean Data: Statistical Artefacts Wash Out Replication Efforts , 2014 .

[68]  Thomas Mayer,et al.  Economics as a Hard Science: Realistic Goal or Wishful Thinking? , 1980 .

[69]  H. Vinod,et al.  Verifying the Solution from a Nonlinear Solver: A Case Study , 2004 .

[70]  Mary R. Anderson,et al.  Meta-Analysis for Public Management and Policy , 2013 .

[71]  Raylene M. Pierce,et al.  Stock Market Efficiency Withstands another Challenge: Solving the "Sell in May/Buy after Halloween" Puzzle , 2004 .

[72]  Hristos Doucouliagos,et al.  Publication Bias in the Economic Freedom and Economic Growth Literature , 2005 .

[73]  Lant Pritchett,et al.  It pays to be ignorant: A simple political economy of rigorous program evaluation , 2002 .

[74]  C. Link,et al.  Social Returns to Quantity and Quality of Education: A Further Statement , 1975 .