The Power of Bias in Economics Research

We investigate two critical dimensions of the credibility of empirical economics research: statistical power and bias. We survey 159 empirical economics literatures that draw upon 64,076 estimates of economic parameters reported in more than 6,700 empirical studies. Half of the research areas have nearly 90% of their results under‐powered. The median statistical power is 18%, or less. A simple weighted average of those reported results that are adequately powered (power ≥ 80%) reveals that nearly 80% of the reported effects in these empirical economics literatures are exaggerated; typically, by a factor of two and with one‐third inflated by a factor of four or more.

[1]  John A. List,et al.  To Replicate or Not to Replicate? Exploring Reproducibility in Economics Through the Lens of a Model and a Pilot Study , 2017 .

[2]  John P A Ioannidis,et al.  Finding the power to reduce publication bias , 2017, Statistics in medicine.

[3]  Cyrus Samii,et al.  The impact of land property rights interventions on investment and agricultural productivity in developing countries: a systematic review , 2017 .

[4]  P. Laroche A Meta‐Analysis of the Union–Job Satisfaction Relationship , 2016 .

[5]  J. Ioannidis,et al.  Obtaining evidence by a single well-powered trial or several modestly powered trials , 2016, Statistical methods in medical research.

[6]  John P A Ioannidis,et al.  Anticipating consequences of sharing raw data and code and of awarding badges for sharing. , 2016, Journal of clinical epidemiology.

[7]  Piers Steel,et al.  Does ICT Generate Economic Growth? A Meta‐Regression Analysis , 2018 .

[8]  T. Havránek MEASURING INTERTEMPORAL SUBSTITUTION: THE IMPORTANCE OF METHOD CHOICES AND SELECTIVE REPORTING , 2015 .

[9]  John A. List,et al.  How to make experimental economics research more reproducible: lessons from other disciplines and a new proposal. , 2015 .

[10]  T. Havránek,et al.  Financial Development and Economic Growth: A Meta‐Analysis , 2013 .

[11]  Brian A. Nosek,et al.  Promoting an open research culture , 2015, Science.

[12]  Hristos Doucouliagos,et al.  Neither fixed nor random: weighted least squares meta‐analysis , 2015, Statistics in medicine.

[13]  I. Martin What Property Tax Limitations Do to Local Finances: A Meta-Analysis , 2015 .

[14]  F. Santeramo,et al.  The income-elasticity of calories, macro and micro nutrients: What is the literature telling us? , 2015 .

[15]  Tomas Havranek,et al.  Bank Competition and Financial Stability: Much Ado About Nothing? , 2015 .

[16]  J. Ioannidis,et al.  Reproducibility in Science: Improving the Standard for Basic and Preclinical Research , 2015, Circulation research.

[17]  Hristos Doucouliagos,et al.  Wine Prices and Quality Ratings: A Meta‐Regression Analysis , 2015 .

[18]  Eric Gedajlovic,et al.  How Does Family Control Influence Firm Strategy and Performance? A Meta‐Analysis of US Publicly Listed Firms , 2015 .

[19]  Dean Karlan,et al.  Six Randomized Evaluations of Microcredit: Introduction and Further Steps † , 2015 .

[20]  R. Perman,et al.  Okun's Law - A Meta‐Analysis , 2015 .

[21]  W. Robert Reed,et al.  Replications in Economics: A Progress Report , 2014 .

[22]  Jae H. Kim,et al.  Market Efficiency in Asian and Australasian Stock Markets: A Fresh Look at the Evidence , 2014 .

[23]  A. Rathgeber,et al.  Determinants of corporate hedging: A (statistical) meta-analysis , 2014 .

[24]  Hristos Doucouliagos,et al.  The income elasticity of air travel: a meta-analysis. , 2014 .

[25]  S. Lazzaroni,et al.  Natural disasters' impact, factors of resilience and development: A meta-analysis of the macroeconomic literature , 2014 .

[26]  John P. A. Ioannidis,et al.  How to Make More Published Research True , 2014, PLoS medicine.

[27]  John P. A. Ioannidis,et al.  Assessing value in biomedical research: the PQRST of appraisal and reward. , 2014, JAMA.

[28]  Krishna B. Kumar,et al.  The Impact of Labor Market Regulation on Employment in Low‐Income Countries: A Meta‐Analysis , 2014 .

[29]  J. Vaessen,et al.  Meta-analysis of the impact of microcredit on women’s control over household decisions: methodological issues and substantive findings , 2014 .

[30]  Hristos Doucouliagos,et al.  Meta‐regression approximations to reduce publication selection bias , 2014, Research synthesis methods.

[31]  Christopher R. Jones,et al.  The impact of household energy efficiency measures on health: A meta-analysis , 2014 .

[32]  Andreas Peichl,et al.  The Own-Wage Elasticity of Labor Demand: A Meta-Regression Analysis , 2014, SSRN Electronic Journal.

[33]  W. Viscusi,et al.  Publication Selection and the Income Elasticity of the Value of a Statistical Life , 2013, Journal of health economics.

[34]  Leif D. Nelson,et al.  P-Curve: A Key to the File Drawer , 2013, Journal of experimental psychology. General.

[35]  T. Elfring,et al.  Social Capital of Entrepreneurs and Small Firm Performance: A Meta-Analysis of Contextual and Methodological Moderators , 2014 .

[36]  J. P. Nelson,et al.  Estimating the Price Elasticity of Beer: Meta-Analysis of Data with Heterogeneity, Dependence, and Publication Bias , 2013, Journal of health economics.

[37]  Gregory Shailer,et al.  Disentangling the Relationship between Ownership Concentration and Firm Performance in Emerging Markets: A Meta-Analysis , 2010 .

[38]  S. Cueto,et al.  Getting teachers back to the classroom. A systematic review on what works to improve teacher attendance in developing countries , 2013 .

[39]  J. Ioannidis,et al.  What's to Know About the Credibility of Empirical Economics? , 2013 .

[40]  A. Ortmann,et al.  Exploring the Meaning of Significance in Experimental Economics , 2013 .

[41]  J. Ioannidis Clarifications on the application and interpretation of the test for excess significance and its extensions , 2013 .

[42]  Selwyn Moons,et al.  A meta-analysis of economic diplomacy and its effect on international economic flows , 2013 .

[43]  S. Afesorgbor Revisiting the Effectiveness of African Economic Integration: A Meta-Analytic Review and Comparative Estimation Methods , 2013 .

[44]  John P A Ioannidis,et al.  Optimal type I and type II error pairs when the available sample size is fixed. , 2013, Journal of clinical epidemiology.

[45]  Claire Buckley,et al.  The Effect of Copayments for Prescriptions on Adherence to Prescription Medicines in Publicly Insured Populations; A Systematic Review and Meta-Analysis , 2013, PloS one.

[46]  Brian A. Nosek,et al.  Power failure: why small sample size undermines the reliability of neuroscience , 2013, Nature Reviews Neuroscience.

[47]  Hristos Doucouliagos,et al.  Unions and Innovation: New Insights from the Cross‐Country Evidence , 2013 .

[48]  T. Stanley,et al.  Are All Economic Facts Greatly Exaggerated? Theory Competition and Selectivity , 2013 .

[49]  Hristos Doucouliagos,et al.  Meta‐Analysis of Economics Research Reporting Guidelines , 2013 .

[50]  Rebecca M. Turner,et al.  The Impact of Study Size on Meta-analyses: Examination of Underpowered Studies in Cochrane Reviews , 2013, PloS one.

[51]  James Andreoni,et al.  The Power of Revealed Preference Tests : Ex-Post Evaluation of Experimental Design ∗ , 2013 .

[52]  J. Wicherts,et al.  The Rules of the Game Called Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[53]  J. Ioannidis Why Science Is Not Necessarily Self-Correcting , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[54]  Ethan M Balk,et al.  Influence of Reported Study Design Characteristics on Intervention Effect Estimates From Randomized, Controlled Trials , 2012, Annals of Internal Medicine.

[55]  Tommy Stanley,et al.  Meta-Regression Analysis in Economics and Business , 2012 .

[56]  Is China different?: A Meta-Analysis of Export-Led Growth , 2012 .

[57]  C. Begley,et al.  Drug development: Raise standards for preclinical cancer research , 2012, Nature.

[58]  Christer Ljungwall,et al.  Is China different? A meta-analysis of China's financial sector development , 2012 .

[59]  J. Saavedra,et al.  Impacts of Conditional Cash Transfer Programs on Educational Outcomes in Developing Countries , 2012 .

[60]  Maia Yang T. D. Stanley Micro-Credit and Income: A Literature Review and Meta-analysis , 2012 .

[61]  T. Stanley,et al.  Are estimates of the value of a statistical life exaggerated? , 2012, Journal of health economics.

[62]  Antonis Adam,et al.  The effect of globalization on capital taxation: What have we learned after 20 years of empirical studies? , 2013 .

[63]  John P A Ioannidis,et al.  Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects. , 2011, Journal of clinical epidemiology.

[64]  J. Ioannidis,et al.  Public Availability of Published Research Data in High-Impact Journals , 2011, PloS one.

[65]  T. Havránek,et al.  How to Solve the Price Puzzle? A Meta-Analysis , 2011 .

[66]  Nina Rosenbusch,et al.  Is innovation always beneficial? A meta-analysis of the relationship between innovation and performance in SMEs , 2011 .

[67]  Leon Bax,et al.  Beyond publication bias. , 2011, Journal of clinical epidemiology.

[68]  Jon P. Nelson,et al.  Alcohol Marketing, Adolescent Drinking and Publication Bias in Longitudinal Studies: A Critical Survey Using Meta-Analysis , 2011 .

[69]  J. Plantenga,et al.  Child Care Prices and Female Labour Force Participation: A Meta-Analysis , 2011 .

[70]  Masayuki Henmi,et al.  Confidence intervals for random effects meta‐analysis and robustness to publication bias , 2010, Statistics in medicine.

[71]  Peter Nijkamp,et al.  Joint impacts of immigration on wages and employment: review and meta-analysis , 2010, J. Geogr. Syst..

[72]  J. Hirschberg,et al.  Alcohol Consumption and Human Capital: A Retrospective Study of the Literature , 2010 .

[73]  Hristos Doucouliagos,et al.  Could It Be Better to Discard 90% of the Data? A Statistical Paradox , 2010 .

[74]  D. Headey,et al.  The Effect of Population Growth on Economic Growth: A Meta-Regression Analysis of the Macroeconomic Literature , 2009 .

[75]  T. Stanley,et al.  Efficiency Wages, Productivity and Simultaneity: A Meta-Regression Analysis , 2009 .

[76]  François Bellavance,et al.  The value of a statistical life: a meta-analysis with a mixed effects regression model. , 2009, Journal of health economics.

[77]  T. Stanley,et al.  Publication Selection Bias in Minimum-Wage Research? A Meta-Regression Analysis , 2009 .

[78]  T. Stanley,et al.  Theory competition and selectivity: are all economic facts greatly exaggerated? , 2008 .

[79]  Charles William Eliot,et al.  The Harvard classics , 2008 .

[80]  Margit Osterloh,et al.  Management Fashion Pay-for-Performance for CEOs , 2008 .

[81]  Xiao-Hua Zhou,et al.  Statistical Methods for Meta‐Analysis , 2008 .

[82]  T. Peters,et al.  Exploring differences in empirical time preference rates for health: an application of meta-regression. , 2008, Health economics.

[83]  Sasi Iamsiraroj A comprehensive analysis of foreign direct investment and economic growth , 2008 .

[84]  B. Pomeroy,et al.  Meta-analysis and the Accounting Literature: The Case of Audit Committee Independence and Financial Reporting Quality , 2007 .

[85]  J. Ioannidis,et al.  An exploratory test for an excess of significant findings , 2007, Clinical trials.

[86]  D. Hamermesh,et al.  Replication in Economics , 2007, SSRN Electronic Journal.

[87]  C. Schmid,et al.  Hospital Ownership and Financial Performance: What Explains the Different Findings in the Empirical Literature? , 2007, Inquiry : a journal of medical care organization, provision and financing.

[88]  Tommy Stanley,et al.  Meta-Regression Methods for Detecting and Estimating Empirical Effects in the Presence of Publication Selection , 2007 .

[89]  Henri L. F. de Groot,et al.  The Impact of Uncertainty on Investment: A Meta-Analysis , 2006 .

[90]  J. Fidrmuc,et al.  Meta-Analysis of the Business Cycle Correlation between the Euro Area and the Ceecs , 2006, SSRN Electronic Journal.

[91]  J. Ioannidis Why Most Published Research Findings Are False , 2005, PLoS medicine.

[92]  J. Ioannidis Contradicted and initially stronger effects in highly cited clinical research. , 2005, JAMA.

[93]  J. Poot,et al.  The Last Word on the Wage Curve? , 2005 .

[94]  Jon P. Nelson,et al.  Cigarette Advertising Regulation: A Meta-Analysis , 2005 .

[95]  R. Florax,et al.  A Meta-Analysis of Beta- Convergence , 2005 .

[96]  José Allouche,et al.  A Meta-analytical investigation of the relationship between corporate social and financial performance , 2005 .

[97]  Stephen T. Ziliak,et al.  Size Matters: The Standard Error of Regressions in the American Economic Review , 2004 .

[98]  S. Maxwell The persistence of underpowered studies in psychological research: causes, consequences, and remedies. , 2004, Psychological methods.

[99]  Peter Nijkamp,et al.  Price and Income Elasticities of Residential Water Demand: A Meta-Analysis , 2003, Land Economics.

[100]  Karl Rihaczek,et al.  1. WHAT IS DATA MINING? , 2019, Data Mining for the Social Sciences.

[101]  E. Strobl,et al.  Multinational Companies and Productivity Spillovers: A Meta‐Analysis , 2001 .

[102]  Chamu Sundaramurthy,et al.  A Meta-analysis of Board Leadership Structure and Financial Performance: are "two heads better than one"? , 2001 .

[103]  Ian Witten,et al.  Data Mining , 2000 .

[104]  K R Abrams,et al.  Modelling publication bias in meta-analysis: a review. , 2000, Statistical methods in medical research.

[105]  S Greenland,et al.  Random-effects meta-analyses are not always conservative. , 1999, American journal of epidemiology.

[106]  J. Copas What works?: selectivity models and meta‐analysis , 1999 .

[107]  J. L. Stimpert,et al.  BOARD COMPOSITION AND ORGANIZATIONAL PERFORMANCE: TWO STUDIES OF INSIDER/OUTSIDER EFFECTS , 1998 .

[108]  G. Smith,et al.  Bias in meta-analysis detected by a simple, graphical test , 1997, BMJ.

[109]  Deirdre N. McCloskey,et al.  The Standard Error of Regressions , 1996 .

[110]  Theodor D. Sterling,et al.  Publication decisions revisited: the effect of the outcome of statistical tests on the decision to p , 1995 .

[111]  David Card,et al.  Time-Series Minimum-Wage Studies: A Meta-analysis , 1995 .

[112]  A. Quisumbing Gender Differences In Agricultural Productivity: A Survey of Empirical Evidence , 1995 .

[113]  R. Hanka The Handbook of Research Synthesis , 1994 .

[114]  Kevin Lang,et al.  Are all Economic Hypotheses False? , 1992, Journal of Political Economy.

[115]  Jacob Cohen Statistical Power Analysis , 1992 .

[116]  John E. Hunter,et al.  Methods of Meta-Analysis: Correcting Error and Bias in Research Findings , 1991 .

[117]  Stephen B. Jarrell,et al.  Meta-Regression Analysis: A Quantitative Method of Literature Surveys , 1989 .

[118]  C. Begg,et al.  Publication bias : a problem in interpreting medical data , 1988 .

[119]  Donald N. McCloskey The Loss Function Has Been Mislaid: The Rhetoric of Significance Tests , 1985 .

[120]  Edward E. Leamer,et al.  Reporting the Fragility of Regression Estimates , 1983 .

[121]  Edward E. Leamer,et al.  Let's Take the Con Out of Econometrics , 1983 .

[122]  G. Glass,et al.  Meta-analysis in social research , 1981 .

[123]  R. Rosenthal The file drawer problem and tolerance for null results , 1979 .

[124]  B. Wolman,et al.  Handbook of clinical psychology , 1965 .

[125]  Gordon Tullock Publication Decisions and Tests of Significance—A Comment , 1959 .

[126]  T. Sterling Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa , 1959 .