Reusing Natural Experiments

After a natural experiment is first used, other researchers often reuse the setting, examining different outcome variables. We examine the consequences of reusing an experimental setting using two extensively studied natural experiments, business combination laws and the Regulation SHO pilot. We apply multiple hypothesis corrections and our findings suggest many results in the existing literature are false positives. We provide guidelines for inference when an experiment is reused using simulation evidence for several popular empirical settings including difference-in-differences regressions, instrumental variables regressions, and regression discontinuity designs.

[1]  Joshua D. Angrist,et al.  The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con Out of Econometrics , 2010, SSRN Electronic Journal.

[2]  Jeffry Netter,et al.  Observations on research and publishing from nineteen years as editors of the Journal of Corporate Finance , 2018 .

[3]  Yushu Zhu,et al.  Short‐Selling Constraints and Corporate Payout Policy , 2019 .

[4]  Benjamin M. Blau,et al.  Price clustering and the stability of stock prices , 2016 .

[5]  Bruce D. Meyer Natural and Quasi- Experiments in Economics , 1994 .

[6]  A. Heyes,et al.  Methods Matter: P-Hacking and Causal Inference in Economics , 2018, SSRN Electronic Journal.

[7]  O. J. Dunn Multiple Comparisons among Means , 1961 .

[8]  G. J. Alexander,et al.  The effect of price tests on trader behavior and market quality: An analysis of Reg SHO ☆ , 2008 .

[9]  I. Hasan,et al.  Corporate Governance and Dividend Payout Policy: A Test Using Antitakeover Legislation , 2011 .

[10]  John A. List,et al.  Multiple hypothesis testing in experimental economics , 2016, Experimental Economics.

[11]  Benjamin Bennett,et al.  The Real Effects of Financial Markets: Do Short Sellers Cause CEOs to Be Fired? , 2018 .

[12]  Stephen A. Hillegeist,et al.  Conference Calls and Information Asymmetry , 2003 .

[13]  Murillo Campello,et al.  The Rise of the Equity Lending Market: Implications for Corporate Policies , 2015 .

[14]  J. Angrist,et al.  Journal of Economic Perspectives—Volume 15, Number 4—Fall 2001—Pages 69–85 Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments , 2022 .

[15]  Joseph P. Romano,et al.  Efficient Computation of Adjusted P-Values for Resampling-Based Stepdown Multiple Testing , 2016 .

[16]  Michael Wolf,et al.  Control of generalized error rates in multiple testing , 2007, 0710.2258.

[17]  Matthew D. Cain,et al.  Do Takeover Laws Matter? Evidence from Five Decades of Hostile Takeovers , 2016 .

[18]  G. Lynch,et al.  The Control of the False Discovery Rate in Fixed Sequence Multiple Testing , 2016, 1611.03146.

[19]  Joseph Engelberg,et al.  Are Cross-Sectional Predictors Good Market-Level Predictors? , 2019, SSRN Electronic Journal.

[20]  R. Morck,et al.  Economics, History, and Causation , 2010 .

[21]  Hayong Yun The Choice of Corporate Liquidity and Corporate Governance , 2007 .

[22]  M. Massa,et al.  Competition of the Informed: Does the Presence of Short Sellers Affect Insider Selling? , 2014 .

[23]  Michael L. Davis,et al.  Takeover Protection and Managerial Myopia: Evidence from Real Earnings Management , 2012 .

[24]  Julian Atanassov Do Hostile Takeovers Stifle Innovation? Evidence from Antitakeover Legislation and Corporate Patenting , 2012 .

[25]  Edward E. Leamer,et al.  Let's Take the Con Out of Econometrics , 1983 .

[26]  Isaiah Andrews,et al.  Identification of and Correction for Publication Bias , 2017, American Economic Review.

[27]  Ian Appel Governance by Litigation , 2016 .

[28]  A. Wagner,et al.  The Executive Turnover Risk Premium , 2013 .

[29]  Yuanzhi Li,et al.  Does Corporate Governance Matter More for High Financial Slack Firms? , 2017, Manag. Sci..

[30]  Filippos Papakonstantinou,et al.  Reassessing False Discoveries in Mutual Fund Performance: Skill, Luck, or Lack of Power? , 2019, The Journal of Finance.

[31]  Jonathan M. Karpoff,et al.  Institutional and Legal Context in Natural Experiments: The Case of State Antitakeover Laws , 2017 .

[32]  J. Pearl Causal diagrams for empirical research , 1995 .

[33]  Jay M. Chung,et al.  Short selling and stock price crash risk , 2020 .

[34]  Kose John,et al.  Corporate Governance and Financing Policy: New Evidence , 2010 .

[35]  Dacheng Xiu,et al.  Thousands of Alpha Tests , 2020, The Review of Financial Studies.

[36]  Edward Leamer Tantalus on the Road to Asymptopia , 2010 .

[37]  Allen H. Huang,et al.  Short Selling and Earnings Management: A Controlled Experiment , 2015 .

[38]  Michael Wolf,et al.  The Romano–Wolf multiple-hypothesis correction in Stata , 2020, The Stata Journal: Promoting communications on statistics and Stata.

[39]  Michael Wolf,et al.  Centre De Referència En Economia Analítica Barcelona Economics Working Paper Series Working Paper Nº 17 Stewise Multiple Testing as Formalized Data Snooping Stepwise Multiple Testing as Formalized Data Snooping , 2022 .

[40]  Tarun Chordia,et al.  p-Hacking: Evidence from Two Million Trading Strategies , 2017 .

[41]  S. Mullainathan,et al.  Enjoying the Quiet Life? Corporate Governance and Managerial Preferences , 2003, Journal of Political Economy.

[42]  Kate Litvak,et al.  The SEC's Busted Randomized Experiment: What Can and Cannot Be Learned , 2016 .

[43]  Larry Yu Does Corporate Governance Matter , 2008 .

[44]  Yan Liu,et al.  Evaluating Trading Strategies , 2014, The Journal of Portfolio Management.

[45]  L. Kogan,et al.  Technological Innovation, Resource Allocation, and Growth , 2012 .

[46]  Kuan-Hui Lee,et al.  It's Sho Time! Short-Sale Price-Tests and Market Quality , 2007 .

[47]  J. Jones Earnings Management During Import Relief Investigations , 1991 .

[48]  Abel Brodeur,et al.  Star Wars: The Empirics Strike Back , 2012, SSRN Electronic Journal.

[49]  D. Cohen,et al.  Corporate Governance and the Information Environment: Evidence from State Antitakeover Laws , 2011 .

[50]  Kevin Tseng,et al.  Short-Termist CEO Compensation in Speculative Markets: A Controlled Experiment , 2017 .

[51]  Dean P. Foster,et al.  α‐investing: a procedure for sequential control of expected false discoveries , 2008 .

[52]  Gustavo Grullon,et al.  The Effects of Short-Selling Threats on Incentive Contracts: Evidence from an Experiment , 2016 .

[53]  David Easley,et al.  One Day in the Life of a Very Common Stock , 1997 .

[54]  P. Pasquariello Agency Costs and Strategic Speculation in the U.S. Stock Market , 2016, The Review of Corporate Finance Studies.

[55]  Sendhil Mullainathan,et al.  Is There Discretion in Wage Setting? a Test Using Takeover Legislation , 1998 .

[56]  N. Fuchs-Schündeln,et al.  Natural Experiments in Macroeconomics , 2015 .

[57]  John K. Wald,et al.  The Effect of State Laws on Capital Structure , 2007 .

[58]  Jérôme P. Taillard,et al.  What's your Identification Strategy? Innovation in Corporate Finance Research , 2015, Manag. Sci..

[59]  J. Pearl Causal inference in statistics: An overview , 2009 .

[60]  Zexi Wang Short sellers, institutional investors, and corporate cash holdings , 2013 .

[61]  Michael Wolf,et al.  Balanced Control of Generalized Error Rates , 2008 .