Statistical Analysis of Results from Laboratory Studies in Experimental Economics: A Critique of Current Practice

Drawing on examples of recently published and widely-cited studies in experimental economics, we show that behavioral games are frequently analyzed in a manner that is prone to biased causal inference. First, deficiencies in design and implementation jeopardize the crucial assumption that treatments are statistically independent of potential outcomes. Researchers frequently do not randomly assign treatments or do not focus on randomly assigned factors when interpreting results. Second, many analyses of second mover behaviors in two-stage games, such as the ultimatum game and the trust game, are susceptible to bias. Third, uncontrolled stimuli, such as face-to-face interaction among subjects or the presentation of subjects’ photos, may also cause bias. Fourth, we discuss the limits of causal inference in repeated games, such as the public goods game. We recommend adjusting laboratory procedures and estimation methods in order to lessen reliance on substantive assumptions not grounded in experimental design.

[1]  L. Mittone,et al.  Peer pressure, social spillovers, and reciprocity: an experimental analysis , 2011 .

[2]  J. Heckman,et al.  Lab Experiments Are a Major Source of Knowledge in the Social Sciences , 2009, Science.

[3]  U. Fischbacher,et al.  Are People Conditionally Cooperative? Evidence from a Public Goods Experiment , 2001 .

[4]  T. Speed,et al.  On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9 , 1990 .

[5]  Jordi Brandts,et al.  The strategy versus the direct-response method: a first survey of experimental comparisons , 2011 .

[6]  D. Moher,et al.  The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. , 2001, Journal of the American Podiatric Medical Association.

[7]  Montserrat Viladrich-Grau,et al.  The sustainability of the commons: giving and receiving , 2011 .

[8]  Colin Camerer,et al.  A Cognitive Hierarchy Model of Games , 2004 .

[9]  V. Smith,et al.  Positive reciprocity and intentions in trust games , 2003 .

[10]  J. Henrich,et al.  Markets, Religion, Community Size, and the Evolution of Fairness and Punishment , 2010, Science.

[11]  J. Carpenter,et al.  Playing both roles in the trust game , 2003 .

[12]  Edward,et al.  Measuring Trust , 2000 .

[13]  Catherine C. Eckel,et al.  Chivalry and Solidarity in Ultimatum Games , 2001 .

[14]  J. Cornfield.  Sequential Trials, Sequential Analysis and the Likelihood Principle , 1966 .

[15]  B. Ruffieux,et al.  Do price-tags influence consumers’ willingness to pay? On the external validity of using auctions for measuring value , 2011 .

[16]  Joel A. Middleton,et al.  Bias of the regression estimator for experiments using clustered random assignment , 2008 .

[17]  Daniel Houser,et al.  Revisiting Kindness and Confusion in Public Goods Experiments , 2002 .

[18]  U. Fischbacher,et al.  Are People Conditionally Cooperative? Evidence from a Public Goods Experiment , 2001 .

[19]  James H. Fowler,et al.  Heritability of cooperative behavior in the trust game , 2008, Proceedings of the National Academy of Sciences.

[20]  M. Kirchler,et al.  Experimental asset markets with endogenous choice of costly asymmetric information , 2011 .

[21]  Robert S. Gazzale,et al.  Remain Silent and Ye Shall Suffer: Seller Exploitation of Reticent Buyers in an Experimental Reputation System , 2008 .

[22]  D. Rubin.  [On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9.] Comment: Neyman (1923) and Causal Inference in Experiments and Observational Studies , 1990 .

[23]  Jeffrey D. Scargle,et al.  Publication Bias: The “File-Drawer” Problem in Scientific Inference , 2000 .

[24]  Robin P. Cubitt,et al.  Framing and Free Riding: Emotional Responses and Punishment in Social Dilemma Games Framing and Free Riding: Emotional Responses and Punishment in Social Dilemma Games , 2022 .

[25]  G. Charness,et al.  What's in a Name? Anonymity and Social Distance in Dictator and Ultimatum Games , 2003 .

[26]  Tālis J. Putniņš,et al.  Pricing accuracy, liquidity and trader behavior with closing price manipulation , 2010 .

[27]  Daniel Friedman,et al.  Experimental Methods: A Primer for Economists , 1994 .

[28]  L. Putterman,et al.  Cooperation Under the Threat of Expulsion in a Public Goods Experiment , 2005 .

[29]  Dean S. Karlan,et al.  Using Experimental Economics to Measure Social Capital and Predict Financial Decisions , 2005 .

[30]  Lucy F. Ackert,et al.  Are benevolent dictators altruistic in groups? A within-subject design , 2011 .

[31]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[32]  H. Gintis,et al.  Social Capital and Community Governance , 2002 .

[33]  A. Cabrales,et al.  Hidden information, bargaining power, and efficiency: an experiment , 2011 .

[34]  N. Uler,et al.  Public goods provision, inequality and taxes , 2011 .

[35]  James C. Cox,et al.  How to identify trust and reciprocity , 2004, Games Econ. Behav..

[36]  Alessandra Cassar,et al.  Economics Lab: An Intensive Course in Experimental Economics , 2004 .

[37]  Rachel T. A. Croson,et al.  Trust and gender: An examination of behavior and beliefs in the Investment Game , 2008 .

[38]  Stephen J. Rassenti,et al.  Reciprocity, Trust, and Payoff Privacy in Extensive Form Bargaining , 1998 .

[39]  Nathaniel T. Wilcox,et al.  Saving behavior and cognitive abilities , 2011 .

[40]  Jordi Brandts,et al.  Cooperation and Noise in Public Goods Experiments: Applying the Contribution Function Approach , 1999 .

[41]  Urs Fischbacher,et al.  The behavioral validity of the strategy method in public good experiments , 2012 .

[42]  S. Blount.  When Social Outcomes Aren't Fair: The Effect of Causal Attributions on Preferences , 1995 .

[43]  J. Robins,et al.  Identifiability and Exchangeability for Direct and Indirect Effects , 1992, Epidemiology.

[44]  Joshua D. Angrist,et al.  Identification of Causal Effects Using Instrumental Variables , 1993 .

[45]  Joshua D. Angrist,et al.  Long-Term Educational Consequences of Secondary School Vouchers: Evidence from Administrative Records in Colombia , 2006 .

[46]  R. Boyd,et al.  In Search of Homo Economicus: Behavioral Experiments in 15 Small- Scale Societies , 2001 .

[47]  Jeffrey M. Wooldridge,et al.  Cluster-Sample Methods in Applied Econometrics , 2003 .

[48]  V. Smith,et al.  Friend-or-foe intentionality priming in an extensive form trust game , 2000 .

[49]  Colin Camerer.  Behavioral Game Theory: Experiments in Strategic Interaction , 2003 .

[50]  Erik O. Kimbrough.  Learning to respect property by refashioning theft into trade , 2011 .

[51]  D. Rubin.  Causal Inference Using Potential Outcomes , 2005 .

[52]  C. Fershtman,et al.  Trust and discrimination in a segmented society: An experimental approach , 2001 .

[53]  Edward H. Kaplan,et al.  Problems and Methods in the Study of Politics: The illusion of learning from observational research , 2004 .

[54]  J. Anderies,et al.  Coordination and cooperation in asymmetric commons dilemmas , 2011 .

[55]  Nagore Iriberri,et al.  The role of role uncertainty in modified dictator games , 2008 .

[56]  U. Fischbacher,et al.  Oxytocin increases trust in humans , 2005, Nature.

[57]  Kenneth C. Williams,et al.  Experimental Political Science and the Study of Causality: From Nature to the Lab , 2010 .

[58]  Jordi Brandts,et al.  Hot vs. Cold: Sequential Responses and Preference Stability in Experimental Games , 2000 .

[59]  Donald B. Rubin,et al.  Formal modes of statistical inference for causal effects , 1990 .

[60]  Catherine C. Eckel,et al.  Is Trust a Risky Decision? , 2004 .

[61]  H. Oosterbeek,et al.  Cultural Differences in Ultimatum Game Experiments: Evidence from a Meta-Analysis , 2001 .

[62]  Matthew R. Roelofs,et al.  Heterogeneous preferences for altruism: gender and personality, social status, giving and taking , 2011 .

[63]  David Fielding,et al.  Does trust extend beyond the village? Experimental trust and social distance in Cameroon , 2011 .

[64]  Lisa R. Anderson,et al.  Measuring altruism in a public goods experiment: a comparison of U.S. and Czech subjects , 2011 .

[65]  J. Brandts,et al.  Hot vs. Cold: Sequential Responses and Preference Stability in Experimental Games , 1998 .

[66]  E. Fehr,et al.  Cooperation and Punishment in Public Goods Experiments , 1999, SSRN Electronic Journal.

[67]  Robert Shupp,et al.  The Effect of Rewards and Sanctions in Provision of Public Goods , 2006 .

[68]  Carmela Di Mauro,et al.  Kindness, confusion, or … ambiguity? , 2011 .

[69]  J. Weinstein,et al.  Coethnicity: Diversity and the Dilemmas of Collective Action , 2009 .

[70]  D. Freedman.  Statistical Models and Causal Inference: On Regression Adjustments in Experiments with Several Treatments , 2008, 0803.3757.

[71]  Till Requate,et al.  “A profit table or a profit calculator?” A note on the design of Cournot oligopoly experiments , 2011 .

[72]  Donald P. Green,et al.  Field Experiments: Design, Analysis, and Interpretation , 2012 .

[73]  S. Solnick.  Gender differences in the ultimatum game , 2001 .

[74]  J. Henrich.  Does culture matter in economic behavior? Ultimatum game bargaining among the machiguenga , 2000 .

[75]  Jonathan D. Cohen,et al.  The Neural Basis of Economic Decision-Making in the Ultimatum Game , 2003, Science.

[76]  I. Bohnet,et al.  Decomposing trust and trustworthiness , 2006 .

[77]  M. Aickin,et al.  Adjusting for multiple testing when reporting research results: the Bonferroni vs Holm methods. , 1996, American journal of public health.

[78]  ปิยดา สมบัติวัฒนา.  Behavioral Game Theory: Experiments in Strategic Interaction , 2013 .

[79]  Amadou Boly.  On the incentive effects of monitoring: evidence from the lab and the field , 2011 .

[80]  Erte Xiao,et al.  Classification of natural language messages using a coordination game , 2011 .

[81]  Ferdinand M. Vieider.  Separating real incentives and accountability , 2011 .

[82]  Philipp E. Otto,et al.  Matching markets with price bargaining , 2011 .

[83]  Ernst Fehr,et al.  Are people conditionally cooperative , 2000 .

[84]  J. Andreoni,et al.  Public goods experiments without confidentiality: a glimpse into fund-raising , 2004 .