Assessing the Correspondence between Experimental Results Obtained in the Lab and Field: A Review of Recent Social Science Research*

A small but growing social science literature examines the correspondence between experimental results obtained in lab and field settings. This article reviews this literature and reanalyzes a set of recent experiments carried out in parallel in both the lab and field. Using a standardized format that calls attention to both the experimental estimates and the statistical uncertainty surrounding them, the study analyzes the overall pattern of lab-field correspondence, which is found to be quite strong (Spearman's ρ = 0.73). Recognizing that this correlation may be distorted by the ad hoc manner in which lab-field comparisons are constructed (as well as the selective manner in which results are reported and published), the article concludes by suggesting directions for future research, stressing in particular the need for more systematic investigation of treatment effect heterogeneity.

[1]  Judd B. Kessler,et al.  The External Validity of Laboratory Experiments: Qualitative Rather Than Quantitative Effects , 2015 .

[2]  Olivier Armantier,et al.  Comparing Corruption in the Laboratory and in the Field in Burkina Faso and in Canada , 2013 .

[3]  Carlos Scartascini,et al.  Tax Compliance and Enforcement in the Pampas: Evidence from a Field Experiment , 2013 .

[4]  Rachel Glennerster,et al.  Running Randomized Evaluations: A Practical Guide , 2013 .

[5]  G. Charness,et al.  Social and Moral Norms in Allocation Choices in the Laboratory , 2013 .

[6]  Hans-Theo Normann,et al.  Do short-term laboratory experiments provide valid descriptions of long-term economic interactions? A study of Cournot markets , 2013 .

[7]  Rupert Sausgruber,et al.  TESTING ENFORCEMENT STRATEGIES IN THE FIELD: THREAT, MORAL APPEAL AND SOCIAL INFORMATION , 2013 .

[8]  Scott Clifford,et al.  Comparing Contemporaneous Laboratory and Field Experiments on Media Effects , 2013 .

[9]  Fair Wages and Effort Provision: Combining Evidence from the Lab and the Field , 2013 .

[10]  Macartan Humphreys,et al.  Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration , 2012, Political Analysis.

[11]  Sarah Baird,et al.  Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review , 2013 .

[12]  M. Humphreys,et al.  Policing politicians: Citizen empowerment and political accountability in Uganda -Preliminary analysis (IGC Working Paper) , 2013 .

[13]  Jan Stoop,et al.  From the Lab to the Field: Cooperation among Fishermen , 2012, Journal of Political Economy.

[14]  D. Green,et al.  Statistical Analysis of Results from Laboratory Studies in Experimental Economics: A Critique of Current Practice , 2012 .

[15]  Donald P. Green,et al.  Field Experiments: Design, Analysis, and Interpretation , 2012 .

[16]  Alex Imas,et al.  Paying to Be Nice: Consistency and Costly Prosocial Behavior , 2011, Manag. Sci..

[17]  A. Chereni,et al.  Do micro-credit, micro-savings and micro-leasing serve as effective financial inclusion interventions enabling poor people, and especially women, to engage in meaningful economic opportunities in low- and middle-income countries? A systematic review of the evidence. , 2012 .

[18]  J. Stoop.  From the lab to the field: envelopes, dictators and manners , 2013, Experimental Economics.

[19]  Colin F. Camerer,et al.  The Promise and Success of Lab-Field Generalizability in Experimental Economics: A Critical Reply to Levitt and List , 2011 .

[20]  Alexandra A. Mislin,et al.  Trust games: A meta-analysis , 2011 .

[21]  Colin Camerer,et al.  Testing Game Theory in the Field: Swedish LUPI Lottery Games , 2010 .

[22]  Dean Karlan,et al.  Information Dissemination and Local Governments' Electoral Returns, Evidence from a Field Experiment in Mexico , 2011 .

[23]  E. King,et al.  AN EXPERIMENTAL FIELD STUDY OF INTERPERSONAL DISCRIMINATION TOWARD MUSLIM JOB APPLICANTS , 2010 .

[24]  Elizabeth A Stuart,et al.  Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research. , 2010, Psychological methods.

[25]  Kenneth C. Williams,et al.  Experimental Political Science and the Study of Causality: From Nature to the Lab , 2010 .

[26]  S. Levinson,et al.  WEIRD languages have misled us, too , 2010, Behavioral and Brain Sciences.

[27]  Christoph Engel,et al.  Dictator games: a meta study , 2010 .

[28]  Abhijit Banerjee,et al.  Do Informed Voters Make Better Choices? Experimental Evidence from Urban India , 2010 .

[29]  John A. List,et al.  Can Field Experiments Return Agricultural Economics to the Glory Days , 2009 .

[30]  David H. Reiley,et al.  What Happens in the Field Stays in the Field: Exploring Whether Professionals Play Minimax in Laboratory Experiments , 2009 .

[31]  Rachel T. A. Croson,et al.  Gender Differences in Preferences , 2009 .

[32]  J. Heckman,et al.  Lab Experiments Are a Major Source of Knowledge in the Social Sciences , 2009, Science.

[33]  Mirco Tonin,et al.  Disentangling the Sources of Pro-Social Behavior in the Workplace: A Field Experiment , 2009, SSRN Electronic Journal.

[34]  J. Weinstein,et al.  Coethnicity: Diversity and the Dilemmas of Collective Action , 2009 .

[35]  F. Norwood,et al.  How Closely Do Hypothetical Surveys and Laboratory Experiments Predict Field Behavior? , 2009 .

[36]  G. Charness,et al.  Cooperation and Competition in Intergenerational Experiments in the Field and the Laboratory , 2009 .

[37]  E. Paluck.  Reducing intergroup prejudice and conflict using the media: a field experiment in Rwanda. , 2009, Journal of personality and social psychology.

[38]  Kevin Arceneaux,et al.  Who Is Mobilized to Vote? A Re-Analysis of 11 Field Experiments , 2009 .

[39]  Steven D. Levitt,et al.  FIELD EXPERIMENTS IN ECONOMICS : THE PAST , THE PRESENT , AND THE FUTURE , 2008 .

[40]  G. Charness,et al.  Incentives to Exercise , 2008 .

[41]  Rachel T. A. Croson,et al.  Identity Congruency Effects on Donations , 2008 .

[42]  M. Johannesson,et al.  Crowding Out in Blood Donation: Was Titmuss Right? , 2008 .

[43]  P. Henry.  College Sophomores in the Laboratory Redux: Influences of a Narrow Data Base on Social Psychology's View of the Nature of Prejudice , 2008 .

[44]  J. Abeler,et al.  Fungibility , Labels , and Consumption * , 2008 .

[45]  Charles M. Brooks,et al.  Trip chaining behavior in multi-destination shopping trips: A field experiment and laboratory replication , 2008 .

[46]  Azar M. Khazian,et al.  Using normative social influence to promote conservation among hotel guests , 2008 .

[47]  Stephan Meier,et al.  Do People Behave in Experiments as in the Field? Evidence from Donations , 2006 .

[48]  John A. List,et al.  Homo economicus Evolves , 2008, Science.

[49]  M. R. Larson,et al.  The Illinois Field Study: A Significant Contribution to Understanding Real World Eyewitness Identification Issues , 2008, Law and human behavior.

[50]  Christopher W. Larimer,et al.  Social Pressure and Voter Turnout: Evidence from a Large-Scale Field Experiment , 2008, American Political Science Review.

[51]  John A. List,et al.  Matching and challenge gifts to charity: evidence from laboratory and natural field experiments , 2008, SSRN Electronic Journal.

[52]  O. Volij,et al.  Experientia Docet: Professionals Play Minimax in Laboratory Experiments ∗ , 2008 .

[53]  A. Schram,et al.  Public opinion polls, voter turnout, and welfare: An experimental study , 2010 .

[54]  K. Leonard,et al.  Moving from the lab to the field: Exploring scrutiny and duration effects in lab experiments , 2008 .

[55]  P. Dupas,et al.  Free Distribution or Cost-Sharing? Evidence from a Randomized Malaria Prevention Experiment , 2007 .

[56]  Armin Falk,et al.  Gift Exchange in the Field , 2007 .

[57]  Jesse M. Shapiro,et al.  Can Higher Prices Stimulate Product Use? Evidence from a Field Experiment in Zambia , 2010 .

[58]  Steven D. Levitt,et al.  Viewpoint: On the Generalizability of Lab Behaviour to the Field , 2007 .

[59]  Steven D. Levitt,et al.  What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World , 2007 .

[60]  Charles Bellemare,et al.  Gift Exchange within a Firm: Evidence from a Field Experiment , 2007, SSRN Electronic Journal.

[61]  Shelley J. Correll,et al.  Getting a Job: Is There a Motherhood Penalty?1 , 2007, American Journal of Sociology.

[62]  Alois Stutzer,et al.  Active Decisions and Prosocial Behaviour: A Field Experiment on Blood Donation , 2011 .

[63]  Egil Matsen,et al.  Joker: Choice in a simple game with large stakes , 2006 .

[64]  Peter C. Boxall,et al.  Do Ethical Consumers Care About Price? A Revealed Preference Analysis of Fair Trade Coffee Purchases , 2006 .

[65]  George Wu,et al.  The uncertainty effect: When a risky prospect is valued less than its worst possible outcome , 2006 .

[66]  C. Puppe,et al.  Putting Reciprocity to Work - Positive Versus Negative Responses in the Field , 2006 .

[67]  M. Bateson,et al.  Cues of being watched enhance cooperation in a real-world setting , 2006, Biology Letters.

[68]  Dean S. Karlan,et al.  Does Price Matter in Charitable Giving? Evidence from a Large-Scale Natural Field Experiment , 2006 .

[69]  David H. Reiley,et al.  Demand Reduction in Multi-Unit Auctions with Varying Numbers of Bidders: Theory and Evidence from a Field Experiment , 2006 .

[70]  David H. Reiley.  Field experiments on the effects of reserve prices in auctions: more Magic on the Internet , 2006 .

[71]  J. List,et al.  Using Hicksian Surplus Measures to Examine Consistency of Individual Preferences: Evidence from a Field Experiment , 2006 .

[72]  J. List.  The Behavioralist Meets the Market: Measuring Social Preferences and Reputation Effects in Actual Transactions , 2005 .

[73]  Heike Hennig-Schmidt,et al.  In search of worker's real effort reciprocity: A field and a laboratory experiment , 2010 .

[74]  Stephan Meier,et al.  Do Subsidies Increase Charitable Giving in the Long Run? Matching Donations in a Field Experiment , 2006 .

[75]  Laura O. Taylor,et al.  Altruism Spillovers: Are Behaviors in Context-Free Experiments Predictive of Altruism Toward a Naturally Occurring Public Good? , 2005 .

[76]  Dean S. Karlan,et al.  Tying Odysseus to the Mast: Evidence from a Commitment Savings Product in the Philippines , 2005 .

[77]  J. Carpenter,et al.  Do Social Preferences Increase Productivity? Field Experimental Evidence from Fishermen in Toyama Bay , 2005, SSRN Electronic Journal.

[78]  Dean S. Karlan,et al.  Using Experimental Economics to Measure Social Capital and Predict Financial Decisions , 2005 .

[79]  V. J. Hotz,et al.  Predicting the efficacy of future training programs using past experiences at other locations , 2005 .

[80]  J. Morgan,et al.  A Test of the Revenue Equivalence Theorem Using Field Experiments on Ebay , 2005 .

[81]  John A. List,et al.  The RAND Corporation Conspiracies and Secret Price Discounts in the Marketplace : Evidence from Field Experiments , 2008 .

[82]  J. Randal,et al.  Voluntary contributions to a public good: A natural field experiment , 2005 .

[83]  G. Harrison,et al.  Naturally Occurring Preferences and Exogenous Laboratory Experiments: A Case Study of Risk Aversion , 2004 .

[84]  G. Harrison,et al.  Field experiments , 1924, The Journal of Agricultural Science.

[85]  Ernan Haruvy,et al.  The Inefficiency of Splitting the Bill , 2004 .

[86]  Aldo Rustichini,et al.  Gender and competition at a young age , 2004 .

[87]  J. List.  The nature and extent of discrimination in the marketplace: Evidence from the field , 2004 .

[88]  Joshua D. Clinton,et al.  “Targeted” Advertising and Voter Turnout: An Experimental Study of the 2000 Presidential Election , 2004, The Journal of Politics.

[89]  R. Thaler,et al.  Save More Tomorrow™: Using Behavioral Economics to Increase Employee Saving , 2004, Journal of Political Economy.

[90]  Jonathan E. Alevy,et al.  Information Cascades: Evidence from a Field Experiment with Financial Market Professionals , 2004 .

[91]  John A. List,et al.  Young, Selfish and Male: Field Evidence of Social Preferences , 2004 .

[92]  A. Rustichini,et al.  Performance in Competitive Environments: Gender Differences , 2003 .

[93]  J. List.  Does market experience eliminate market anomalies , 2003 .

[94]  E. Fehr,et al.  Do Workers Work More If Wages are High? Evidence from a Randomized Field Experiment , 2005 .

[95]  Morten I. Lau,et al.  Estimating Individual Discount Rates in Denmark: A Field Experiment , 2002 .

[96]  David Lucking-Reiley,et al.  The Effects of Seed Money and Refunds on Charitable Giving: Experimental Evidence from a University Capital Campaign , 2002, Journal of Political Economy.

[97]  Nicholas A. Valentino,et al.  Group Cues and Ideological Constraint: A Replication of Political Advertising Effects Studies in the Lab and in the Field , 2002 .

[98]  John A. List,et al.  Do explicit warnings eliminate the hypothetical bias in elicitation procedures? Evidence from field auctions for sportscards , 2001 .

[99]  H. Oosterbeek,et al.  Cultural Differences in Ultimatum Game Experiments: Evidence from a Meta-Analysis , 2001 .

[100]  R. Boyd,et al.  In Search of Homo Economicus: Behavioral Experiments in 15 Small- Scale Societies , 2001 .

[101]  Joel Slemrod,et al.  Taxpayer response to an increased probability of audit: evidence from a controlled experiment in Minnesota , 2001 .

[102]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[103]  David H. Reiley,et al.  Demand Reduction in Multi-Unit Auctions: Evidence from a Sportscard Field Experiment: Reply , 2005 .

[104]  M. Haan,et al.  Free riding and the provision of candy bars , 2002 .

[105]  A. Rustichini,et al.  Pay Enough or Don't Pay at All , 2000 .

[106]  David Lucking-Reiley,et al.  Using field experiments to test equivalence between auction formats: Magic on the internet , 1999 .

[107]  R. G. Cummings,et al.  Unbiased Value Estimates for Environmental Goods: A Cheap Talk Design for the Contingent Valuation Method , 1999 .

[108]  James J. Lindsay,et al.  Research in the Psychological Laboratory , 1999 .

[109]  Colin Camerer.  Can Asset Markets Be Manipulated? A Field Experiment With Racetrack Betting , 1998, Journal of Political Economy.

[110]  C. Anderson,et al.  External Validity of “Trivial” Experiments: The Case of Laboratory Aggression , 1997 .

[111]  R. Bond,et al.  Culture and conformity: A meta-analysis of studies using Asch's (1952b, 1956) line judgment task. , 1996 .

[112]  Nicholas A. Valentino,et al.  Does Attack Advertising Demobilize the Electorate? , 1994, American Political Science Review.

[113]  Ido Erev,et al.  Constructive Intergroup Competition as a Solution to the Free Rider Problem: A Field Experiment , 1993 .

[114]  Leonard M. Lodish,et al.  Comparing Dynamic Consumer Choice in Real and Computer-simulated Environments , 1992 .

[115]  Ido Erev,et al.  Intergroup competition as a structural solution to social dilemmas. , 1990 .

[116]  Lionel Standing,et al.  Asch fails again , 1990 .

[117]  M. Frank,et al.  The dark side of self- and social perception: black uniforms and aggression in professional sports. , 1988, Journal of personality and social psychology.

[118]  D. O. Sears.  College sophomores in the laboratory: Influences of a narrow data base on social psychology's view of human nature. , 1986 .

[119]  Gary L. Wells,et al.  Improving eyewitness identifications from lineups: Simultaneous versus sequential lineup presentation. , 1985 .

[120]  D. Mook,et al.  In defense of external invalidity. , 1983 .

[121]  R. Rosenthal.  The file drawer problem and tolerance for null results , 1979 .

[122]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[123]  John R. Nevin,et al.  Laboratory Experiments for Estimating Consumer Demand: A Validation Study , 1974 .

[124]  C. B. Colby.  The weirdest people in the world , 1973 .

[125]  S. Milgram.  BEHAVIORAL STUDY OF OBEDIENCE. , 1963, Journal of abnormal psychology.

[126]  Student.  Probable Error of a Correlation Coefficient , 1908 .