Is there selection bias in laboratory experiments? The case of social and risk preferences

Laboratory experiments are frequently used to examine the nature of individuals’ social and risk preferences and inform economic theory. However, it is unknown whether the preferences of volunteer participants are representative of the population from which the participants are drawn, or whether they differ due to selection bias. To answer this question, we measured the preferences of 1,173 students in a classroom experiment using a trust game and a lottery choice task. Separately, we invited all students to participate in a laboratory experiment using common recruitment procedures. To evaluate whether there is selection bias, we compare the social and risk preferences of students who eventually participated in a laboratory experiment to those who did not, and find that they do not differ significantly. However, we also find that people who sent less in a trust game were more likely to participate in a laboratory experiment, and discuss possible explanations for this behavior.

[1]  James C. Cox,et al.  Risk aversion in experiments , 2008 .

[2]  Robin P. Cubitt,et al.  Experimental Economics: Rethinking the Rules , 2009 .

[3]  James J. Heckman,et al.  Characterizing Selection Bias Using Experimental Data , 1998 .

[4]  Robert L. Slonim,et al.  Opting-In: Participation Biases in the Lab , 2012, SSRN Electronic Journal.

[5]  Glenn W. Harrison,et al.  Risk Attitudes, Randomization to Treatment, and Self-Selection into Experiments , 2005 .

[6]  Steven D. Levitt,et al.  What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World , 2007 .

[7]  Colin Camerer,et al.  Measuring Social Norms and Preferences Using Experimental Games: A Guide for Social Scientists , 2002 .

[8]  Daniel Friedman,et al.  Experimental Methods: A Primer for Economists , 1994 .

[9]  Charles A. Holt,et al.  2 Risk Aversion and Incentive Effects , 2002 .

[10]  A. Rustichini,et al.  Self Selection Does Not Increase Other-Regarding Preferences Among Adult Laboratory Subjects, But Student Subjects May Be More Self-Regarding than Adults , 2010, SSRN Electronic Journal.

[11]  Werner Güth,et al.  Bargaining Outside the Lab a Newspaper Experiment of a Three-Person Ultimatum Game , 2005 .

[12]  John H. Kagel,et al.  Gaming against Managers in Incentive Systems: Experimental Results with Chinese Students and Chinese Managers , 1999 .

[13]  Robert Rosenthal,et al.  The Volunteer Subject , 1976 .

[14]  I. Bohnet,et al.  Decomposing trust and trustworthiness , 2006 .

[15]  E. Fehr,et al.  Fairness and Retaliation: The Economics of Reciprocity , 2000, SSRN Electronic Journal.

[16]  Catherine C. Eckel,et al.  Volunteers and Pseudo-Volunteers: The Effect of Recruitment Method in Dictator Experiments , 2000 .

[17]  James C. Cox,et al.  How to identify trust and reciprocity , 2004, Games Econ. Behav..

[18]  Stephan Meier,et al.  Do People Behave in Experiments as in the Field? Evidence from Donations , 2006 .

[19]  Ben Greiner,et al.  The Online Recruitment System ORSEE 2.0 - A Guide for the Organization of Experiments in Economics , 2004 .

[20]  J. Heckman.  Sample selection bias as a specification error , 1979 .

[21]  M. Orne.  On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. , 1962 .

[22]  Joel Sobel,et al.  INTERDEPENDENT PREFERENCES AND RECIPROCITY , 2005 .

[23]  J. Carpenter,et al.  Altruistic behavior in a representative dictator experiment , 2008 .

[24]  Rachel T. A. Croson,et al.  Gender Differences in Preferences , 2009 .

[25]  Colin Camerer,et al.  In search of homo economicus: Experiments in 15 small-scale societies , 2001 .

[26]  S. Zamir,et al.  Bargaining and Market Behavior in Jerusalem, Ljubljana, Pittsburgh, and Tokyo: An Experimental Study , 1991 .

[27]  Paola Sapienza,et al.  Can We Infer Social Preferences from the Lab? Evidence from the Trust Game , 2010 .

[28]  D. Friedman.  Preferences, beliefs and equilibrium: What have experiments taught us? , 2010 .

[29]  S. Gächter.  Behavioral Game Theory , 2008, Encyclopedia of Evolutionary Psychological Science.

[30]  Armin Falk,et al.  Do Lab Experiments Misrepresent Social Preferences? The Case of Self-Selected Student Samples , 2013 .

[31]  J. Carpenter,et al.  Comparing Students to Workers: The Effects of Social Framing on Behavior in Distribution Games , 2004, SSRN Electronic Journal.

[32]  Lise Vesterlund,et al.  Trust in Children , 2002 .

[33]  R. Rosenthal,et al.  The Volunteer Subject , 1965 .

[34]  Klaus M. Schmidt,et al.  The Economics of Fairness, Reciprocity and Altruism--Experimental Evidence and New Theories. , 2005 .

[35]  D. Fudenberg,et al.  A Dual Self Model of Impulse Control , 2004, The American economic review.

[36]  Glenn W. Harrison,et al.  Preference Heterogeneity in Experiments: Comparing the Field and Laboratory , 2007 .

[37]  C. Carter.  Freefall: America, Free Markets, and the Sinking of the World Economy , 2011 .

[38]  J. Tirole,et al.  Incentives and Prosocial Behavior , 2005 .

[39]  Hawaii,et al.  Supporting Online Material Materials and Methods Figs. S1 to S6 Tables S1 and S2 Database S1 Antisocial Punishment across Societies , 2022 .

[40]  R. Zeckhauser,et al.  Trust, Risk and Betrayal , 2003 .

[41]  John H. Miller,et al.  NOTES AND COMMENTS GIVING ACCORDING TO GARP: AN EXPERIMENTAL TEST OF THE CONSISTENCY OF PREFERENCES FOR ALTRUISM , 2002 .

[42]  Nikos Nikiforakis,et al.  Is There Selection Bias in Laboratory Experiments? , 2010 .

[43]  Thomas Gilovich,et al.  Does Studying Economics Inhibit Cooperation , 1993 .

[44]  J. Kagel,et al.  Handbook of Experimental Economics , 1997 .

[45]  Armin Falk,et al.  Did We Overestimate the Role of Social Preferences? The Case of Self-Selected Student Samples , 2010, SSRN Electronic Journal.

[46]  Joyce E. Berg,et al.  Trust, Reciprocity, and Social History , 1995 .

[47]  M. Rabin.  Published by: American , 2022 .

[48]  Steven D. Levitt,et al.  Viewpoint: On the Generalizability of Lab Behaviour to the Field , 2007 .

[49]  Iris Bohnet,et al.  Trust, Risk and Betrayal , 2004 .

[50]  Ananish Chaudhuri,et al.  Propensities to engage in and punish corrupt behavior: Experimental evidence from Australia, India, Indonesia and Singapore , 2009 .

[51]  Robert Slonim,et al.  The robustness of trust and reciprocity across a heterogeneous U.S. population , 2009 .

[52]  E. Rosen.  Differences between volunteers and non-volunteers for psychological studies. , 1951 .

[53]  H. P. Binswanger.  Attitudes toward Risk: Theoretical Implications of an Experiment in Rural India , 1981 .

[54]  Charles Bellemare,et al.  On Representative Social Capital , 2004, SSRN Electronic Journal.

[55]  John H. Kagel,et al.  Partial Gift Exchange in an Experimental Labor Market: Impact of Subject Population Differences, Productivity Differences, and Effort Requests on Behavior* , 2002, Journal of Labor Economics.

[56]  W. Greene.  Sample Selection Bias as a Specification Error: Comment , 1981 .

[57]  B. Roe,et al.  Risk-attitude selection bias in subject pools for experiments involving neuroimaging and blood samples , 2009 .

[58]  C. Plott,et al.  Handbook of Experimental Economics Results , 2008 .

[59]  John C. Ham,et al.  Selection Bias, Demographic Effects and Ability Effects in Common Value Auction Experiments , 2005 .

[60]  Charles A. Holt,et al.  Risk Aversion and Incentive Effects , 2002 .

[61]  C. B. Colby.  The weirdest people in the world , 1973 .

[62]  Catherine C. Eckel,et al.  Forecasting Risk Attitudes: An Experimental Study Using Actual and Forecast Gamble Choices , 2008 .

[63]  Dean S. Karlan,et al.  Using Experimental Economics to Measure Social Capital and Predict Financial Decisions , 2005 .