Can incentives improve survey data quality in developing countries?: results from a field experiment in India

Summary We report results of an experiment designed to assess whether the payment of contingent incentives to respondents in Karnataka, India, impacts the quality of survey data. Of 2276 households sampled at the city block level, 934 were randomly assigned to receive a small one-time payment at the time of the survey, whereas the remaining households did not receive this incentive. We analyse the effects of incentives across a range of questions that are common in survey research in less developed countries. Our study suggests that incentives reduced unit non-response. Conditionally on participation, we also find little impact of incentives on a broad range of sociodemographic, behavioural and attitudinal questions. In contrast, we consistently find that households that received incentives reported substantially lower consumption and income levels and fewer assets. Given random assignment and very high response rates, the most plausible interpretation of this finding is that incentivizing respondents in this setting may increase their motivation to present themselves as more needy, whether to justify the current payment or to increase the chance of receiving resources in the future. Therefore, despite early indications that contingent incentives may raise response rates, the net effect on data quality must be carefully considered.

[1]  Gustav Peebles.  The Anthropology of Credit and Debt , 2010 .

[2]  G. Fong,et al.  Prepaid Monetary Incentives—Predictors of Taking the Money and Completing the Survey , 2014, Sociological methods & research.

[3]  T. Dyson.  Causes and Consequences of Skewed Sex Ratios , 2012 .

[4]  Eleanor Singer,et al.  Does the Payment of Incentives Create Expectation Effects , 1998 .

[5]  Tom W. Smith,et al.  ASKING SENSITIVE QUESTIONS THE IMPACT OF DATA COLLECTION MODE, QUESTION FORMAT, AND QUESTION CONTEXT , 1996 .

[6]  A. Cross,et al.  HIV testing in national population-based surveys: experience from the Demographic and Health Surveys. , 2006, Bulletin of the World Health Organization.

[7]  N. C. Schaeffer,et al.  Requests, Blocking Moves, and Rational (Inter)action in Survey Introductions. , 2010, American sociological review.

[8]  Eleanor Singer,et al.  The Use and Effects of Incentives in Surveys , 2013 .

[9]  Alexander A. Weinreb,et al.  The Limitations of Stranger-Interviewers in Rural Kenya , 2006 .

[10]  F. K. Shuttleworth.  A study of questionnaire technique. , 1931 .

[11]  Jean M. Converse,et al.  THE EFFECTS OF BLACK AND WHITE INTERVIEWERS ON BLACK RESPONSES IN 1968 , 1971 .

[12]  Hans Baumgartner,et al.  Response Styles in Marketing Research: A Cross-National Investigation , 2001 .

[13]  Angus Deaton,et al.  Data and Dogma: The Great Indian Poverty Debate , 2004 .

[14]  A. H. Church.  ESTIMATING THE EFFECT OF INCENTIVES ON MAIL SURVEY RESPONSE RATES: A META-ANALYSIS , 1993 .

[15]  A. Deaton.  The Analysis of Household Surveys : A Microeconometric Approach to Development Policy , 1997 .

[16]  E. Deci.  Effects of Externally Mediated Rewards on Intrinsic Motivation. , 1971 .

[17]  Sandrine Mesplé-Somps,et al.  Measuring Inequalities: Do Household Surveys Paint a Realistic Picture? , 2010 .

[18]  E. Singer,et al.  Differential Incentives: Beliefs about Practices, Perceptions of Equity, and Effects on Survey Participation , 1999 .

[19]  Frederick G. Conrad,et al.  Misunderstanding Standardized Language in Research Interviews , 2004 .

[20]  J. Brick,et al.  Explaining Rising Nonresponse Rates in Cross-Sectional Surveys , 2013 .

[21]  Hanna Herzog.  On Home Turf: Interview Location and Its Social Meaning , 2005 .

[22]  Arie W. Kruglanski,et al.  The effects of extrinsic incentive on some qualitative aspects of task performance1 , 1971 .

[23]  Tom W. Smith.  THE IMPACT OF THE PRESENCE OF OTHERS ON A RESPONDENT'S ANSWERS TO QUESTIONS , 1997 .

[24]  Howard Schuman,et al.  WHITE RESPONDENTS AND RACE-OF-INTERVIEWER EFFECTS , 1975 .

[25]  S. Presser,et al.  Data collection mode and social desirability bias in self-reported religious attendance : Church attendance in the United States , 1998 .

[26]  Stanley Lemeshow,et al.  Goodness-of-fit Test for a Logistic Regression Model Fitted using Survey Sample Data , 2006 .

[27]  E. Singer,et al.  Leverage-saliency theory of survey participation: description and an illustration. , 2000, Public opinion quarterly.

[28]  Paul C. Hewett,et al.  The reporting of sensitive behavior by adolescents: A methodological experiment in Kenya , 2003, Demography.

[29]  Margaret Walsh,et al.  Gifts and Gifting , 2010 .

[30]  N. C. Schaeffer,et al.  Are Interactional Behaviors Exhibited When the Self-Reported Health Question is Asked Associated with Health Status? , 2011, Social science research.

[31]  Diane K. Willimack,et al.  EFFECTS OF A PREPAID NONMONETARY INCENTIVE ON RESPONSE RATES AND RESPONSE QUALITY IN A FACE-TO-FACE SURVEY , 1995 .

[32]  Stanley Presser,et al.  Changes in Telephone Survey Nonresponse over the Past Quarter Century , 2005 .

[33]  Troy Devon Thomas,et al.  Response Styles in Survey Research: A Literature Review of Antecedents, Consequences, and Remedies , 2013 .

[34]  E. Singer,et al.  Experiments with incentives in telephone surveys. , 2000, Public opinion quarterly.

[35]  M. Gupta.  Selective Discrimination against Female Children in Rural Punjab, India , 1987 .

[36]  Young Ik Cho,et al.  The Relation Between Culture and Response Styles , 2005 .

[37]  Gifting Practices and Interorganizational Relations: Constructing Obligation Networks in the Electronics Sector , 2003 .

[38]  R. Hayes,et al.  Asking semi‐literate adolescents about sexual behaviour: the validity of assisted self‐completion questionnaire (ASCQ) data in rural Tanzania , 2004, Tropical medicine & international health : TM & IH.

[39]  S. B. Assche.  Are we measuring what we want to measure? Individual consistency in survey response in rural Malawi , 2003 .

[40]  Stephen Mark Rosenbaum,et al.  Let’s be honest: A review of experimental evidence of honesty and truth-telling , 2014 .

[41]  R. Tourangeau,et al.  Response Quality in Telephone Surveys Do Prepaid Cash Incentives Make a Difference , 2015 .

[42]  Achim Koch,et al.  Are Incentive Effects on Response Rates and Nonresponse Bias in Large-scale, Face-to-face Surveys Generalizable to Germany? Evidence from Ten Experiments , 2015 .

[43]  J. Zagorsky,et al.  The Effects of Promised Monetary Incentives on Attrition in a Long-Term Panel Survey , 2008 .

[44]  D. Cantor,et al.  How Much Gets You How Much? Monetary Incentives and Response Rates in Household Surveys , 2015 .

[45]  R. Brant.  Assessing proportionality in the proportional odds model for ordinal logistic regression. , 1990, Biometrics.

[46]  S. Timmermans,et al.  A World of Standards but not a Standard World: Toward a Sociology of Standards and Standardization* , 2010 .

[47]  Rutstein So,et al.  Assessment of the quality of data used to ascertain eligibility and age in the Demographic and Health Surveys. , 1990 .

[48]  B. Frey,et al.  The Cost of Price Incentives: An Empirical Analysis of Motivation Crowding-Out , 1997 .

[49]  Å. Vernby,et al.  Can incentives undermine intrinsic motivation to participate in epidemiologic surveys? , 2010, European Journal of Epidemiology.

[50]  F. Gino,et al.  Good Lamps Are the Best Police , 2010, Psychological science.

[51]  Frederick G. Conrad,et al.  Does Conversational Interviewing Reduce Survey Measurement Error , 1997 .

[52]  W. Aquilino,et al.  Response Effects Due to Bystander Presence in CASI and Paper-and-Pencil Surveys of Drug Use and Alcohol Use , 2000, Substance use & misuse.

[53]  R. Hayes,et al.  “A bit more truthful”: the validity of adolescent sexual behaviour data collected in rural northern Tanzania using five methods , 2004, Sexually Transmitted Infections.

[54]  Roger Tourangeau,et al.  Experiments in Producing Nonresponse Bias , 2006 .