An Experimental Evaluation of Popular Well-Being Measures

Drawing on data from two multitrait multimethod experiments carried out in the context of the German Socio-Economic Panel Study (SOEP), this paper identifies questionnaire designs that minimize measurement error in reports of subjective well-being. Among the survey instruments most often used to measure well-being, the analysis focuses on three response formats (11-point, 7-point and magnitude satisfaction scales) and three modes of data collection (self-administered paperand-pencil questionnaires (SAQ), personal paper-and-pencil interviews (PAPI) and computer-assisted personal interviews (CAPI)). Results show that both the choice of a response format and the choice of a mode of data collection make a difference in terms of measurement error: The 11-point satisfaction scale and both CAPI and PAPI improve the quality of subjective well-being data. The paper also reports differences between response formats in terms of their ease of administration and illustrates that the choice of a survey instrument affects conclusions drawn from applied well-being research.

[1]  R. P. McDonald,et al.  Evaluation of Measurement Instruments by Meta-Analysis of Multitrait Multimethod Studies. , 1992 .

[2]  M. Browne,et al.  The decomposition of multitrait-multimethod matrices. , 1984, The British journal of mathematical and statistical psychology.

[3]  Elizabeth Martin,et al.  METHODS FOR TESTING AND EVALUATING SURVEY QUESTIONS , 2004 .

[4]  Paul Frijters,et al.  The measurement of welfare and well-being; the Leyden approach , 1999 .

[5]  Toni C. Antonucci,et al.  Quality of Life in Older Persons: Meaning and Measurement , 1980 .

[6]  B. Frey,et al.  Happiness, Economy and Institutions , 2000, SSRN Electronic Journal.

[7]  Wolfgang Glatzer,et al.  Lebensqualität in der Bundesrepublik : objektive Lebensbedingungen und subjektives Wohlbefinden , 1984 .

[8]  S. Nolen-Hoeksema,et al.  Gender differences in well-being. , 1999 .

[9]  M. Kroh.  Surveying the Left-Right Dimension : The Choice of a Response Format , 2005 .

[10]  R. Fernandez,et al.  A Multilevel Model of Life Satisfaction: Effects of Individual Characteristics and Neighborhood Composition , 1981 .

[11]  Richard E. Lucas,et al.  Subjective Weil-Being: Three Decades of Progress , 2004 .

[12]  J. Krosnick,et al.  AN EVALUATION OF A COGNITIVE THEORY OF RESPONSE-ORDER EFFECTS IN SURVEY MEASUREMENT , 1987 .

[13]  M. Hughes,et al.  The continuing significance of race: a study of race, class, and quality of life in America, 1972-1985. , 1986, American sociological review.

[14]  K. Motamedi.  The quality of American life , 1976 .

[15]  W. Saris.  The Strength of the Causal Relationship between Living Conditions and Satisfaction , 2001 .

[16]  D. Ehrlich.  Americans View Their Mental Health. , 1961 .

[17]  M. Hughes,et al.  The continuing significance of race revisited : A study of race, class, and quality of life in America, 1972 to 1996 , 1998 .

[18]  M. Kroh,et al.  Rethinking the dependent variable in voting behavior: On the measurement and analysis of electoral utilities , 2006 .

[19]  Willem E. Saris,et al.  8. A New Approach to Evaluating the Quality of Measurement Instruments: The Split-Ballot MTMM Design , 2004 .

[20]  P. Amato,et al.  The Effects of Divorce and Marital Discord on Adult Children's Psychological Well-being , 2001, American Sociological Review.

[21]  D. Kahneman,et al.  Well-being : the foundations of hedonic psychology , 1999 .

[22]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[23]  R. Easterlin.  Happiness in Economics , 2002 .

[24]  Andrew J. Oswald,et al.  Preferences over Inflation and Unemployment: Evidence from Surveys of Happiness , 2001 .

[25]  R. Inglehart.  Culture Shift in Advanced Industrial Society , 1991 .

[26]  Willem E. Saris.  A measurement model for psychophysical scaling , 1988 .

[27]  Duane F. Alwin,et al.  Distributive Justice and Satisfaction with Material Well-Being , 1987 .

[28]  E. Diener,et al.  Further validation of the Satisfaction with Life Scale: evidence for the cross-method convergence of well-being measures. , 1991, Journal of personality assessment.

[29]  M. Larsen,et al.  The Psychology of Survey Response , 2002 .

[30]  Frank M. Andrews,et al.  Social Indicators of Well-Being , 1976 .

[31]  M. Kroh,et al.  Documentation of Sample Sizes and Panel Attrition in the German Socio Economic Panel (SOEP) 1984 - 2004 , 2005 .

[32]  Marilyn Petersen,et al.  Quality of Life in Older Persons: Meaning and Measurement. , 1981 .

[33]  Richard E. Lucas,et al.  Discriminant validity of well-being measures. , 1996, Journal of personality and social psychology.

[34]  Ulf Olsson,et al.  Maximum likelihood estimation of the polychoric correlation coefficient , 1979 .

[35]  Ed Diener,et al.  An evaluation of subjective well-being measures , 1985 .

[36]  D. Paulhus.  Measurement and control of response bias. , 1991 .

[37]  D. Alwin,et al.  Feeling Thermometers Versus 7-Point Scales , 1997 .

[38]  Jon A. Krosnick,et al.  The Reliability of Survey Attitude Measurement , 1991 .

[39]  John P. Robinson,et al.  Measures Of Personality And Social Psychological Attitudes , 1991 .

[40]  R. Winkelmann,et al.  Why Are the Unemployed So Unhappy? Evidence from Panel Data , 1998 .

[41]  N. Bradburn.  The structure of psychological well-being , 1969 .

[42]  Annette Scherpenzeel,et al.  The Evaluation of Measurement Instruments By Meta-Analysis of Multitrait-Multimethod Studies , 1993 .

[43]  A. Shapiro,et al.  On the multivariate asymptotic distribution of sequential Chi-square statistics , 1985 .

[44]  Michael Seadle.  Measurement , 2007, The Measurement of Information Integrity.

[45]  Marvin K. Opler,et al.  Reports on happiness , 1965 .

[46]  R. Easterlin.  Does Economic Growth Improve the Human Lot? Some Empirical Evidence , 1974 .

[47]  B. Frey,et al.  What Can Economists Learn from Happiness Research? , 2001, SSRN Electronic Journal.

[48]  R. Veenhoven,et al.  Happiness in nations: Subjective appreciation of life in 56 nations 1946–1992. , 1993 .

[49]  C. S. Siara.  Komponenten der Wohlfahrt : Materialien zu Lebensbedingungen und Lebensqualität in der Bundesrepublik Deutschland , 1980 .

[50]  Willem E. Saris,et al.  Memory effects in MTMM studies , 1995 .

[51]  E. Diener,et al.  Subjective well-being. , 1984, Psychological bulletin.

[52]  Roel Bosker,et al.  Multilevel analysis : an introduction to basic and advanced multilevel modeling , 1999 .

[53]  Michael Argyle,et al.  Causes and correlates of happiness. , 1999 .

[54]  Jeffrey M. Woodbridge.  Econometric Analysis of Cross Section and Panel Data , 2002 .

[55]  Jelke Bethlehem,et al.  Computer Assisted Survey Information Collection , 1998 .

[56]  E. Diener,et al.  Review of the Satisfaction with Life Scale , 1993 .

[57]  Norbert Schwarz,et al.  Assimilation and Contrast Effects in Part-Whole Question Sequences: A Conversational Logic Analysis , 1991 .

[58]  Annette Scherpenzeel,et al.  The Validity and Reliability of Survey Questions , 1997 .

[59]  M. Stones,et al.  Structural relationships among happiness scales: A second order factorial study , 1985 .

[60]  D. Campbell,et al.  Hedonic relativism and planning the good society , 1971 .

[61]  Werner Wothke,et al.  Models for Multitratit-Multimethod Matrix Analysis , 1996 .

[62]  E. Diener,et al.  Money and happiness: Income and subjective well-being across nations. , 2000 .

[63]  Charles Kenny,et al.  Does Growth Cause Happiness, or Does Happiness Cause Growth? , 1999 .

[64]  J. Krosnick.  Response strategies for coping with the cognitive demands of attitude measures in surveys , 1991 .

[65]  A. Oswald,et al.  Unhappiness and Unemployment , 1994 .

[66]  B. Fredrickson,et al.  Measurement issues in emotion research. , 1999 .

[67]  R. Easterlin.  Income and Happiness: Towards a Unified Theory , 2001 .

[68]  M. Couper,et al.  METHODS FOR TESTING AND EVALUATING SURVEY QUESTIONS , 2004 .

[69]  Roger Tourangeau,et al.  MEASURING HAPPINESS IN SURVEYS: A TEST OF THE SUBTRACTION HYPOTHESIS , 1991 .

[70]  H. Marsh.  Confirmatory Factor Analyses of Multitrait-Multimethod Data: Many Problems and a Few Solutions , 1989 .