Enhancing the Validity and Cross-cultural Comparability of Survey Research 1

We offer a new approach to writing survey questions and a new statistical model that together at least partially ameliorate two long-standing problems in survey research. The first is how to measure complicated concepts, such as freedom, health, political efficacy, pornography, etc., that researchers know how to define clearly only with reference to examples. The second problem is when different respondents interpret identical survey questions in incomparable ways, as can occur when comparing respondents in different countries speaking different languages, but it also occurs frequently with different groups in the same country. Our approach to these problems is to ask respondents for self-assessments of the concept being measured along with assessments, on the same scale, of each of several hypothetical individuals described by short vignettes. The actual (but not necessarily reported) levels for the people in the vignettes are, by the design of the survey, invariant over respondents and thus provide anchors for our statistical model to transform the self-assessments to a comparable scale. With analysis, simulations, and real surveys in several countries, we show how ignoring these problems can lead to the wrong substantive conclusions and how our approach can fix them. Our methods build on insights from application-specific research on voters and legislators in political science to produce a more general measurement device.

[1]  L. Jones.  Measurement of Values , 1959, Nature.

[2]  A. Furnham,et al.  The Perceived Efficacy and Risks of Complementary and Alternative Medicine and Conventional Medicine: A Vignette Study1 , 2001 .

[3]  L. Suchman,et al.  Interactional Troubles in Face-to-Face Survey Interviews , 1990 .

[4]  Cass R. Sunstein,et al.  Shared Outrage and Erratic Awards: The Psychology of Punitive Damages , 1998 .

[5]  T. Johnson.  Approaches to equivalence in cross-cultural and cross-national survey research , 1998 .

[6]  Hadley Cantril,et al.  The pattern of human concerns , 1965 .

[7]  J. Chang,et al.  Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition , 1970 .

[8]  Jeffrey B. Lewis,et al.  Estimating Voter Preference Distributions from Individual-Level Voting Data , 2001, Political Analysis.

[9]  William Stout,et al.  A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF , 1993 .

[10]  Keith T. Poole,et al.  Ideology, Party, and Voting in the U.S. Congress, 1959–1980 , 1985, American Political Science Review.

[11]  Andrew W. Mead.  Review of the Development of Multidimensional Scaling Methods , 1992 .

[12]  G. Powell.  Workplace Romances between Senior- Level Executives and Lower-Level Employees: An Issue of Work Disruption and Gender , 2001 .

[13]  L. Baum.  Measuring Policy Change in the U.S. Supreme Court , 1988, American Political Science Review.

[14]  J. Alt,et al.  Individual differences scaling and group attitude structures: British party imagery in 1974 , 1976 .

[15]  K. T. Poole,et al.  Patterns of congressional voting , 1991 .

[16]  M. Konstantareas,et al.  Preschoolers perceptions of the unfairness of maternal disciplinary practices. , 2001, Child abuse & neglect.

[17]  Joshua D. Clinton,et al.  The Statistical Analysis of Roll Call Data , 2004, American Political Science Review.

[18]  David Firth,et al.  Modelling subjective use of an ordinal response scale in a many period crossover experiment , 2002 .

[19]  G. King,et al.  Analyzing Incomplete Political Science Data: An Alternative Algorithm for Multiple Imputation , 2001, American Political Science Review.

[20]  Wim Groot,et al.  Job satisfaction and preference drift , 1999 .

[21]  James M. Enelow,et al.  The Spatial Theory of Voting: An Introduction , 1984 .

[22]  Keith T. Poole,et al.  RECOVERING A BASIC SPACE FROM A SET OF ISSUE SCALES , 1998 .

[23]  Lawrence E. Jones,et al.  Risk perception and victim perception: The judgment of HIV cases , 1996 .

[24]  D. Freedman,et al.  A solution to the ecological inference problem , 1997 .

[25]  D. B. Clarkson.  A random effects individual difference multidimensional scaling model , 2000 .

[26]  Thomas R. Palfrey,et al.  Experimental foundations of political science , 1993 .

[27]  A. Stewart,et al.  Health-Related Quality-of-Life Assessments in Diverse Population Groups in the United States , 2000, Medical care.

[28]  Alex McConnachie,et al.  The impact of three years' ethics teaching, in an integrated medical curriculum, on students′ proposed behaviour on meeting ethical dilemmas , 2002, Medical education.

[29]  J. Teresi,et al.  Modern psychometric methods for detection of differential item functioning: application to cognitive assessment measures. , 2000, Statistics in medicine.

[30]  Jim Albert,et al.  Ordinal Data Modeling , 2000 .

[31]  John H. Aldrich,et al.  A Method of Scaling with Applications to the 1968 and 1972 Presidential Elections , 1977, American Political Science Review.

[32]  Joseph L. Zinnes,et al.  Theory and Methods of Scaling. , 1958 .

[33]  Paul M. Sniderman,et al.  Innovations in Experimental Design in Attitude Surveys , 1996 .

[34]  J. Heckman,et al.  Linear Probability Models of the Demand for Attributes with an Empirical Application to Estimating the Preferences of Legislators , 1996 .

[35]  Henry E. Brady.  Factor and ideal point analysis for interpersonally incomparable data , 1989 .

[36]  Gary King,et al.  A Unified Method of Evaluating Electoral Systems and Redistricting Plans , 1994 .

[37]  Gordon W. Cheung,et al.  Assessing Extreme and Acquiescence Response Sets in Cross-Cultural Research Using Structural Equations Modeling , 2000 .

[38]  R. Fay,et al.  An application of Rasch analysis to questionnaire design : using vignettes to study the meaning of work in the Current Population Survey , 1991 .

[39]  J. Carroll,et al.  How separate are the sensory, emotional, and motivational dimensions of pain? A multidimensional scaling analysis. , 2001, Somatosensory & motor research.

[40]  Peter H. Rossi,et al.  Measuring Social Judgements: The Factorial Survey Approach. , 1984 .

[41]  R. J. Mokken,et al.  Handbook of modern item response theory , 1997 .

[42]  W. Groot,et al.  Adaptation and scale of reference bias in self-assessments of quality of life. , 2000, Journal of health economics.

[43]  Amartya Sen,et al.  Health: perception versus observation , 2002, BMJ : British Medical Journal.

[44]  N. J. Finkel,et al.  When principles collide in hard cases: A commonsense moral analysis. , 2001 .

[45]  J. Douglas Carroll,et al.  Psychometric Methods in Marketing Research: Part II, Multidimensional Scaling , 1997 .

[46]  Steven D. Levitt,et al.  Comparing Interest Group Scores across Time and Chambers: Adjusted ADA Scores for the U.S. Congress , 1999, American Political Science Review.

[47]  Howard Wainer,et al.  Detection of differential item functioning using the parameters of item response models. , 1993 .