Problems in the estimation and interpretation of the reliability of survey data

In this paper I discuss several of the difficulties involved in estimating the reliability of survey measurement. Reliability is defined on the basis of classical true-score theory, as the correlational consistency of multiple measures of the same construct, net of true change. This concept is presented within the framework of a theoretical discussion of the sources of error in survey data and the design requirements for separating response variation into components representing such response consistency and measurement errors. Discussion focuses on the potential sources of random and nonrandom errors, including “invalidity” of measurement, the term frequently used to refer to components of method variance. Problems with the estimation of these components are enumerated and discussed with respect to both cross-sectional and panel designs. Empirical examples are given of the estimation of the quantities of interest, which are the basis of a discussion of the interpretational difficulties encountered in reliability estimation. Data are drawn from the ISR's Quality of Life surveys, the National Election Studies and the NORC's General Social Surveys. The general conclusion is that both cross-sectional and panel estimates of measurement reliability are desirable, but for the purposes of isolating the random component of error, panel designs are probably the most advantageous.

[1]  Hubert M. Blalock,et al.  Estimating Measurement Error Using Multiple Indicators and Several Points in Time , 1970 .

[2]  L. F. Carter,et al.  Sociological Methodology 1980. , 1981 .

[3]  K. Jöreskog.  Statistical analysis of sets of congeneric tests , 1971 .

[4]  Robert L. Linn,et al.  Quantifying Unmeasured Variables , 1974 .

[5]  K. Krippendorff.  Bivariate Agreement Coefficients for Reliability of Data , 1970 .

[6]  Michael E. Borus,et al.  Response Bias in Reports of Father's Education and Socioeconomic Status , 1973 .

[7]  M. Susan Marquis,et al.  Survey Measurement Design and Evaluation Using Reliability Theory. , 1977 .

[8]  L. Cronbach.  Further Evidence on Response Sets and Test Design , 1950 .

[9]  Jack Block,et al.  The challenge of response sets , 1965 .

[10]  Robert M. Hauser,et al.  Response Error in Earnings Functions for Nonblack Males , 1977 .

[11]  Duane F. Alwin,et al.  Making Inferences from Attitude-Behavior Correlations* , 1973 .

[12]  Naihua Duan,et al.  Response Errors in Sensitive Topic Surveys: Estimates, Effects, and Correction Options , 1981 .

[13]  L. Opie.  Editor's introduction , 2004, Cardiovascular Drugs and Therapy.

[14]  M. H. Quenouille,et al.  Survey Methods in Social Investigation , 1973 .

[15]  Peter V. Miller,et al.  Matching Survey Responses to Official Records: An Exploration of Validity in Victimization Reporting , 1985 .

[16]  Paul D. Allison,et al.  Problems in Estimating Measurement Error From Panel Data , 1976 .

[17]  Robert M. Hauser,et al.  Response Errors of Nonblack Males in Models of the Stratification Process , 1977 .

[18]  Herbert B. Asher.  Some Consequences of Measurement Error in Survey Data , 1974 .

[19]  Peter V. Miller,et al.  Research on Interviewing Techniques , 1981 .

[20]  F. M. Andrews.  Construct Validity and Error Components of Survey Measures: A Structural Modeling Approach , 1984 .

[21]  Duane F. Alwin,et al.  MEASUREMENT MODELS FOR RESPONSE ERRORS IN SURVEYS: ISSUES AND APPLICATIONS , 1980 .

[22]  Arthur H. Miller,et al.  American national election studies data sourcebook, 1952-1978 , 1981 .

[23]  Robert L. Linn,et al.  A Simplex Model for Analyzing Academic Growth , 1977 .

[24]  G. William Walster,et al.  Effect of Reliability and Validity on Power of Statistical Tests , 1970 .

[25]  Donald A. Rock,et al.  Applications of Quasi-Markov Simplex Models Across Populations , 1981 .

[26]  A. Herzog,et al.  Interviewing older adults: the accuracy of factual information. , 1987, Journal of gerontology.

[27]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[28]  Richard T. Campbell,et al.  Analyzing Panel Data in Studies of Aging , 1982 .

[29]  Duane F. Alwin,et al.  Distributive Justice and Satisfaction with Material Well-Being , 1987 .

[30]  Karl G. Jöreskog,et al.  Statistical estimation of structural models in longitudinal-developmental investigations , 1979 .

[31]  Isabella C. M. Cunningham,et al.  The Ipsative Process to Reduce Response Set Bias , 1977 .

[32]  Peter Schmidt,et al.  Is there a “Socratic Effect” in Nonexperimental Panel Studies? , 1987 .

[33]  Frank M. Andrews,et al.  The Quality of Survey Data as Related to Age of Respondent , 1986 .

[34]  Karl G. Jöreskog,et al.  Analyzing psychological data by structural analysis of covariance matrices , 1974 .

[35]  Hubert M. Blalock,et al.  Causal models in panel and experimental designs , 1985 .

[36]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[37]  Edward G. Carmines,et al.  Measurement in the social sciences , 1980 .

[38]  David E. Wiley,et al.  The Estimation of Measurement Error in Panel Data , 1970 .

[39]  William T. Bielby.  Response Errors of Black and Nonblack Males in Models of Status Inheritance and Mobility. Discussion Papers 357-76. , 1976 .

[40]  D. Alwin.  Approaches to the Interpretation of Relationships in the Multitrait-Multimethod Matrix , 1973 .

[41]  Donald A. Rock,et al.  Validating Psychometric Assumptions within and between Several Populations , 1977 .

[42]  Christopher H. Achen.  Mass Political Attitudes and the Survey Response , 1975, American Political Science Review.

[43]  M. Corcoran,et al.  Sex Differences in Measurement Error in Status Attainment Models , 1980 .

[44]  Edward G. Carmines,et al.  Assessing the Reliability of Linear Composites , 1980 .

[45]  Shu-Ling Tsai,et al.  A Model of Stratification with Response Error in Social and Psychological Variables , 1983 .

[46]  Dag Sörbom,et al.  DETECTION OF CORRELATED ERRORS IN LONGITUDINAL DATA , 1975 .

[47]  Robert L. Linn,et al.  Path analysis: Psychological examples. , 1970 .

[48]  D. Hamilton,et al.  Personality attributes associated with extreme response style. , 1968, Psychological bulletin.

[49]  Wolfgang Jagodzinski,et al.  Estimation of Reliability and Stability in Single-Indicator Multiple-Wave Models , 1987 .

[50]  David R. Heise,et al.  VALIDITY, INVALIDITY, AND RELIABILITY , 1970 .

[51]  John R. Nesselroade,et al.  Longitudinal Research in the Study of Behavior and Development , 1979 .

[52]  R. D. Bock,et al.  Analysis of covariance structures , 1966, Psychometrika.

[53]  Robert S. Erikson,et al.  The SRC Panel Data and Mass Political Attitudes , 1979, British Journal of Political Science.

[54]  Duane F. Alwin,et al.  Attitude Scales an Congeneric Tests: A Re-Examination of an Attitude-Behavior Model. , 1976 .

[55]  L. Cronbach.  Response Sets and Test Validity , 1946 .

[56]  Charles N. Weaver,et al.  Validity Of Reported Date of Birth, Salary, and Seniority , 1974 .

[57]  Jon A. Krosnick,et al.  The Measurement of Values in Surveys: A Comparison of Ratings and Rankings , 1985 .

[58]  N. Feather,et al.  The measurement of values: Effects of different assessment procedures , 1973 .

[59]  D. R. Heise.  Separating reliability and stability in test-retest correlation. , 1969 .

[60]  H. Costner.  Theory, Deduction, and Rules of Correspondence , 1969, American Journal of Sociology.

[61]  Robert M. Hauser,et al.  Response Errors of Black and Nonblack Males in Models of the Intergenerational Transmission of Socioeconomic Status , 1977, American Journal of Sociology.

[62]  Karl G. Jöreskog.  ESTIMATION AND TESTING OF SIMPLEX MODELS , 1970 .

[63]  Kanichi Takagi,et al.  Human error and human reliability , 1967 .

[64]  David H. Krantz,et al.  Measurement, psychophysics, and neural information processing , 1974 .

[65]  J. B. Pearson,et al.  Methodology in Social Research. , 1968 .

[66]  J. Krosnick,et al.  AN EVALUATION OF A COGNITIVE THEORY OF RESPONSE-ORDER EFFECTS IN SURVEY MEASUREMENT , 1987 .

[67]  Ronald G. Stansfield,et al.  Sociological Methodology 1982 , 1983 .

[68]  Edgar F. Borgatta,et al.  Sociological Methodology, 1969. , 1969 .

[69]  Graham Kalton,et al.  Survey Methods in Social Investigation. , 1972 .

[70]  Van Ark,et al.  Sociological Methodology 1971 , 1973 .

[71]  K. Jöreskog.  Structural analysis of covariance and correlation matrices , 1978 .

[72]  K. Schaie,et al.  Methodological issues in aging research , 1990 .

[73]  Duane F. Alwin,et al.  Structural equation models in research on human development and aging. , 1988 .

[74]  Patrick M. O'Malley,et al.  When Four Months Equal a Year: Inconsistencies in Student Reports of Drug Use , 1981 .

[75]  Gregory B. Markus,et al.  Political Attitudes during an Election Year: A Report on the 1980 NES Panel Study , 1982, American Political Science Review.

[76]  Robert M. Groves,et al.  RESEARCH ON SURVEY DATA QUALITY , 1987 .

[77]  R. Whitman.  Response set in personality assessment , 1967 .

[78]  K. Motamedi.  The quality of American life , 1976 .

[79]  H. Schuman,et al.  The Effect of the Question on Survey Responses: A Review , 1982 .

[80]  A. Inkeles,et al.  International Encyclopedia of the Social Sciences. , 1968 .

[81]  D. Sörbom.  A GENERAL METHOD FOR STUDYING DIFFERENCES IN FACTOR MEANS AND FACTOR STRUCTURE BETWEEN GROUPS , 1974 .

[82]  David R. Heise,et al.  Comment on "The Estimation of Measurement Error in Panel Data" , 1970 .

[83]  Duane F. Alwin,et al.  FAMILY ORIGINS AND THE SCHOOLING PROCESS: EARLY VERSUS LATE INFLUENCE OF PARENTAL CHARACTERISTICS* , 1984 .

[84]  G. Bohrnstedt,et al.  Robustness in Regression Analysis , 1971 .

[85]  Philip E. Converse,et al.  Plus ça change…: The New CPS Election Study Panel , 1979, American Political Science Review.

[86]  Charles E. Werts,et al.  Using Longitudinal Data to Estimate Reliability in the Presence of Correlated Measurement Errors , 1980 .

[87]  S. Presser,et al.  Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context , 1996 .

[88]  B. Muthén,et al.  Assessing Reliability and Stability in Panel Models , 1977 .