Construct Validity and Error Components of Survey Measures: A Structural Modeling Approach

Measurement errors can have profound effects on statistical relationships, and better information on the quality of measures seems needed. This study uses a new technology-structural modeling of data from special supplements to regular surveys-to generate estimates of construct validity, method effects (a major source of correlated error), and residual error (mainly random error) for a broad set of measures obtained from five national surveys and an organizational survey (total respondents = 7,706). Analysis of these estimates suggested that a typical survey item, when administered by a respected survey organization to a general population sample, can be expected to yield 50-83 percent valid variance, 0-7 percent method effects variance, and 14-48 percent residual variance. Multivariate analysis showed that over two-thirds of the variation in measurement quality could be explained by 13 survey design characteristics; characteristics of respondents explained a small additional portion. Results provide: (a) information on design conditions associated with better (or worse) measurement quality, (b) empirically based suggestions for improving measurement quality in future surveys, and (c) a set of coefficients for predicting the quality of measures not studied here. Frank M. Andrews is Program Director in the Survey Research Center and Professor of Psychology and of Population Planning at the University of Michigan. Gerald A. Cole and Mary Grace Moore made numerous and substantial contributions to the work reported here. The author is grateful to David Bowers, Angus Campbell, Charles Cannell, Philip Converse, Richard Curtin, Daniel Denison, and Robert Groves for allowing us to include methodological supplements in some of their surveys. Earlier versions of this paper were presented at the 1980 Annual Meeting of the American Psychological Association, at the 1982 Annual Meeting of the American Association for Public Opinion Research, and at the 1982 Conference on Health Survey Research Methods. This research was supported by grant #SOC78-07676 from the National Science Foundation. Public Opinion Quarterly Vol. 48:409-442 ? 1984 by the Trustees of Columbia University Published by Elsevier Science Publishing Co., Inc. 0033-362X/84/0048-409/$2.50 This content downloaded from 157.55.39.174 on Wed, 31 May 2017 18:33:53 UTC All use subject to http://about.jstor.org/terms 410 FRANK M. ANDREWS complex. Under certain combinations of error, an observed relationship can be "wrong" in both direction and magnitude. However, if one has information about the validity and error composition of the measures being analyzed, more informed judgments can be made about the underlying relationships that are of primary interest. Insightful survey researchers have always been interested in the quality of their data, and new information about data quality has been a major contributor to the development of survey technology. Much attention has been devoted to sampling errors, and there now exist good ways to estimate their magnitudes and much knowledge about how to reduce them. One kind of measurement error, bias (a consistent tendency for a measure to be higher or lower than it "should be") has also received considerable attention (Sudman and Bradburn, 1974). However, while bias can produce serious distortions in percentages, means, and other measures of central tendency, and hence is a threat that must always be considered, a bias that is constant for all respondents does not affect linear relationships at either the bivariate or multivariate level. It is other kinds of measurement errors that intrude on relationships-random and correlated measurement errors. 1 (Key terms are defined below.) These are the kinds of measurement errors investigated in this study. For each of the measures included in any particular analysis, one would like, ideally, to be able to apportion the total variance into three components: valid variance, correlated error variance, and random error variance. From this, one could know the extent to which the true bivariate relationships (i.e., the relationships among the concepts being investigated) were being attenuated (because of random measurement error) and/or inflated (because of correlated measurement error). In addition, one could sort out the complex effects that random and correlated measurement errors have on multivariate statistics such as regression coefficients, multiple and partial correlation coefficients, and path coefficients.2 A pair of examples, taken from the data of this study, will illustrate how misleading even a simple bivariate relationship between observed measures can be when allowance is not made for the effects of measurement errors. In Survey 2, the observed product-moment correlation between items having to do with perceptions about changes in I The conceptualization of measurement quality used here is similar to that discussed by Heise and Bohrnstedt (1970) and by Zeller and Carmines (1980). 2 The important impact that measurement errors have on statistics of relationships has received some attention in recent years (e.g., in sociology by Bohrnstedt and Carter, 1971; in psychology by Linn and Werts, 1973; in political science by Asher, 1974; in statistics by Cochran, 1970), but it still goes unrecognized by many data analysts. This content downloaded from 157.55.39.174 on Wed, 31 May 2017 18:33:53 UTC All use subject to http://about.jstor.org/terms CONSTRUCT VALIDITY AND ERROR COMPONENTS OF SURVEYS 411 business conditions over the past year and in the coming year averaged .41.3 After allowing for measurement error, however, the true relationship between respondents' perceptions was estimated to be .70. Thus in terms of overlapped variance, the observed relationship was only about one-third of what it should have been (17 percent versus 49 percent). In this case, random errors led to a gross deflation of the relationship. However, this does not always occur. Survey 5 produced a relationship of .44 between evaluations of own health and of work that had to be done around the house; but after allowing for measurement error, the true relationship was estimated to be .30. Here, correlated error overwhelmed random error, and the observed percentage of overlapped variance was more than double what it should have been (19 percent versus 9 percent). This study has four major goals, none of which has been pursued previously in a large-scale and systematic way: (1) Test the feasibility of incorporating a particular kind of methodological supplement in regular ongoing national and organizational surveys and of using structural model estimation techniques to generate estimates of measurement quality. (2) Provide descriptive information about estimated construct validity, method effects, and residual error for a broad range of survey measures as implemented by the standard data collection procedures of a respected survey organization. (3) Account for why some survey measures have higher (or lower) measurement quality than others. (4) Provide a means for predicting the construct validity and error components of other survey measures not actually examined in this study. These goals lead to a more general outcome of considerable importance to survey researchers and other users of survey data: more knowledge about how to produce better data. 1. Basic Notions About Validity and Measurement Error