Impact of the level of aggregation on response accuracy in surveys of behavioral frequency

Through observations of AT&T data and the results of two of our own studies, we discovered interesting asymmetrical effects of the level of aggregation of the question on the response accuracy in surveys of behavioral frequency. We find that disaggregating a question to a lower, less comfortable level of aggregation creates greater uncertainty, leading to larger absolute errors in survey responses. However, if a question is asked at a higher less comfortable level, the majority of respondents escape by splitting questions down to the natural level, thereby avoiding greater uncertainty and thus responding more accurately. We argue that for greater accuracy in surveys, one should identify the natural level of aggregation at which a question should be posed. But when in doubt, it is better to ask a question at a higher level of aggregation because of possible escapability downward.

[1]  R. E. Turner,et al.  Error in Judgmental Sales Forecasts: Theory and Results , 1973 .

[2]  Eleanor Rosch,et al.  Principles of Categorization , 1978 .

[3]  Scot Burton,et al.  TASK CONDITIONS, RESPONSE FORMULATION PROCESSES, AND RESPONSE ACCURACY FOR BEHAVIORAL FREQUENCY QUESTIONS IN SURVEYS , 1991 .

[4]  E. Blair,et al.  Cognitive Processes Used by Survey Respondents to Answer Behavioral Frequency Questions , 1987 .

[5]  P. Golden,et al.  Methodology of the Survey of Consumer Expenditures in 1950, Study of Consumer Expenditures, Incomes and Savings in the United States. , 1960 .

[6]  Wayne D. Gray,et al.  Basic objects in natural categories , 1976, Cognitive Psychology.

[7]  E. Rosch,et al.  Cognition and Categorization , 1980 .

[8]  John McKenzie,et al.  The Accuracy of Telephone Call Data Collected by Diary Methods , 1983 .

[9]  L Hasher,et al.  Automatic processing of fundamental information: the case of frequency of occurrence. , 1984, The American psychologist.

[10]  K. Fiedler,et al.  Two halfs may be more than one whole : category-split effects on frequency illusions , 1994 .

[11]  Yoram Wind,et al.  On the Measurement of Purchase Data: Surveys versus Purchase Diaries , 1979 .

[12]  G. Menon,et al.  The Effects of Accessibility of Information in Memory on Judgments of Behavioral Frequencies , 1993 .

[13]  Christiaan Grootaert,et al.  The Use of Multiple Diaries in a Household Expenditure Survey in Hong Kong , 1986 .

[14]  D. Wittink,et al.  A comparison and an exploration of the forecasting accuracy of a loglinear model at different levels of aggregation , 1994 .

[15]  N. Schwarz,et al.  BEHAVIORAL FREQUENCY JUDGMENTS - AN ACCESSIBILITY DIAGNOSTICITY FRAMEWORK , 1995 .

[16]  A. Rifkin,et al.  Evidence for a basic level in event taxonomies , 1985, Memory & cognition.

[17]  S. Sudman.  On the Accuracy of Recording of Consumer Panels: I , 1964 .

[18]  John B. Lansing,et al.  An investigation of response error 2 , 1961 .