The order of questions in a survey on citizen satisfaction with public services: Lessons from a split-ballot experiment

Surveys of citizen satisfaction with local public services have become widespread, with the results increasingly used to reorganize services, to allocate budgets, and to hold managers accountable. But evidence from a split-ballot experiment that we conducted suggests that the order of questions in a citizen survey has important effects on reported satisfaction with specific public services as well as overall citizen satisfaction. Moreover, the correlations of specific service ratings with overall satisfaction, and thus the identification of key drivers of overall satisfaction, also turn out to be highly sensitive to question order. These findings are in line with research on priming and question order effects in the survey methodology literature, but these effects have not been carefully examined before in the context of citizen surveys and public administration research. Policy and management implications of these finding are discussed. © 2011 The Authors. Public Administration

[1]  David H. Coursey.  Information Credibility and Choosing Policy Alternatives: An Experimental Test of Cognitive-Response Theory , 1992 .

[2]  L. Jacobs,et al.  The Impact of Political Debate on Government Trust: Reminding the Public What the Federal Government Does , 1999 .

[3]  G. G. Ryzin.  Citizen perceptions of road smoothness: evidence from New York with implications for comparative performance measurement , 2008 .

[4]  Donald P. Moynihan,et al.  What Do We Talk About When We Talk About Performance? Dialogue Theory and Performance Budgeting , 2005 .

[5]  Dennis Wittmer.  Ethical Sensitivity and Managerial Decisionmaking: An Experiment , 1992 .

[6]  Cindy D. Kam,et al.  Beyond the “Narrow Data Base”: Another Convenience Sample for Experimental Research , 2007 .

[7]  David Swindell,et al.  Linking Citizen Satisfaction Data to Performance Measures: A Preliminary Evaluation , 2000 .

[8]  David Swindell,et al.  Performance Measurement Versus City Service Satisfaction: Intra‐City Variations in Quality?* , 2005 .

[9]  Paul C. Nutt,et al.  Comparing Public and Private Sector Decision-Making Practices , 2006 .

[10]  Brian Stipak,et al.  Local Governments' Use of Citizen Surveys , 1980 .

[11]  Robert P. Berrens,et al.  The Advent of Internet Surveys for Political Research: A Comparison of Telephone and Internet Samples , 2003, Political Analysis.

[12]  R. Parks.  Linking Objective and Subjective Measures of Performance , 1984 .

[13]  G. G. Ryzin,et al.  Measuring Street Cleanliness: A Comparison of New York City’s Scorecard and Results from a Citizen Survey , 2008 .

[14]  Jeffrey L. Brudney,et al.  Urban Policy Making and Subjective Service Evaluations: Are They Compatible? , 1982 .

[15]  Philip B. Coulter,et al.  Subjective and Objective Measures of Police Service Delivery , 1983 .

[16]  Y. Schul,et al.  Measuring satisfaction with organizations. Predictions from information accessibility. , 1993, Public opinion quarterly.

[17]  Gregg G. Van Ryzin.  Validity of an On-Line Panel Approach to Citizen Surveys , 2008 .

[18]  Geert Bouckaert,et al.  Assessing the Relation Between Satisfaction with Public Service Delivery and Trust in Government. The Impact of the Predisposition of Citizens Toward Government on Evalutations of Its Performance , 2006 .

[19]  J. Benton,et al.  A QUESTION ORDER EFFECT IN A LOCAL GOVERNMENT SURVEY , 1991 .

[20]  J. Kampen,et al.  Deep Impact for High Impact Agencies? Assessing the Role of Bureaucratic Encounters in Evaluations of Government , 2003 .

[21]  B. Bozeman,et al.  Laboratory Experiments in Public Policy and Management , 1992 .

[22]  Gary J. Miller,et al.  Adaptive Incrementalism and Complexity: Experiments with Two‐Person Cooperative Signaling Games , 2003 .

[23]  James H. Kuklinski,et al.  The Growth and Development of Experimental Research in Political Science , 2006, American Political Science Review.

[24]  David Swindell,et al.  A Multiple–Indicator Approach to Municipal Service Evaluation: Correlating Performance Measurement and Citizen Satisfaction across Jurisdictions , 2002 .

[25]  D. West,et al.  Citizen Evaluations of Local Police , 2007 .

[26]  S. McFarland.  Effects of Question Order on Survey Responses , 1981 .

[27]  David Landsbergen,et al.  Decision Quality, Confidence, and Commitment with Expert Systems: An Experimental Study , 1997 .

[28]  Van Ryzin,et al.  Expectations, performance, and citizen satisfaction with urban services , 2004 .

[29]  B. Stipak.  Attitudes and Belief Systems Concerning Urban Services , 1977 .

[30]  Thomas I. Miller,et al.  Standards of Excellence: U. S. Residents' Evaluations of Local Government Services , 1991 .

[31]  J. Weiss.  Coping with complexity: An experimental study of public policy decision-making , 1982 .

[32]  P. Scott.  Assessing Determinants of Bureaucratic Discretion: An Experiment in Street-Level Decision Making , 1997 .

[33]  Carol C. Bienstock,et al.  Question order effects in measuring service quality , 2003 .

[34]  Joakim Ekman,et al.  Satisfaction with democracy: A note on a frequently used indicator in comparative politics , 2003 .

[35]  D. Green,et al.  The Effects of Canvassing, Telephone Calls, and Direct Mail on Voter Turnout: A Field Experiment , 2000, American Political Science Review.

[36]  Chris McIntyre,et al.  The Effect of Economic Priorities on the Measurement of Value Change: New Experimental Evidence , 1999, American Political Science Review.

[37]  D. Swindell,et al.  The Case For The Inexperienced User , 2003 .

[38]  Brian Stipak,et al.  Citizen Satisfaction With Urban Services: Potential Misuse as a Performance Indicator , 1979 .

[39]  Citizens and Politics: The Role of Affect in Symbolic Politics , 2001 .

[40]  Paul M. Sniderman,et al.  Innovations in Experimental Design in Attitude Surveys , 1996 .

[41]  S. Walle.  Confidence in the Criminal Justice System: Does Experience Count? , 2009 .

[42]  Lisa M. V. Gulick,et al.  Drivers and Consequences of Citizen Satisfaction: An Application of the American Customer Satisfaction Index Model to New York City , 2004 .

[43]  J. Kampen,et al.  Potential for comparative public opinion research in public administration , 2005 .

[44]  S. Bretschneider,et al.  "internal rationality" and the effects of perceived decision difficulty: Results of a public management decisionmaking experiment , 1992 .

[45]  A. Tversky,et al.  Judgment under uncertainty: Judgment under uncertainty: Heuristics and biases , 1982 .

[46]  Brian Wansink,et al.  Predicting the future of consumer panels , 2002 .

[47]  Oliver James,et al.  Evaluating the Expectations Disconfirmation and Expectations Anchoring Approaches to Citizen Satisfaction with Local Public Services , 2007 .

[48]  Brian J. Gaines,et al.  The Logic of the Survey Experiment Reexamined , 2007, Political Analysis.