I Think (I Am Doing Well), Therefore I Am: Assessing the Validity of Administrators' Self-Assessments of Performance

ABSTRACT Several prominent public management data sets rely on administrators' and sometimes bureaucrats' self-assessments of how their programs or organizations are performing. While subjective assessments of performance, particularly by clientele, are valuable, assessments by administrators raise the issue of bias. Even if there is no systematic bias, such assessments may still be problematic statistically. This analysis uses original survey and archive data to systematically compare administrative self-assessments of performance with other performance indicators. The results show that administrators' perceptions of performance are biased in predictable ways, that these biases do not reflect sophisticated assessments of organizational situations, and that the measures can produce spurious results. We caution against using administrators' perceptions of performance without other corresponding performance indicators.

[1]  René Torenvlied.  Subjective and Objective Agency Performance: A Multilevel—Multistage Approach , 2009 .

[2]  George Alexander Boyne,et al.  Public management reform and organizational performance: An empirical assessment of the U.K. Labour government's public service improvement strategy , 2006 .

[3]  Kenneth J. Meier,et al.  Employee Turnover and Organizational Performance: Testing a Hypothesis from Classical Public Administration , 2007 .

[4]  J. Kagan,et al.  Rational choice in an uncertain world , 1988 .

[5]  Petra E. Todd,et al.  On the Specification and Estimation of the Production Function for Cognitive Achievement , 2003 .

[6]  Shelley E. Taylor.  Positive illusions: Creative self-deception and the healthy mind. , 1989 .

[7]  J. Betts,et al.  Reduced-Class Distinctions: Effort, Ability, and the Education Production Function , 2009 .

[8]  Edward G. Carmines,et al.  Measurement in the social sciences , 1980 .

[9]  M. Bazerman.  Judgment in Managerial Decision Making , 1990 .

[10]  Donald P. Moynihan,et al.  The Dynamics of Performance Management: Constructing Information and Reform , 2008 .

[11]  Jennifer Zoltners Sherer,et al.  Organizational Routines as Coupling Mechanisms , 2011 .

[12]  Friedhelm Pfeiffer,et al.  Age-Dependent Skill Formation and Returns to Education: Simulation Based Evidence , 2007 .

[13]  J. S. Long,et al.  Regression models for categorical dependent variables using Stata, 2nd Edition , 2005 .

[14]  Kevin B. Smith.  Ideology of Education, The: The Commonwealth, the Market, and America's Schools , 2003 .

[15]  Marc J. Dollinger,et al.  Interorganizational and Collective Strategies in Small Firms: Environmental Effects and Performance , 1992 .

[16]  N. Shainess,et al.  Positive Illusions: Creative Self-Deception and the Healthy Mind , 1990 .

[17]  William H. Glick,et al.  Common Methods Bias: Does Common Methods Variance Really Bias Results? , 1998 .

[18]  D. Rubin,et al.  Constructing a Control Group Using Multivariate Matched Sampling Methods That Incorporate the Propensity Score , 1985 .

[19]  William D. Berry.  Understanding regression assumptions , 1993 .

[20]  Petra E. Todd,et al.  Reconciling Conflicting Evidence on the Performance of Propensity-Score Matching Methods , 2001 .

[21]  Rajeev Dehejia,et al.  Propensity Score-Matching Methods for Nonexperimental Causal Studies , 2002, Review of Economics and Statistics.

[22]  Rajeev Dehejia.  Practical propensity score matching: a reply to Smith and Todd , 2005 .

[23]  R. Schwarz.  Participative Decision Making , 1989 .

[24]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[25]  J. Guthrie.  High-Involvement Work Practices, Turnover, and Productivity: Evidence from New Zealand , 2001 .

[26]  T. C. Powell.  Organizational alignment as competitive advantage , 1992 .

[27]  Marco Caliendo,et al.  Some Practical Guidance for the Implementation of Propensity Score Matching , 2005, SSRN Electronic Journal.

[28]  Gene A. Brewer.  Public Service Performance: All measures of performance are subjective: More evidence on US federal agencies , 2006 .

[29]  William H. Bommer,et al.  ON THE INTERCHANGEABILITY OF OBJECTIVE AND SUBJECTIVE MEASURES OF EMPLOYEE PERFORMANCE: A META-ANALYSIS , 1995 .

[30]  M. Frese,et al.  Innovation is not enough: climates for initiative and psychological safety, process innovations, and firm performance , 2003 .

[31]  Hindy Lauer Schachter,et al.  Objective and Subjective Performance Measures , 2010 .

[32]  T. Wall,et al.  On the validity of subjective measures of company performance , 2004 .

[33]  Joseph A. Cote,et al.  Estimating Trait, Method, and Error Variance: Generalizing across 70 Construct Validation Studies , 1987 .

[34]  Richard M. Walker,et al.  Public Service Performance: Subjective and objective measures of organizational performance: An empirical exploration , 2006 .

[35]  J. Guzmán.  Regression Models for Categorical Dependent Variables Using Stata , 2013 .

[36]  Scott B. MacKenzie,et al.  Common method biases in behavioral research: a critical review of the literature and recommended remedies. , 2003, The Journal of applied psychology.

[37]  Robert Carton,et al.  Measuring organizational performance , 2006 .

[38]  Gregory G. Dess,et al.  Measuring organizational performance in the absence of objective measures: The case of the privately-held firm and conglomerate business unit , 1984 .

[39]  Eric A. Hanushek,et al.  Generalizations about Using Value-Added Measures of Teacher Quality , 2010 .

[40]  A. Somech.  Participative Decision Making in Schools: A Mediating-Moderating Analytical Framework for Understanding School and Teacher Outcomes , 2010 .

[41]  Laurence J. O'Toole,et al.  Subjective Organizational Performance and Measurement Error: Common Source Bias and Spurious Relationships , 2013 .

[42]  A. Tversky,et al.  Choices, Values, and Frames , 2000 .

[43]  Rhys William Andrews,et al.  Assessing Organizational Performance: Exploring Differences Between Internal and External Measures , 2010 .

[44]  R E Miles,et al.  Organizational strategy, structure, and process. , 1978, Academy of management review. Academy of Management.

[45]  Contextual Factors in the Assessment of the Effect of School-based Management on School Effectiveness , 2011 .

[46]  David W. Pitts.  Diversity Management, Job Satisfaction, and Performance: Evidence from U.S. Federal Agencies , 2009 .

[47]  Mark A. Huselid,et al.  The Impact of Human Resource Management Practices on Perceptions of Organizational Performance , 1996 .