Assessing Innovation: A 360‐Degree Appraisal Study

This study examines the evaluation of innovation in a wider competency framework and within a 360‐degree rating procedure among managerial‐level job holders. The total sample of 2,979 individuals consisted of 296 target employees and their 318 bosses, 1208 peers, 828 direct reports, and 329 others who provided ratings on a competency framework. The results showed significant differences in innovation‐related competence ratings between different raters. Self-ratings were significantly lower compared to the overall observer ratings and were correlated only with peer ratings. Different patterns of results were found for the lower and upper quartiles based on self-ratings. For instance, no correlations were observed between self-ratings and the ratings of any observers in the group of best self-rated individuals. Implications for practice and future research in assessment and evaluation of innovation are discussed in conclusion.

[1]  Nai‐Wen Chi,et al.  A Double-Edged Sword? Exploring the Curvilinear Relationship Between Organizational Tenure Diversity and Team Innovation: The Moderating Role of Team-Oriented HR Practices , 2009 .

[2]  S. Sonnentag,et al.  Routinization, work characteristics and their relationships with creative and proactive behaviors , 2006 .

[3]  F. Meijers,et al.  360 degree feedback: how many raters are needed for reliable ratings on the capacity to develop competences, with personal qualities as developmental goals? , 2010 .

[4]  M. Frese,et al.  Three Avenues for Future Research on Creativity, Innovation, and Initiative , 2004 .

[5]  Christina E. Shalley,et al.  Effects of coaction, expected evaluation, and goal setting on creativity and productivity , 1995 .

[6]  L. James.  Aggregation Bias in Estimates of Perceptual Agreement. , 1982 .

[7]  Markus Baer.  The strength-of-weak-ties perspective on creativity: a comprehensive examination and extension. , 2010, The Journal of applied psychology.

[8]  Teresa M. Amabile,et al.  Multirater assessment of creative contributions to team projects in organizations , 2010 .

[9]  Michael Frese,et al.  Helping to improve suggestion systems: predictors of making suggestions in companies , 1999 .

[10]  Paul S. Hempel,et al.  Transactive memory system links work team characteristics and performance. , 2007, The Journal of applied psychology.

[11]  Chris W. Clegg,et al.  Implicating trust in the innovation process , 2002 .

[12]  Dave Bartram,et al.  The Great Eight competencies: a criterion-centric approach to validation. , 2005, The Journal of applied psychology.

[13]  J. George,et al.  When openness to experience and conscientiousness are related to creative behavior: an interactional approach. , 2001, The Journal of applied psychology.

[14]  M. Frese,et al.  Innovation is not enough: climates for initiative and psychological safety, process innovations, and firm performance , 2003 .

[15]  N. Anderson,et al.  Innovation in top management teams , 1996 .

[16]  P. H. Lodhi,et al.  The Five-Factor Model of Personality , 2002 .

[17]  Michael J. Burke,et al.  On Average Deviation Indices for Estimating Interrater Agreement , 1999 .

[18]  P. Bentler,et al.  Fit indices in covariance structure modeling : Sensitivity to underparameterized model misspecification , 1998 .

[19]  Kerrie L. Unsworth,et al.  Shopfloor innovation: Facilitating the suggestion and implementation of ideas , 2000 .

[20]  Troy V. Mumford,et al.  Coming Full Circle: Using Research and Practice to Address 27 Questions About 360-Degree Feedback Programs. , 2005 .

[21]  Mohan Subramaniam,et al.  The Influence of Intellectual Capital on the Types of Innovative Capabilities , 2005 .

[22]  P. Bentler,et al.  Cutoff criteria for fit indexes in covariance structure analysis : Conventional criteria versus new alternatives , 1999 .

[23]  G. R. Oldham,et al.  Employee Creativity: Personal and Contextual Factors at Work , 1996 .

[24]  N. Anderson,et al.  Gender and ethnic group differences on personality scales in selection: Some British data , 2002 .

[25]  N. Anderson,et al.  The construct validity of three entry level personality inventories used in the UK: cautionary findings from a multiple‐inventory investigation , 2003 .

[26]  S. T. Hunt,et al.  GENERIC WORK BEHAVIOR: AN INVESTIGATION INTO THE DIMENSIONS OF ENTRY-LEVEL, HOURLY JOB PERFORMANCE , 1996 .

[27]  Winston Bennett,et al.  Effectiveness of training in organizations: a meta-analysis of design and evaluation features. , 2003, The Journal of applied psychology.

[28]  L. Mann,et al.  The relationship between individual creativity and team creativity: aggregating across people and time , 2004 .

[29]  E. Naveh,et al.  Do personal characteristics and cultural values that promote innovation, quality, and efficiency compete or complement each other? , 2004 .

[30]  Jens B. Asendorpf,et al.  Validity of Big Five personality judgments in childhood: a 9 year longitudinal study , 2003 .

[31]  Leanne E. Atwater,et al.  Antecedents and consequences of reactions to developmental 360° feedback , 2005 .

[32]  Charles E. Miller,et al.  Examining Contrast Effects in Performance Appraisals: Using Appropriate Controls and Assessing Accuracy , 2002, The Journal of psychology.

[33]  J. Salgado.  The Five Factor Model of personality and job performance in the European Community. , 1997, The Journal of applied psychology.

[34]  Bradley J. Alge,et al.  Information privacy in organizations: empowering creative and extrarole performance. , 2006, The Journal of applied psychology.

[35]  R. Woodman,et al.  Innovative Behavior in the Workplace: The Role of Performance and Image Outcome Expectations , 2010 .

[36]  T. Wall,et al.  Promoting innovation: A change study , 2006 .

[37]  M. D. Dunnette.  Handbook of Industrial and Organizational Psychology , 2005 .

[38]  P. Bliese.  Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. , 2000 .

[39]  Scott B. MacKenzie,et al.  Common method biases in behavioral research: a critical review of the literature and recommended remedies. , 2003, The Journal of applied psychology.

[40]  Laura B. Cardinal.  Technological Innovation in the Pharmaceutical Industry: The Use of Organizational Control in Managing Research and Development , 2001 .

[41]  S. Scullen.  When ratings from one source have been averaged, but ratings from another source have not : Problems and solutions , 1997 .

[42]  A. Erez,et al.  The Role-Based Performance Scale: Validity Analysis of A Theory-Based Measure , 1998 .

[43]  Hannah R. Rothstein,et al.  Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe. , 1990 .

[44]  C. Shalley,et al.  A Little Creativity Goes a Long Way: An Examination of Teams’ Engagement in Creative Processes , 2004 .

[45]  Andrew P. McAfee,et al.  Race against the machine : how the digital revolution is accelerating innovation, driving productivity, and irreversibly transforming employment and the economy , 2011 .

[46]  Michael A. West,et al.  Innovation, cultural values, and the management of change in British hospitals , 1992 .

[47]  Teresa M. Amabile,et al.  Time Pressure and Creativity in Organizations: A Longitudinal Field Study , 2002 .

[48]  Michael R. Braun,et al.  Managerial Risk, Innovation, and Organizational Decline , 2009 .

[49]  M. Lindell,et al.  A Revised Index of Interrater Agreement for Multi-Item Ratings of a Single Target , 1999 .

[50]  L. Atwater,et al.  Understanding self‐perception accuracy: Implications for human resource management , 1993 .

[51]  B. Nijstad,et al.  The routinization of innovation research: A constructively critical review of the state-of-the-science , 2004 .

[52]  Wen Feng,et al.  Research excellence and departmental climate in British universities , 1998 .

[53]  Yair Berson,et al.  CEO values, organizational culture and firm outcomes. , 2008 .

[54]  Murray R. Barrick,et al.  Personality and Performance at the Beginning of the New Millennium: What Do We Know and Where Do We Go Next? , 2001 .

[55]  Jeremy Dawson,et al.  “I don't see me like you see me, but is that a problem?” Cultural influences on rating discrepancy in 360-degree feedback instruments , 2010 .

[56]  Terry A. Beehr,et al.  Evaluation of 360 degree feedback ratings: relationships with each other and with performance and selection predictors† , 2001 .

[57]  Cheri Ostroff,et al.  UNDERSTANDING SELF-OTHER AGREEMENT: A LOOK AT RATER AND RATEE CHARACTERISTICS, CONTEXT, AND OUTCOMES , 2004 .

[58]  Richard R. Reilly,et al.  An investigation of the validity of expert true score estimates in appraisal research , 1989 .

[59]  Michael D. Ensley,et al.  A reciprocal and longitudinal investigation of the innovation process: the central role of shared vision in product and process innovation teams (PPITs) , 2004 .

[60]  Ute R. Hülsheger,et al.  Team-level predictors of innovation at work: a comprehensive meta-analysis spanning three decades of research. , 2009, The Journal of applied psychology.

[61]  C. Shalley,et al.  Effects of social-psychological factors on creative performance: the role of informational and controlling expected evaluation and modeling experience. , 2001, Organizational behavior and human decision processes.

[62]  L. James,et al.  Estimating within-group interrater reliability with and without response bias. , 1984 .

[63]  M. Frese,et al.  Action as the core of work psychology: A German approach. , 1994 .