Differences in stakeholder perceptions about training evaluation: a concept mapping/pattern matching investigation

Concept mapping and pattern matching techniques were used in exploratory research to investigate diAerences in stakeholder perceptions of training results and evaluation in a major division of a multinational network-design and engineering company. Referencing a research framework informed by multiple constituency views of organizational eAectiveness, this single case study examined group perceptions of the results and evaluation of training among 39 knowledge workers organized into three equal groups of 13. These groups consisted of (1) line managers as sponsors of training, (2) product developers as participants in training, and (3) training professionals as providers of training. A set of 100 statements were ‘‘brainstormed’’ by these groups to describe ideal results of training in terms of the success of the case organization. After sorting the 100 statements for conceptual similarity, all study participants also rated each statement’s importance twice — once as a training result, and once as a training evaluation criterion. Using the concept maps developed by the groups, a set of 21 pattern matches were performed to investigate stakeholder diAerences both within (intra) and between (inter) groups. Correlation (Pearson r) coeAcients were also calculated for each pattern match and displayed in tabular form for comparison. The overall results revealed that all stakeholder groups agreed reasonably well about the importance of training results in the organization. However, substantial diAerences were found regarding how each stakeholder group rated these results in terms of their importance for training evaluation. These diAerences were further found to correspond quite well with the unique organizational role of each stakeholder group. The results are discussed in terms of the potentials and limitations of concept mapping and pattern matching in training evaluation research, and their implications for training evaluation practice. 7 2000 Published by Elsevier Science Ltd. All rights reserved.

[1]  Jack J. Phillips,et al.  Handbook of Training Evaluation and Measurement Methods , 1987 .

[2]  Beverly Geber,et al.  Does Your Training Make a Difference? Prove It!. , 1995 .

[3]  M. Patton.  Utilization-focused evaluation: The new century text , 1997 .

[4]  G. Rao,et al.  Human resource development , 2018, Human Resource Management.

[5]  Carol H. Weiss,et al.  The stakeholder approach to evaluation: Origins and promise , 1983 .

[6]  P. Drucker.  Post-Capitalist Society , 1993 .

[7]  Kenneth H. Blanchard,et al.  Great ideas revisited , 1996 .

[8]  Lyn M. Shulha,et al.  Evaluation Use: Theory, Research, and Practice since 1986 , 1997 .

[9]  Darl Kolb,et al.  Concept Mapping Organizational Cultures , 1997 .

[10]  Anthony S. Bryk,et al.  Stakeholder-based evaluation , 1984 .

[11]  Robert O. Brinkerhoff,et al.  Evaluating Training Programs in Business and Industry. , 1989 .

[12]  James W. Altschuld,et al.  Assessing the Effectiveness of Research Organizations , 1995 .

[13]  Stephen Joel Gill.  Using evaluation to build commitment to training , 1989 .

[14]  Alvin Toffler,et al.  パワーシフト : 21世紀へと変容する知識と富と暴力 = Powershift : knowledge, wealth, and violence at the edge of the 21st century , 1990 .

[15]  Stuart Jay Deutsch,et al.  Organizational Effectiveness: A Multiple-Constituency Approach , 1980 .

[16]  Arnold J. Love,et al.  Internal Evaluation: Building Organizations from Within , 1991 .

[17]  Jeffrey Pfeffer,et al.  Competitive advantage through people : unleashing the power of the work force , 1995 .

[18]  Jack J. Phillips.  ROI: The Search for Best Practices. , 1996 .

[19]  S. Mann.  Intellectual Capital: The New Wealth of Organizations , 1999 .

[20]  Eleanor Chelimsky.  The politics of program evaluation , 1987 .

[21]  William M. K. Trochim,et al.  An introduction to concept mapping for planning and evaluation. , 1989 .

[22]  W. Trochim.  Outcome pattern matching and program theory , 1989 .

[23]  R. L. Shotland,et al.  Stakeholder-Based Evaluation and Value Judgments , 1985 .

[24]  John B. Miner,et al.  The management process: cases and readings , 1973 .

[25]  Joseph A. Maxwell,et al.  Qualitative Research Design: An Interactive Approach , 1996 .

[26]  Milbrey W. McLaughlin,et al.  Evaluation and education : at quarter century , 1992 .

[27]  Marshall W. Meyer,et al.  Power in Organizations. , 1982 .

[28]  Timothy R. Hinkin,et al.  A Brief Tutorial on the Development of Measures for Use in Survey Questionnaires , 1998 .

[29]  A. Bandura.  Social Foundations of Thought and Action: A Social Cognitive Theory , 1985 .

[30]  William M. K. Trochim,et al.  Pattern Matching, Validity, and Conceptualization in Program Evaluation , 1985 .

[31]  Robert E. Stake,et al.  Stakeholder influence in the evaluation of Cities‐in‐Schools , 1983 .

[32]  Donald L. Kirkpatrick,et al.  Great Ideas Revisited. Techniques for Evaluating Training Programs. Revisiting Kirkpatrick's Four-Level Model. , 1996 .

[33]  Robert O. Brinkerhoff.  Using evaluation to transform training , 1989 .

[34]  Leonard Nadler,et al.  The Handbook of human resource development , 1984 .

[35]  Mary L. Broad,et al.  Transfer Of Training: Action-packed Strategies To Ensure High Payoff From Training Investments , 1992 .

[36]  William M. K. Trochim,et al.  Getting to parallel: Assessing the return on expectations of training , 1998 .

[37]  Laurie J. Bassi,et al.  Changes in Benchmarked Training. , 1996 .

[38]  Jac Fitz-enz.  Proving the Value of Training. , 1988 .

[39]  Jeff Moad.  Calculating the real benefit of training , 1995 .

[40]  G. Alliger,et al.  KIRKPATRICK'S LEVELS OF TRAINING CRITERIA: THIRTY YEARS LATER , 1989 .

[41]  Donna J. Abernathy.  Presentation Tips From the Pros , 1999 .

[42]  R. Light,et al.  Evaluation practice in review , 1987 .

[43]  Michael Quinn Patton,et al.  Utilization-focused evaluation: The new century text (3rd ed.). , 1997 .

[44]  Michael R. Anderberg,et al.  Cluster Analysis for Applications , 1973 .

[45]  Laurie J. Bassi.  The Top Ten Trends. , 1996 .

[46]  Lorna Earl,et al.  The Case for Participatory Evaluation , 1992 .

[47]  C. Darrah.  Workplace training, workplace learning : a case study , 1995 .

[48]  Tom Carney.  Fourth Generation Evaluation , 1991 .

[49]  James D. Thompson.  Organizations in Action , 1967 .

[50]  Dominick L. Flarey,et al.  Managing in a Time of Great Change , 1996 .

[51]  C. Weiss.  Where Politics and Evaluation Research Meet , 1993 .

[52]  Trevor J Williams,et al.  Powershift: Knowledge, wealth and violence at the edge of the 21st century , 1991 .

[53]  Donna J. Abernathy.  Thinking outside the evaluation box , 1999 .

[54]  Greg Kearsley.  Costs, benefits, and productivity in training systems , 1982 .

[55]  W. Trochim.  Concept mapping: Soft science or hard art? , 1989 .

[56]  J. Brown,et al.  Organizational Learning and Communities-of-Practice: Toward a Unified View of Working, Learning, and Innovation , 1991 .

[57]  L. Earl,et al.  Participatory Evaluation In Education: Studies Of Evaluation Use And Organizational Learning , 1995 .

[58]  Colla J. MacDonald,et al.  Conceptualizing the successful product development project as a basis for evaluating management training in technology-based companies: a participatory concept mapping application , 1998 .

[59]  D. Palumbo,et al.  The Politics of Program Evaluation , 1988 .

[60]  Henry Mintzberg,et al.  Power in and Around Organizations , 1983 .