A Review and Analysis of the Policy-Capturing Methodology in Organizational Research: Guidelines for Research and Practice

Policy-capturing has been employed extensively in the past to examine how organizational decision makers use the information available to them when making evaluative judgments. The purpose of this article is to provide researchers with guidelines for enhancing the reliability and validity of their studies. More specifically, the authors identify issues researchers may want to consider when designing such studies and offer suggestions for effectively addressing them. They draw on a review of 37 articles from 5 major journals to identify “best practice” and discuss the advantages and disadvantages of alternative approaches to resolving the various issues. The key issues are (a) the realism of the approach and its effect on both internal and external validity, (b) the limits of the full factorial design, (c) the need for orthogonal cues, (d) sample size and statistical power, and (e) the assessment of reliability. The analysis also includes comparisons with conjoint analysis, a similar methodology used in the marketing research literature.

[1]  P. Green,et al.  Conjoint Analysis in Consumer Research: Issues and Outlook , 1978 .

[2]  Edward G. Carmines,et al.  Reliability and Validity Assessment , 1979 .

[3]  Donald P. Schwab,et al.  The role of pay and market pay variability in job application decisions. , 1983 .

[4]  Thomas W. Zimmerer,et al.  Modeling Strategic Acquisition Policies: A Simulation of Executives' Acquisition Decisions , 1984 .

[5]  Mary E. Graham,et al.  Consideration of the Incomplete Block Design for Policy-Capturing Research , 2001 .

[6]  Ray W. Cooksey,et al.  Judgment analysis : theory, methods, and applications , 1996 .

[7]  Dick R. Wittink,et al.  Comparing Derived Importance Weights Across Attributes , 1982 .

[8]  Wayne F. Cascio,et al.  Performance Appraisal Decisions as a Function of Rater Training and Purpose of the Appraisal , 1982 .

[9]  Jane S. Webster,et al.  Rational and social theories as complementary explanations of communication media choices: two polic , 1995 .

[10]  Sheldon Zedeck,et al.  An information processing model and approach to the study of motivation , 1977 .

[11]  A. Tversky,et al.  Prospect theory: analysis of decision under risk , 1979 .

[12]  F. W. Gibson,et al.  Clarifying Performance Appraisal Criteria , 1981 .

[13]  Shreya T. M. Sarkar-Barney,et al.  Research Design, Measurement, Andeffects of Attribute Range on Job Choice:More Than Meets the Eye , 1999 .

[14]  Timothy A. Judge,et al.  Effects of Work Values on Job Choice Decisions , 1992 .

[15]  T. Judge,et al.  PAY PREFERENCES AND JOB SEARCH DECISIONS: A PERSON-ORGANIZATION FIT PERSPECTIVE , 1994 .

[16]  J. Elashoff,et al.  Multiple Regression in Behavioral Research. , 1975 .

[17]  Peter H. Rossi,et al.  Measuring social judgments : the factorial survey approach , 1983 .

[18]  Stephen A. Stumpf,et al.  Capturing Rater Policies in Evaluating Candidates for Promotion , 1981 .

[19]  Lynda Aiman-Smith,et al.  Conducting Studies of Decision Making in Organizational Contexts: A Tutorial for Policy-Capturing and Other Regression-Based Techniques , 2002 .

[20]  H. N. Wheeler,et al.  MANAGERIAL DECISION MAKING ABOUT EMPLOYEE DISCIPLINE: A POLICY‐CAPTURING APPROACH , 1990 .

[21]  Herbert G. Heneman,et al.  MANAGERIAL SALARY‐RAISE DECISIONS: A POLICY‐ CAPTURING APPROACH , 1987 .

[22]  Mark V. Roehling,et al.  “EXTRACTING” POLICY FROM JUDICIAL OPINIONS: THE DANGERS OF POLICY CAPTURING IN A FIELD SETTING , 1993 .

[23]  Kenneth M. York.  Defining Sexual Harassment in Workplaces: A Policy-Capturing Approach , 1989 .

[24]  Timothy A. Judge,et al.  The Role of Human Resource Systems in Job Applicant Decision Processes , 1994 .

[25]  Ronald J. Ebert,et al.  Policy capturing in the employment interview. , 1986 .

[26]  J. Madden.  USING POLICY-CAPTURING TO MEASURE ATTITUDES IN ORGANIZATIONAL DIAGNOSIS , 1981 .

[27]  Abdelmagid M. Mazen.  The moderating role of social desirability, age, and experience in human judgment: Just how indirect is policy capturing? , 1990 .

[28]  John J. Lawler,et al.  A policy-capturing investigation of the role of expectancies in decisions to pursue job alternatives. , 1983 .

[29]  Paul E. Green,et al.  Conjoint Analysis in Marketing: New Developments with Implications for Research and Practice , 1990 .

[30]  Michael A. Hitt,et al.  A Methodology to Develop the Criteria and Criteria Weightings for Assessing Subunit Effectiveness in Organizations , 1979 .

[31]  H. J. Arnold,et al.  Social desirability response bias in self-report choice situations. , 1981 .

[32]  Murray R. Barrick,et al.  Relative importance of personality and general mental ability in managers' judgments of applicant qualifications. , 1995, The Journal of applied psychology.

[33]  S. Zedeck,et al.  Capturing rater policies for processing evaluation data. , 1977, Organizational behavior and human performance.

[34]  Edward L. Levine,et al.  Determining important tasks within jobs: A policy-capturing approach. , 1989 .

[35]  Steve H. Barr,et al.  Managerial selection decision models: examination of configural cue processing , 1989 .

[36]  Timothy A. Judge,et al.  A Policy-Capturing Approach to Individuals' Decisions to Be Absent , 1994 .

[37]  David M. Lane,et al.  Measuring the importance of cues in policy capturing , 1982 .

[38]  R. Darlington,et al.  Multiple regression in psychological research and practice. , 1968, Psychological bulletin.

[39]  Richard W. Beatty,et al.  A Policy-Capturing Approach to the Study of United States and Japanese Managers' Compensation Decisions , 1988 .

[40]  Vijay Mahajan,et al.  On the Creation of Acceptable Conjoint Analysis Experimental Designs , 1991 .

[41]  Michael T. Brannick,et al.  Nonlinear and noncompensatory processes in performance evaluation , 1989 .

[42]  Amy L. Pablo.  Determinants of acquisition integration level: A decision-making perspective. , 1994 .

[43]  Dick R. Wittink,et al.  The effect of differences in the number of attribute levels on conjoint results , 1990 .

[44]  A. Baron.  Experimental Designs , 1990, The Behavior analyst.

[45]  Joseph J. Martocchio,et al.  CHINESE AND AMERICAN MANAGERS' COMPENSATION AWARD DECISIONS: A COMPARATIVE POLICY‐CAPTURING STUDY , 2001 .

[46]  Paul E. Green,et al.  On the Design of Choice Experiments Involving Multifactor Alternatives , 1974 .

[47]  Timothy A. Judge,et al.  When We Don’t See Eye to Eye: Discrepancies between Supervisors and Subordinates in Absence Disciplinary Decisions , 1995 .

[48]  H. J. Arnold,et al.  Position choice: Comparing the importance of organizational and job factors. , 1978 .

[49]  M. Barrick,et al.  Decision-making effects on compensation surveys : implications for market wages , 1992 .

[50]  J. Hollenbeck,et al.  Goal Importance, Self-Focus, and the Goal-Setting Process , 1987 .

[51]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[52]  C. Olson,et al.  A Comparison of Interest Arbitrator Decisionmaking in Experimental and Field Settings , 1992 .

[53]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[54]  P. Sackett,et al.  The Role of Prescribed and Nonprescribed Behaviors in Estimating the Dollar Value of Performance , 1989 .

[55]  T. Judge,et al.  Dispositional Influences on Attributions Concerning Absenteeism , 1996 .

[56]  P. Muchinsky,et al.  Assessing raters' policies in evaluating proposed services for transporting the physically handicapped. , 1984, The Journal of applied psychology.