A Test-Retest Reliability Generalization Meta-Analysis of Judgments Via the Policy-Capturing Technique

Policy capturing is a widely used technique, but the temporal stability of policy-capturing judgments has long been a cause for concern. This article emphasizes the importance of reporting reliabil...

[1]  R. Cooksey,et al.  Judgment of Factors Influencing Interest: An Australian Study , 2001 .

[2]  R. Teas.  An analysis of the temporal stability and structural reliability of metric conjoint analysis procedures , 1985 .

[3]  D. Cable,et al.  The determinants of job seekers' reputation perceptions , 2000 .

[4]  Joseph L. Mundy,et al.  Change Detection , 2014, Computer Vision, A Reference Guide.

[5]  Mary W. Downton,et al.  Judgments of photographs vs. field observations in studies of perception and judgment of the visual environment , 1984 .

[6]  Daniel M. Cable,et al.  Are You Attracted? Do You Intend to Pursue? A Recruiting Policy-Capturing Study , 2001 .

[7]  David W. Williams,et al.  Opportunity Evaluation as Rule‐Based Decision Making , 2014 .

[8]  Keith Levi,et al.  Expert systems should be more accurate than human experts: evaluation procedures from human judgement and decision making , 1989, IEEE Trans. Syst. Man Cybern..

[9]  L. L. Martins,et al.  Racioethnicity, community makeup, and potential employees' reactions to organizational diversity management approaches. , 2016, The Journal of applied psychology.

[10]  R. Crocker,et al.  Factors influencing teacher decisions on school, classroom, and curriculum , 1986 .

[11]  Jeffery S. McMullen,et al.  Venture capitalists' decision policies across three countries: an institutional theory perspective , 2007 .

[12]  M. Barrick,et al.  Decision-making effects on compensation surveys : implications for market wages , 1992 .

[13]  Seth A. Kaplan,et al.  Unpackaging Manager Mistrust in Allowing Telework: Comparing and Integrating Theoretical Perspectives , 2018 .

[14]  W Bevan,et al.  Free Recall and Abstractness of Stimuli , 1971, Science.

[15]  Brian R. Dineen,et al.  Perceived fairness of web‐based applicant screening procedures: Weighing the rules of justice and the role of individual differences , 2004 .

[16]  Wolfgang Viechtbauer,et al.  Conducting Meta-Analyses in R with the metafor Package , 2010 .

[17]  James H. Dulebohn,et al.  Employees' Perceptions of the Distributive Justice of Pay Raise Decisions: A Policy Capturing Approach , 1998 .

[18]  T. Judge,et al.  PAY PREFERENCES AND JOB SEARCH DECISIONS: A PERSON-ORGANIZATION FIT PERSPECTIVE , 1994 .

[19]  Duane E. Leigh.  Union Preferences, Job Satisfaction, and the Union-Voice Hypothesis , 1986 .

[20]  Noah Eisenkraft,et al.  Positive is usually good, negative is not always bad: The effects of group affect on social integration and task performance. , 2015, The Journal of applied psychology.

[21]  Evan C. Carter,et al.  A series of meta-analytic tests of the depletion effect: Self-control does not seem to rely on a limited resource. , 2015, Journal of experimental psychology. General.

[22]  Nick Sevdalis,et al.  Judgment analysis: a method for quantitative evaluation of trainee surgeons' judgments of surgical risk. , 2008, American journal of surgery.

[23]  R. Rafal,et al.  Shifting visual attention between objects and locations: evidence from normal and parietal lesion subjects. , 1994, Journal of experimental psychology. General.

[24]  S. Bonaccio,et al.  Evaluating advisors: A policy‐capturing study under conditions of complete and missing information , 2009 .

[25]  M. Gowan,et al.  Corporate Social Responsibility, Applicants’ Individual Traits, and Organizational Attraction: A Person–Organization Fit Perspective , 2011, Journal of Business and Psychology.

[26]  S. Groeneveld,et al.  Do tax officials use double standards in evaluating citizen-clients? A policy-capturing study among Dutch frontline tax officials , 2018 .

[27]  Herbert G. Heneman,et al.  MANAGERIAL SALARY‐RAISE DECISIONS: A POLICY‐ CAPTURING APPROACH , 1987 .

[28]  Toyah L. Miller,et al.  Assessing Mission and Resources for Social Change: An Organizational Identity Perspective on Social Venture Capitalists‘ Decision Criteria , 2010 .

[29]  L. Sulsky,et al.  Do Personal Values Influence the Propensity for Sustainability Actions? A Policy-Capturing Study , 2015 .

[30]  V. Catano,et al.  Assessing the Reliability of Situational Judgment Tests Used in High‐Stakes Situations , 2012 .

[31]  Robert M. Guion,et al.  The Prediction of Judgment in Realistic Tasks: An Investigation of Self-Insight , 2000 .

[32]  Lynda Aiman-Smith,et al.  Conducting Studies of Decision Making in Organizational Contexts: A Tutorial for Policy-Capturing and Other Regression-Based Techniques , 2002 .

[33]  S. Deshpande,et al.  Variation in Compensation Decisions by Managers: An Empirical Investigation , 1994 .

[34]  L. J. Williams,et al.  Condition 9 and 10 Tests of Model Confirmation: A Review of James, Mulaik, and Brett (1982) and Contemporary Alternatives , 2020 .

[35]  Filip Lievens,et al.  The importance of traits and abilities in supervisors' hirability decisions as a function of method of assessment , 2005 .

[36]  P. Harries,et al.  Training Novices to Make Expert, Occupationally Focused, Community Mental Health Referral Decisions , 2011 .

[37]  T. Brown,et al.  The Impact of Participant Characteristics, Perceived Motives, and Job Behaviors on Co‐Workers' Evaluations of Workplace Romances1 , 1996 .

[38]  Bethany S Cockburn,et al.  Meta‐Analysis of Coefficient Alpha: A Reliability Generalization Study , 2018 .

[39]  Dirk Heerwegh,et al.  Mode Differences Between Face-to-Face and Web Surveys: An Experimental Investigation of Data Quality and Social Desirability Effects , 2009 .

[40]  Michael Stephens,et al.  Understanding Student Judgments of Teaching Effectiveness: A “Policy Capturing” Approach , 1994 .

[41]  P. Harries,et al.  Identifying occupational therapists' referral priorities in community health. , 2003, Occupational therapy international.

[42]  Melissa Woodard Barringer,et al.  A Review and Analysis of the Policy-Capturing Methodology in Organizational Research: Guidelines for Research and Practice , 2002 .

[43]  Sheldon Zedeck,et al.  An information processing model and approach to the study of motivation , 1977 .

[44]  Deniz S. Ones,et al.  Determinants of Perceived Pay Inequities: The Effects of Comparison Other Characteristics and Pay‐System Communication1 , 2000 .

[45]  Charles Y. Murnieks,et al.  Drawn to the fire: The role of passion, tenacity and inspirational leadership in angel investing , 2016 .

[46]  Sabine B. Rau,et al.  Application of metric conjoint analysis in family business research , 2014 .

[47]  Jane M. Mackay,et al.  An Empirical Investigation of the Effects of Decision Aids on Problem-Solving Processes* , 1992 .

[48]  N. Mackworth.  The Breakdown of Vigilance during Prolonged Visual Search 1 , 1948 .

[49]  D. Read,et al.  Justice Judgments: Individual Self-Insight and Between- and Within-Person Consistency , 2016 .

[50]  Jason J. Dahling,et al.  Contextual supports and barriers to academic choices: A policy-capturing analysis , 2010 .

[51]  William H. Clover,et al.  Can we learn anything about interviewing real people from “interviews” of paper people? Two studies of the external validity of a paradigm☆ , 1978 .

[52]  Peter C. M. Molenaar,et al.  The New Person-Specific Paradigm in Psychology , 2009 .

[53]  Thomas R. Stewart,et al.  Analysis of Expert Judgment in a Hail Forecasting Experiment , 1989 .

[54]  Mary E. Graham,et al.  Consideration of the Incomplete Block Design for Policy-Capturing Research , 2001 .

[55]  Elizabeth Barley,et al.  Analysis of the cues used by patients when making assessments of their asthma severity , 2005, European Respiratory Journal.

[56]  Steve H. Barr,et al.  A Comparison of Selection Decision Models in Manager Versus Student Samples. , 1986 .

[57]  C. Joyce,et al.  Experiments on clinical observation and judgement in the assessment of depression: profiled videotapes and Judgement Analysis , 1986, Psychological Medicine.

[58]  R. Baumeister,et al.  Self-Regulation Failure: An Overview , 1996 .

[59]  A. Meade,et al.  Identifying careless responses in survey data. , 2012, Psychological methods.

[60]  Kenneth P. Carson,et al.  Performance Appraisal as Effective Management or Deadly Management Disease , 1991 .

[61]  A. E. M. van Vianen,et al.  A Tailored Policy‐Capturing Study on PO Fit Perceptions: The Ascendancy of Attractive Over Aversive Fit , 2013 .

[62]  L. Sulsky,et al.  Cultural Differences in the Motivation to Seek Performance Feedback: A Comparative Policy-Capturing Study , 2013 .

[63]  Nancy D. Albers-Miller.  Consumer misbehavior: why people buy illicit goods , 1999 .

[64]  R. Dalal,et al.  Within-Person Variability in Job Performance , 2014, The Handbook of Personality Dynamics and Processes.

[65]  Alexander C. LoPilato,et al.  A review of the content, criterion-related, and construct-related validity of assessment center exercises. , 2015, The Journal of applied psychology.

[66]  Evan Mayo-Wilson,et al.  Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. , 2018, The American psychologist.

[67]  P. P. Schoderbek,et al.  Pay-Allocations by Managers: A Policy-Capturing Approach , 1993 .

[68]  Kathryn Stafford,et al.  Beauty Revisited: The Impact of Attractiveness, Ability, and Personality in the Assessment of Employment Suitability , 2009 .

[69]  James M. LeBreton,et al.  Answers to 20 Questions About Interrater Reliability and Interrater Agreement , 2008 .

[70]  K. Behnke,et al.  The Brief Psychiatric Rating Scale. Dimension of schizophrenia--reliability and construct validity. , 1989, Psychopathology.

[71]  F. Lievens,et al.  Social Influences in Recruitment: When is Word‐of‐Mouth Most Effective? , 2016 .

[72]  S. Bonaccio,et al.  What types of advice do decision-makers prefer? , 2010 .

[73]  J. Browne,et al.  Development of a direct weighting procedure for quality of life domains , 1997, Quality of Life Research.

[74]  Dean A. Shepherd,et al.  A New Venture's Cognitive Legitimacy: An Assessment by Customers , 2003 .

[75]  Bi-Fen Hsu,et al.  When P-J fit and P-O fit meet guanxi in a Chinese selection context , 2013 .

[76]  C. Thompson,et al.  Initiating artificial nutrition support: a clinical judgement analysis. , 2012, Journal of human nutrition and dietetics : the official journal of the British Dietetic Association.

[77]  M. Hagger,et al.  Ego depletion and the strength model of self-control: a meta-analysis. , 2010, Psychological bulletin.

[78]  D. Oberoi,et al.  Do Men Know Which Lower Bowel Symptoms Warrant Medical Attention? A Web-Based Video Vignette Survey of Men in Western Australia , 2016, American journal of men's health.

[79]  P. Slovic,et al.  An analysis-of-variance model for the assessment of configural cue utilization in clinical judgment. , 1968, Psychological bulletin.

[80]  M. Rotundo.  The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance: a policy-capturing approach. , 2002, The Journal of applied psychology.

[81]  C. O'boyle,et al.  Assessing the quality of life of the individual: the SEIQoL with a healthy and a gastroenterology unit population , 1991, Psychological Medicine.

[82]  Nathan T. Carter,et al.  Placing Job Characteristics in Context: Cross-Temporal Meta-Analysis of Changes in Job Characteristics Since 1975 , 2018 .

[83]  Theresa M. Beckie,et al.  Physicians’ Tacit and Stated Policies for Determining Patient Benefit and Referral to Cardiac Rehabilitation , 2014, Medical decision making : an international journal of the Society for Medical Decision Making.

[84]  R. Dalal,et al.  The Effects of Performance Extremities on Ratings of Dynamic Performance , 2011 .

[85]  Ingram Olkin,et al.  Estimation of a Single Effect Size: Parametric and Nonparametric Methods , 1985 .

[86]  Donald P. Schwab,et al.  The role of pay and market pay variability in job application decisions. , 1983 .

[87]  K. McGraw,et al.  A common language effect size statistic. , 1992 .

[88]  Herman Aguinis,et al.  Best Practice Recommendations for Designing and Implementing Experimental Vignette Methodology Studies , 2014 .

[89]  Natàlia Cugueró-Escofet,et al.  Righting the Wrong for Third Parties: How Monetary Compensation, Procedure Changes and Apologies Can Restore Justice for Observers of Injustice , 2014 .

[90]  Meni Koslowsky,et al.  On the efficacy of credibility intervals as indicators of moderator effects in meta-analytic research , 1993 .

[91]  Sylvia G. Roch,et al.  Do all raters value task, citizenship, and counterproductive behaviors equally: An investigation of cultural values and performance evaluations , 2017 .

[92]  Ethel Brundin,et al.  Entrepreneurs’ decision making under different levels of uncertainty: the role of emotions , 2013 .

[93]  C. Russell,et al.  A closer look at decisions to quit , 2012 .

[94]  J. M. Cortina,et al.  What Is Coefficient Alpha? An Examination of Theory and Applications , 1993 .

[95]  Ronald J. Ebert,et al.  Bootstrapping the security analyst. , 1978 .

[96]  Chockalingam Viswesvaran,et al.  A Meta-Analytic Method for Testing Hypotheses about Clusters of Decision Makers , 1994 .

[97]  Felix D. Schönbrodt,et al.  At what sample size do correlations stabilize , 2013 .

[98]  James Shanteau,et al.  Why study expert decision making? Some historical perspectives and comments. , 1992 .

[99]  Curtis L. Wesley II.  Assessing Mission and Resources for Social Change: An Organizational Identity Perspective on Social Venture Capitalists' Decision Criteria , 2010 .

[100]  Gary L. Bradshaw,et al.  Forecasting solar flares: Experts and artificial systems , 1992 .

[101]  T. R. Brown.  A comparison of judgmental policy equations obtained from human judges under natural and contrived conditions , 1972 .

[102]  Timothy T. Baldwin,et al.  Reactions to Different Types of Forced Distribution Performance Evaluation Systems , 2009 .

[103]  A. Berchtold.  Test–retest: Agreement or reliability? , 2016 .

[104]  M. Inzlicht,et al.  A Multilab Preregistered Replication of the Ego-Depletion Effect , 2016, Perspectives on psychological science : a journal of the Association for Psychological Science.

[105]  Kelly D. Stamp,et al.  Understanding how nurse practitioners estimate patients' risk for coronary heart disease: a judgment analysis. , 2007, Journal of advanced nursing.

[106]  Charles Y. Murnieks,et al.  Passion for entrepreneurship or passion for the product? A conjoint analysis of angel and VC decision-making , 2018 .

[107]  J Braspenning,et al.  General practitioners' decision making for mental health problems: outcomes and ecological validity. , 1994, Journal of clinical epidemiology.

[108]  C. Harries,et al.  Managing the risk of suicide in acute psychiatric inpatients: A clinical judgement analysis of staff predictions of imminent suicide risk , 2008 .

[109]  Tracy McDonough.  A Policy Capturing Investigation of Battered Women’s Decisions to Stay in Violent Relationships , 2010, Violence and Victims.

[110]  Tomoki Sekiguchi,et al.  The use of person–organization fit and person–job fit information in making selection decisions , 2011 .

[111]  Alison Foster,et al.  The effect of improving task representativeness on capturing nurses’ risk assessment judgements: a comparison of written case simulations and physical simulations , 2013, BMC Medical Informatics and Decision Making.

[112]  Tiago Domingos,et al.  Testing for the survey mode effect on contingent valuation data quality: A case study of web based versus in-person interviews , 2007 .

[113]  Seth A. Kaplan,et al.  Is Policy Capturing Really More Resistant Than Traditional Self-Report Techniques to Socially Desirable Responding? , 2016 .

[114]  D. Kahneman,et al.  Duration neglect in retrospective evaluations of affective episodes. , 1993, Journal of personality and social psychology.

[115]  R. Baer,et al.  Detection of random responding on the MMPI-A. , 1997, Journal of personality assessment.

[116]  R. Hogarth,et al.  Determinants of Linear Judgment: A Meta-Analysis of Lens Model Studies , 2007, Psychological bulletin.

[117]  H. Garb.  Call for a Moratorium on the Use of the Rorschach Inkblot Test in Clinical and Forensic Settings , 1999, Assessment.

[118]  Dean A. Shepherd,et al.  Venture Capitalists' Assessment of New Venture Survival , 1999 .

[119]  A. Friedman.  Framing pictures: the role of knowledge in automatized encoding and memory for gist. , 1979, Journal of experimental psychology. General.

[120]  Hannes Zacher,et al.  Job performance ratings: The relative importance of mental ability, conscientiousness, and career adaptability , 2015 .

[121]  Robert H. Ashton,et al.  A review and analysis of research on the test–retest reliability of professional judgment , 2000 .

[122]  E. Zajac,et al.  Alliance or Acquisition? A Mechanisms‐Based, Policy‐Capturing Analysis , 2017 .

[123]  Per L. Bylund,et al.  The Influence of Tax and Regulatory Policies on Entrepreneurs’ Opportunity Evaluation Decisions , 2016 .

[124]  J. Mestre,et al.  The relation between problem categorization and problem solving among experts and novices , 1989, Memory & cognition.

[125]  Dean A. Shepherd,et al.  The fallacy of "only the strong survive": The effects of extrinsic motivation on the persistence decisions for under-performing firms , 2008 .

[126]  A. Paivio.  Mental imagery in associative learning and memory , 1969 .

[127]  Tony W. Tong,et al.  Executives' Assessments of International Joint Ventures in China: A Multi-Theoretical Investigation , 2012, Management and Organization Review.

[128]  L. Tay,et al.  Cultural and Intellectual Openness Differentially Relate to Social Judgments of Potential Work Partners. , 2017, Journal of personality.

[129]  Paul M. Muchinsky,et al.  Intrasubject predictions of vocational preference: Convergent validation via the decision theoretic paradigm , 1985 .

[130]  C. O'boyle.  The Schedule for the Evaluation of Individual Quality of Life (SEIQoL): The Concept of Quality of Life in Clinical Research , 1994 .