Determinants of Linear Judgment: A Meta-Analysis of Lens Model Studies

The mathematical representation of E. Brunswik's (1952) lens model has been used extensively to study human judgment and provides a unique opportunity to conduct a meta-analysis of studies that covers roughly 5 decades. Specifically, the authors analyzed statistics of the "lens model equation" (L. R. Tucker, 1964) associated with 249 different task environments obtained from 86 articles. On average, fairly high levels of judgmental achievement were found, and people were seen to be capable of achieving similar levels of cognitive performance in noisy and predictable environments. Further, the effects of task characteristics that influence judgment (numbers and types of cues, inter-cue redundancy, function forms and cue weights in the ecology, laboratory versus field studies, and experience with the task) were identified and estimated. A detailed analysis of learning studies revealed that the most effective form of feedback was information about the task. The authors also analyzed empirically under what conditions the application of bootstrapping--or replacing judges by their linear models--is advantageous. Finally, the authors note shortcomings of the kinds of studies conducted to date, limitations in the lens model methodology, and possibilities for future research.

[1]  Gueorgui I. Kolev,et al.  Illusory Correlation in the Remuneration of Chief Executive Officers: It Pays to Play Golf, and Well , 2010 .

[2]  M. Pittler Systematic Reviews in Health Care: Meta‐analysis in Context , 2010 .

[3]  Y. Ganzach Coherence and correspondence in the psychological analysis of numerical predictions: How error-prone heuristics are replaced by ecologically valid heuristics , 2009, Judgment and Decision Making.

[4]  J. Sterne,et al.  Meta‒Analysis in Stata™ , 2008 .

[5]  Hau-San Wong,et al.  Human Computer Interaction , 2008, Encyclopedia of Multimedia.

[6]  P. Freebody,et al.  Assessment as Judgment-in-Context: Analysing how teachers evaluate students' writing 1 , 2007 .

[7]  R. Hogarth,et al.  Heuristic and linear models of judgment: matching rules and environments. , 2007, Psychological review.

[8]  Anna Cuxart,et al.  What Risks Do People Perceive in Everyday Life? A Perspective Gained from the Experience Sampling Method (ESM) , 2007, Risk analysis : an official publication of the Society for Risk Analysis.

[9]  M. Mehl,et al.  The lay assessment of subclinical depression in daily life. , 2006, Psychological assessment.

[10]  Natalia Karelaia,et al.  On Heuristic and Linear Models of Judgment: Mapping the Demand for Knowledge , 2006 .

[11]  S. Gosling,et al.  Personality in its natural habitat: manifestations and implicit folk theories of personality in daily life. , 2006, Journal of personality and social psychology.

[12]  Klaus Fiedler,et al.  Information Sampling and Adaptive Cognition , 2005 .

[13]  R. Youmans,et al.  To thy own self be true: finding the utility of cognitive information feedback , 2005 .

[14]  Robin M. Hogarth,et al.  Simple Models for Multiattribute Choice with Many Alternatives: When It Does and Does Not Pay to Face Trade-offs with Binary Attributes , 2005, Manag. Sci..

[15]  Kai H. Lim,et al.  Improving judgmental forecasts with judgmental bootstrapping and task feedback support , 2005 .

[16]  Reid Hastie,et al.  The robust beauty of majority rules in group decisions. , 2005, Psychological review.

[17]  Robin M. Hogarth,et al.  Information Sampling and Adaptive Cognition: Is Confidence in Decisions Related to Feedback? Evidence from Random Samples of Real-World Behavior , 2005 .

[18]  Mandeep K. Dhami,et al.  The role of representative design in an ecological approach to cognition. , 2004, Psychological bulletin.

[19]  E. Mullet,et al.  Functional learning among children, adolescents, and young adults. , 2004, Journal of experimental child psychology.

[20]  Nigel Harvey,et al.  Effects of judges' forecasting on their later combination of forecasts for the same outcomes , 2004 .

[21]  Human judgments in New York state sales and use tax forecasting , 2004 .

[22]  S. Gosling,et al.  e-Perceptions : Personality Impressions Based on Personal Websites , 2004 .

[23]  William Remus,et al.  Does Mood Influence Judgment Accuracy , 2004 .

[24]  Ann M. Bisantz,et al.  Measuring the Fit between Human Judgments and Automated Alerting Algorithms: A Study of Collision Detection , 2003, Hum. Factors.

[25]  Alex Kirlik,et al.  A Systems Perspective on Situation Awareness II: Experimental Evaluation of a Modeling and Measurement Technique , 2003 .

[26]  K. Gilhooly,et al.  Factors influencing prescribing decisions in the treatment of depression: a social judgement theory approach , 2003 .

[27]  Alex Kirlik,et al.  Inferring rule-based strategies in dynamic judgment tasks: toward a noncompensatory formulation of the lens model , 2003, IEEE Trans. Syst. Man Cybern. Part A.

[28]  S. Thompson,et al.  Quantifying heterogeneity in a meta‐analysis , 2002, Statistics in medicine.

[29]  S. Gosling,et al.  A room with a cue: personality judgments based on offices and bedrooms. , 2002, Journal of personality and social psychology.

[30]  U. Hoffrage,et al.  Fast, frugal, and fit: Simple heuristics for paired comparison , 2002 .

[31]  Douglas G. Altman,et al.  Systematic Reviews in Health Care: Meta-Analysis in Context: Second Edition , 2008 .

[32]  Michael D. Shields,et al.  Why Does Fixation Persist? Experimental Evidence on the Judgment Performance Effects of Expensing Intangibles , 2001 .

[33]  A. Field Meta-analysis of correlation coefficients: a Monte Carlo comparison of fixed- and random-effects methods. , 2001, Psychological methods.

[34]  T. R. Stewart,et al.  The essential brunswik: Beginnings, explications, applications. , 2001 .

[35]  R. Cooksey,et al.  Judgment of Factors Influencing Interest: An Australian Study , 2001 .

[36]  R. Rosenthal,et al.  Meta-analysis: recent developments in quantitative methods for literature reviews. , 2001, Annual review of psychology.

[37]  Arthur D. Fisk,et al.  Modeling and analysis of a dynamic judgment task using a lens model approach , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[38]  Kenneth R. Hammond,et al.  Human Judgment and Social Policy: Irreducible Uncertainty, Inevitable Error, Unavoidable Injustice , 2000 .

[39]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[40]  S Duval,et al.  Trim and Fill: A Simple Funnel‐Plot–Based Method of Testing and Adjusting for Publication Bias in Meta‐Analysis , 2000, Biometrics.

[41]  Robert P. Mahan,et al.  Cognitive adaptation and its consequences: a test of cognitive continuum theory , 2000 .

[42]  Colin Camerer,et al.  The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework , 1999 .

[43]  E. Mullet,et al.  How well do elderly people cope with uncertainty in a learning task? , 1999, Acta psychologica.

[44]  Research methodology in the life, behavioural and social sciences , 1999 .

[45]  Ulrich Hoffrage,et al.  Why does one-reason decision making work? A case study in ecological rationality , 1999 .

[46]  P. Todd,et al.  Simple Heuristics That Make Us Smart , 1999 .

[47]  R. Tweney,et al.  The Role of Data and Feedback Error in Inference and Prediction , 1998 .

[48]  Jack L. Vevea,et al.  Fixed- and random-effects models in meta-analysis. , 1998 .

[49]  T. R. Stewart,et al.  Aging and multiple cue probability learning: the case of inverse relationships. , 1997, Acta psychologica.

[50]  G. Smith,et al.  Bias in meta-analysis detected by a simple, graphical test , 1997, BMJ.

[51]  R. Hogarth,et al.  Research on Judgment and Decision Making: Currents, Connections, and Controversies , 1997 .

[52]  T. R. Stewart,et al.  The Importance of the Task in Analyzing Expert Judgment , 1997 .

[53]  W. Remus,et al.  Does feedback improve the accuracy of recurrent judgmental forecasts? , 1996, Proceedings of the Thirtieth Hawaii International Conference on System Sciences.

[54]  D Kahneman,et al.  On the reality of cognitive illusions. , 1996, Psychological review.

[55]  G. Gigerenzer On Narrow Norms and Vague Heuristics: A Reply to Kahneman and Tversky (1996) , 1996 .

[56]  Ray W. Cooksey,et al.  Judgment analysis : theory, methods, and applications , 1996 .

[57]  G Gigerenzer,et al.  Reasoning the fast and frugal way: models of bounded rationality. , 1996, Psychological review.

[58]  K. Law The Use of Fisher’s Z in Schmidt-Hunter–type Meta-Analyses , 1995 .

[59]  Efrat Neter,et al.  The effect of motivation of judgment depends on the difficulty of the judgment. , 1995 .

[60]  C. Begg,et al.  Operating characteristics of a rank correlation test for publication bias. , 1994, Biometrics.

[61]  B. Brehmer The psychology of linear judgement models , 1994 .

[62]  A. Laduca,et al.  Validation of Professional Licensure Examinations , 1994, Evaluation & the health professions.

[63]  Robert Gifford,et al.  A lens-mapping framework for understanding the encoding and decoding of interpersonal dispositions in nonverbal behavior. , 1994 .

[64]  Eric J. Johnson,et al.  The adaptive decision maker , 1993 .

[65]  N. Sanders,et al.  Journal of behavioral decision making: "The need for contextual and technical knowledge in judgmental forecasting", 5 (1992) 39-52 , 1992 .

[66]  Michael E. Doherty,et al.  The assessment of self-insight in judgment policies , 1992 .

[67]  L. Hammer Advances in behavioral pediatrics , 1992 .

[68]  J. Shanteau Competence in experts: The role of task characteristics , 1992 .

[69]  Leslie B. Hammer,et al.  Task information, cognitive information, or functional validity information: Which components of cognitive feedback affect performance?☆ , 1992 .

[70]  J. Shanteau How much information does an expert use? Is it relevant? , 1992 .

[71]  T G Tape,et al.  Comparing Methods of Learning Clinical Prediction from Case Simulations , 1992, Medical decision making : an international journal of the Society for Medical Decision Making.

[72]  N. Castellan,et al.  Relations between linear models: Implications for the lens model , 1992 .

[73]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[74]  J. Frank Yates,et al.  How Quantity Judgment Changes as the Number of Cues Increases: An Analytical Framework and Review , 1992 .

[75]  J P Ornato,et al.  Use of Clinical Judgment Analysis to Explain Regional Variations in Physicians' Accuracies in Diagnosing Pneumonia , 1991, Medical decision making : an international journal of the Society for Medical Decision Making.

[76]  Colin Camerer,et al.  The process-performance paradox in expert judgment - How can experts know so much and predict so badly? , 1991 .

[77]  P. Werner,et al.  Predicting Violence in Newly Admitted Inmates , 1990 .

[78]  Peter Freebody,et al.  The Ecology of Spelling: A Lens Model Analysis of Spelling Errors and Student Judgments of Spelling Difficulty. , 1990 .

[79]  R S Wigton,et al.  Teaching old dogs new tricks: using cognitive feedback to improve physicians' diagnostic judgments on simulated cases , 1990, Academic medicine : journal of the Association of American Medical Colleges.

[80]  B Kleinmuntz,et al.  Why we still use our heads instead of formulas: toward an integrative approach. , 1990, Psychological bulletin.

[81]  M. Doherty,et al.  Effects of cognitive feedback on performance. , 1989 .

[82]  J. Yesavage,et al.  Social workers' decision making about the violent client , 1989 .

[83]  Keith Levi,et al.  Expert systems should be more accurate than human experts: evaluation procedures from human judgement and decision making , 1989, IEEE Trans. Syst. Man Cybern..

[84]  R. Dawes,et al.  Heuristics and Biases: Clinical versus Actuarial Judgment , 2002 .

[85]  Thomas R. Stewart,et al.  Analysis of Expert Judgment in a Hail Forecasting Experiment , 1989 .

[86]  Michael J. Strube,et al.  Averaging correlation coefficients: Influence of heterogeneity and set size. , 1988 .

[87]  J. Engel,et al.  An Exploratory Study of Physicians' Clinical Judgment , 1988 .

[88]  B. Brehmer,et al.  Human judgment : the SJT view , 1988 .

[89]  D. A. Kenny,et al.  Accuracy in interpersonal perception: a social relations analysis. , 1987, Psychological bulletin.

[90]  Kenneth M. York,et al.  The influence of cue unreliability on judgment in a multiple cue probability learning task , 1987 .

[91]  Paul Slovic,et al.  Graphic Representation of Judgmental Information , 1986, SGCH.

[92]  Michael Firth,et al.  Assessing the accuracy of financial analyst security return predictions , 1987 .

[93]  Berndt Brehmer,et al.  Use of experts in complex decision making: A paradigm for the study of staff work , 1986 .

[94]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.

[95]  Peter Freebody,et al.  Teachers’ Predictions of Children’s Early Reading Achievement: An Application of Social Judgment Theory , 1986 .

[96]  Ronald J. Ebert,et al.  Policy capturing in the employment interview. , 1986 .

[97]  H. Rothstein The effects of time pressure on judgment in multiple cue probability learning , 1986 .

[98]  Mary W. Downton,et al.  Judgments of photographs vs. field observations in studies of perception and judgment of the visual environment , 1984 .

[99]  J. Yesavage,et al.  Reliability, accuracy, and decision-making strategy in clinical predictions of imminent dangerousness. , 1983, Journal of consulting and clinical psychology.

[100]  A. H. Ashton,et al.  An Empirical-Study Of Budget-Related Predictions Of Corporate-Executives , 1982 .

[101]  L. Cohen Can human irrationality be experimentally demonstrated? , 1981, Behavioral and Brain Sciences.

[102]  Colin F. Camerer,et al.  General conditions for the success of bootstrapping models , 1981 .

[103]  L. Adelman The influence of formal, substantive, and contextual task properties on the relative effectiveness of different forms of feedback in multiple-cue probability learning tasks☆ , 1981 .

[104]  Benjamin Kleinmuntz,et al.  Statistical versus clinical lie detection. , 1981 .

[105]  Paul Slovic,et al.  Multidimensional functional learning (MFL) and some new conceptions of feedback , 1981 .

[106]  R. Ashton A Descriptive Study Of Information Evaluation , 1981 .

[107]  R. Ashton,et al.  Feedback And Prediction Achievement In Financial Analysis , 1981 .

[108]  N. Anderson Foundations of information integration theory , 1981 .

[109]  S. Brodsky Criminal Justice and Behavior , 1980 .

[110]  B. Brehmer In one word: Not from experience. , 1980 .

[111]  P. Koele The influence of labeled stimuli on nonlinear multiple-cue probability learning , 1980 .

[112]  Berndt Brehmer,et al.  Content and consistency in probabilistic inference tasks , 1980 .

[113]  B. Fischhoff,et al.  Journal of Experimental Psychology: Human Learning and Memory , 1980 .

[114]  H. J. Einhorn,et al.  Linear regression and process-tracing models of judgment. , 1979 .

[115]  William F. Wright Properties of judgment models in a financial setting , 1979 .

[116]  William H. Clover,et al.  Can we learn anything about interviewing real people from “interviews” of paper people? Two studies of the external validity of a paradigm☆ , 1978 .

[117]  Terry Connolly,et al.  Some Effects of Feedback Error BV Diagnostic Decision Tasks , 1978 .

[118]  Ronald J. Ebert,et al.  Bootstrapping the security analyst. , 1978 .

[119]  Neal Schmitt,et al.  Types of task information feedback in multiple-cue probability learning. , 1977 .

[120]  K. Rudestam,et al.  Effects of Amounts and Units of Information on the Judgmental Process , 1976 .

[121]  M. Lindell Cognitive and Outcome Feedback in Multiple-Cue Probability Learning Tasks. , 1976 .

[122]  B. Brehmer Note on clinical judgment and the formal characteristics of clinical tasks. , 1976 .

[123]  Michael E. Doherty,et al.  Judgment theory applied to the selection of life insurance salesmen , 1976 .

[124]  Larry King,et al.  Feedback and task predictability as determinants of performance in multiple cue probability learning tasks , 1976 .

[125]  Michael E. Doherty,et al.  Feedback effects in a metric multiple-cue probability learning task. , 1976 .

[126]  Robert Libby,et al.  Man versus model of man: some conflicting evidence , 1976 .

[127]  L. R. Goldberg Man versus model of man: just how conflicting is that evidence? , 1976 .

[128]  Derick O. Steinmann The effects of cognitive feedback and task complexity in multiple-cue probability learning , 1976 .

[129]  J. W. Asher,et al.  Educational research and evaluation methods , 1976 .

[130]  J. Gillis,et al.  Interaction of Age and Cue Validities in Multiple-Cue Probability Learning by Children , 1975 .

[131]  P. Muchinsky,et al.  Human inference behavior in abstract and meaningful environments , 1975 .

[132]  R. Hogarth,et al.  Unit weighting schemes for decision making , 1975 .

[133]  R. C. Sherman,et al.  Effect of Lens-Model and Outcome Feedback in a Social Judgment Analogue , 1974 .

[134]  B. Brehmer The effect of cue intercorrelation on interpersonal learning of probabilistic inference tasks , 1974 .

[135]  B. Armelius,et al.  The use of redundancy in multiple-cue judgments: Data from a suppressor-variable task. , 1974 .

[136]  T. R. Stewart,et al.  The Effects of Redundancy in Multiple-Cue Probability Learning , 1974 .

[137]  Derick O. Steinmann Transfer of lens model training , 1974 .

[138]  R. Dawes,et al.  Linear models in decision making. , 1974 .

[139]  Daniel Kahneman,et al.  Availability: A heuristic for judging frequency and probability , 1973 .

[140]  Berndt Brehmer,et al.  Effects of task predictability and cue validity on interpersonal learning of inference tasks involving both linear and nonlinear relations , 1973 .

[141]  B. Brehmer Effects of Cue Validity on Interpersonal Learning of Inference Tasks with Linear and Nonlinear Cues , 1973 .

[142]  N. Castellan,et al.  Comments on the “lens model” equation and the analysis of multiple-cue judgment tasks , 1973 .

[143]  N. John CastellanJr.,et al.  Comments on the “lens model” equation and the analysis of multiple-cue judgment tasks , 1973 .

[144]  Kenneth R. Hammond,et al.  Negative effects of outcome-feedback in multiple-cue probability learning. , 1973 .

[145]  D. Magnusson,et al.  Cue relevance and feedback in a clinical prediction task , 1973 .

[146]  Derick O. Steinmann,et al.  A lens model analysis of a bookbag and poker chip experiment: A methodological note , 1972 .

[147]  A. Tversky,et al.  Subjective Probability: A Judgment of Representativeness , 1972 .

[148]  Hillel J. Einhorn,et al.  Expert measurement and mechanical combination , 1972 .

[149]  Donald H. Deane,et al.  Acquisition and application of knowledge in complex inference tasks. , 1972 .

[150]  P. Miller Do labels mislead? A multiple cue study, within the framework of brunswik's probabilistic functionalism , 1971 .

[151]  N. Wiggins,et al.  Man versus model of man revisited: The forecasting of graduate school success. , 1971 .

[152]  R. Dawes A case study of graduate admissions: Application of three principles of human decision making. , 1971 .

[153]  H. J. Einhorn Use of nonlinear, noncompensatory models as a function of task and amount of information , 1971 .

[154]  Lewis R. Goldberg,et al.  Man versus model of man: A rationale, plus some evidence, for a method of improving on clinical inferences. , 1970 .

[155]  Berndt Brehmer,et al.  Cognitive Dependence on Additive and Configural Cue-Criterion Relations , 1969 .

[156]  Rita C. Summers,et al.  Judgments Based on Different Functional Relationships between Interacting Cues and a Criterion , 1969 .

[157]  J. C. Naylor,et al.  The influence of cue redundancy upon the human inference process for tasks of varying degrees of predictability , 1968 .

[158]  C. N. Uhl Effects of multiple stimulus validity and criterion dispersion on learning of interval concepts. , 1966, Journal of experimental psychology.

[159]  J. C. Naylor,et al.  Characteristics of the human inference process in complex choice behavior situations , 1966 .

[160]  K. R. Hammond,et al.  A research paradigm for the study of interpersonal learning. , 1966, Psychological bulletin.

[161]  K. R. Hammond,et al.  COGNITIVE DEPENDENCE ON LINEAR AND NONLINEAR CUES. , 1965, Psychological review.

[162]  J. R. Newton JUDGMENT AND FEEDBACK IN A QUASI-CLINICAL SITUATION. , 1965, Journal of personality and social psychology.

[163]  L. Tucker A SUGGESTED ALTERNATIVE FORMULATION IN THE DEVELOPMENTS BY HURSCH, HAMMOND, AND HURSCH, AND BY HAMMOND, HURSCH, AND TODD. , 1964, Psychological review.

[164]  K. R. Hammond,et al.  ANALYZING THE COMPONENTS OF CLINICAL INFERENCE. , 1964, Psychological review.

[165]  K. R. Hammond,et al.  SOME METHODOLOGICAL CONSIDERATIONS IN MULTIPLE-CUE PROBABILITY STUDIES. , 1964, Psychological review.

[166]  L. Grebstein Relative accuracy of actuarial prediction, experienced clinicians, and graduate students in a clinical judgment task. , 1963, Journal of consulting psychology.

[167]  D. B. Yntema,et al.  Man-Computer Cooperation in Decisions Requiring Common Sense , 1961 .

[168]  P. Hoffman The paramorphic representation of clinical judgment. , 1960, Psychological bulletin.

[169]  K. R. Hammond Probabilistic functioning and the clinical method. , 1955, Psychological review.

[170]  E. Brunswik Representative design and probabilistic theory in a functional psychology. , 1955, Psychological review.

[171]  E. Brunswik,et al.  The Conceptual Framework of Psychology , 1954 .

[172]  E. Brunswik Organismic achievement and environmental probability. , 1943 .