Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance, Novelty, and Resource Allocation in Science

Selecting among alternative projects is a core management task in all innovating organizations. In this paper, we focus on the evaluation of frontier scientific research projects. We argue that the “intellectual distance” between the knowledge embodied in research proposals and an evaluator’s own expertise systematically relates to the evaluations given. To estimate relationships, we designed and executed a grant proposal process at a leading research university in which we randomized the assignment of evaluators and proposals to generate 2,130 evaluator–proposal pairs. We find that evaluators systematically give lower scores to research proposals that are closer to their own areas of expertise and to those that are highly novel. The patterns are consistent with biases associated with boundedly rational evaluation of new ideas. The patterns are inconsistent with intellectual distance simply contributing “noise” or being associated with private interests of evaluators. We discuss implications for policy, managerial intervention, and allocation of resources in the ongoing accumulation of scientific knowledge.

[1]  F. Knight The economic nature of the firm: From Risk, Uncertainty, and Profit , 2009 .

[2]  J. Ben-David,et al.  Roles and Innovations in Medicine , 1960, American Journal of Sociology.

[3]  R. Nelson Uncertainty, Learning, and the Economics of Parallel Research and Development Efforts , 1961 .

[4]  R. Merton The Matthew Effect in Science , 1968, Science.

[5]  N. Mullins The development of a scientific specialty: The phage group and the origins of molecular biology , 1972 .

[6]  John Law,et al.  The Development of Specialties in Science: the Case of X-ray Protein Crystallography , 1973 .

[7]  H. Simon,et al.  Perception in chess , 1973 .

[8]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[9]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[10]  S. B. Levenberg Professional training, psychodiagnostic skill, and kinetic family drawings. , 1975, Journal of personality assessment.

[11]  James M. Utterback,et al.  A dynamic model of process and product innovation , 1975 .

[12]  Mary Warnock,et al.  Schools of thought , 1977 .

[13]  Baruch Fischhoff,et al.  Calibration of Probabilities: The State of the Art , 1977 .

[14]  John B. Kidd,et al.  Decision Making and Change in Human Affairs , 1979 .

[15]  Dorothea P. Simon,et al.  Expert and Novice Performance in Solving Physics Problems , 1980, Science.

[16]  B. Brehmer In one word: Not from experience. , 1980 .

[17]  Paul J. Feltovich,et al.  Categorization and Representation of Physics Problems by Experts and Novices , 1981, Cogn. Sci..

[18]  J. R. Cole,et al.  Chance and consensus in peer review. , 1981, Science.

[19]  Albert N. Link,et al.  The Simple Economics of Basic Scientific Research: A Test of Nelson's Diversification Hypothesis , 1981 .

[20]  Eric J. Johnson,et al.  Product familiarity and learning new information , 1984 .

[21]  P. E. Johnson,et al.  Multimethod study of clinical judgment. , 1982, Organizational behavior and human performance.

[22]  S. Winter,et al.  An evolutionary theory of economic change , 1983 .

[23]  G. Dosi Technological Paradigms and Technological Trajectories: A Suggested Interpretation of the Determinants and Directions of Technical Change , 1982 .

[24]  B. Marshall,et al.  UNIDENTIFIED CURVED BACILLI ON GASTRIC EPITHELIUM IN ACTIVE CHRONIC GASTRITIS , 1983, The Lancet.

[25]  J. Hicks,et al.  The economics of science , 1996 .

[26]  Gerardo R. Ungson,et al.  Decision Making: An Interdisciplinary Inquiry. , 1984 .

[27]  John W. Payne,et al.  Effort and Accuracy in Choice , 1985 .

[28]  Robert J. Meyer,et al.  The Learning of Multiattribute Judgment Policies , 1987 .

[29]  John H. Holland,et al.  Induction: Processes of Inference, Learning, and Discovery , 1987, IEEE Expert.

[30]  M. Chi,et al.  The Nature of Expertise , 1988 .

[31]  Eric J. Johnson,et al.  Expertise and decision under uncertainty: Performance and process. , 1988 .

[32]  J. Berger Statistical Decision Theory and Bayesian Analysis , 1988 .

[33]  Jean Bédard,et al.  Expertise in auditing: Myth or reality? , 1989 .

[34]  P. Romer Endogenous Technological Change , 1989, Journal of Political Economy.

[35]  Paul Thagard,et al.  Induction: Processes Of Inference , 1989 .

[36]  D. Chubin,et al.  Peerless Science: Peer Review and U. S. Science Policy , 1990 .

[37]  R. Fletcher,et al.  The effects of blinding on the quality of peer review. A randomized trial. , 1990, JAMA.

[38]  M. Dogan,et al.  Creative Marginality: Innovation at the Intersections of Social Sciences. , 1992 .

[39]  K. A. Ericsson,et al.  Toward a general theory of expertise : prospects and limits , 1991 .

[40]  Harold Maurice Collins,et al.  New Light on Old Boys: Cognitive and Institutional Particularism in the Peer Review System , 1991 .

[41]  Norman Hackerman,et al.  Peerless Science: Peer Review and U.S. Science Policy , 1992 .

[42]  J. March Exploration and exploitation in organizational learning , 1991, STUDI ORGANIZZATIVI.

[43]  Colin Camerer,et al.  The process-performance paradox in expert judgment - How can experts know so much and predict so badly? , 1991 .

[44]  Devendra Sahal,et al.  Technological guideposts and innovation avenues , 1993 .

[45]  G. Dosi Technological Paradigms and Technological Trajectories , 1993 .

[46]  P. Anand,et al.  Foundations of Rational Choice Under Risk. , 1993 .

[47]  P. David,et al.  Toward a new economics of science , 1994 .

[48]  S. Schwartzman,et al.  The New Production of Knowledge: The Dynamics of Science and Research in Contemporary Societies , 1994 .

[49]  Craig R. Fox,et al.  Ambiguity Aversion and Comparative Ignorance , 1995 .

[50]  K. A. Ericsson,et al.  The Road To Excellence: The Acquisition of Expert Performance in the Arts and Sciences, Sports, and Games , 1996 .

[51]  Greg A. Stevens,et al.  3,000 Raw Ideas = 1 Commercial Success! , 1997 .

[52]  Daniel A. Levinthal Adaptation on rugged landscapes , 1997 .

[53]  A. Elzinga The New Production of Knowledge. The Dynamics of Science and Research in Contemporary Societies , 1997 .

[54]  F. Godlee,et al.  Effect of open peer review on quality of reviews and on reviewers'recommendations: a randomised trial , 1999, BMJ.

[55]  Paul A. David,et al.  The explicit economics of knowledge codification and tacitness , 2000 .

[56]  D. Simonton Origins of genius : Darwinian perspectives on creativity , 1999 .

[57]  P. Rothwell,et al.  Reproducibility of peer review in clinical neuroscience. Is agreement between reviewers any greater than would be expected by chance alone? , 2000, Brain : a journal of neurology.

[58]  Howard L. Bleich,et al.  Technical Milestone: Medical Subject Headings Used to Search the Biomedical Literature , 2001, J. Am. Medical Informatics Assoc..

[59]  O. Sorenson,et al.  Science as a Map in Technological Search , 2000 .

[60]  Liv Langfeldt,et al.  The Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome , 2001, Peer review in an Era of Evaluation.

[61]  Nigel W. Bond,et al.  A multilevel cross‐classified modelling approach to peer review of grant proposals: the effects of assessor and researcher attributes on assessor ratings , 2003 .

[62]  Benjamin F. Jones The Burden of Knowledge and the 'Death of the Renaissance Man': Is Innovation Getting Harder? , 2004 .

[63]  Benjamin F. Jones Age and Great Invention , 2004 .

[64]  M. Hakel,et al.  An Examination of Sources of Peer-Review Bias , 2006, Psychological science.

[65]  M. Chi Two Approaches to the Study of Experts' Characteristics , 2006 .

[66]  Scott Stern,et al.  Climbing Atop the Shoulders of Giants: The Impact of Institutions on Cumulative Research , 2006 .

[67]  Paul J. Feltovich,et al.  The Cambridge handbook of expertise and expert performance , 2006 .

[68]  Liv Langfeldt,et al.  The policy challenges of peer review: managing bias, conflict of interests and interdisciplinary assessments , 2006 .

[69]  Samir Elhedhli,et al.  The Effectiveness of Simple Decision Heuristics: Forecasting Commercial Success for Early-Stage Ventures , 2006, Manag. Sci..

[70]  Daniel A. Levinthal,et al.  Exploration and Exploitation in Organizational Learning , 2007 .

[71]  Benjamin F. Jones,et al.  Supporting Online Material Materials and Methods Figs. S1 to S3 References the Increasing Dominance of Teams in Production of Knowledge , 2022 .

[72]  Juan Miguel Campanario,et al.  Rejecting highly cited papers: The views of scientists who encounter resistance to their discoveries from other scientists , 2007, J. Assoc. Inf. Sci. Technol..

[73]  Nigel W. Bond,et al.  Peer review process: Assessments by applicant-nominated referees are biased, inflated, unreliable and invalid , 2007 .

[74]  Fiona Murray,et al.  Exploring the Foundations of Cumulative Innovation: Implications for Organization Science , 2007, Organ. Sci..

[75]  Benjamin L. Hallen The Causes and Consequences of the Initial Network Positions of New Organizations: From Whom Do Entrepreneurs Receive Investments? , 2008 .

[76]  Ulf Sandström,et al.  Persistent nepotism in peer-review , 2008, Scientometrics.

[77]  John R. Anderson,et al.  The acquisition of robust and flexible cognitive skills. , 2008, Journal of experimental psychology. General.

[78]  L. Bornmann,et al.  The effectiveness of the peer review process: inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. , 2008, Angewandte Chemie.

[79]  H. Marsh,et al.  Improving the Peer-review Process for Grant Applications , 2022 .

[80]  Michèle Lamont,et al.  How Professors Think: Inside the Curious World of Academic Judgment , 2009 .

[81]  F. Gobet,et al.  The Cambridge handbook of expertise and expert performance , 2006 .

[82]  M. Weitzman,et al.  Recombinant Growth , 2009 .

[83]  Heidi L. Williams,et al.  Intellectual Property Rights and Innovation: Evidence from the Human Genome , 2013, Journal of Political Economy.

[84]  D. Meyer,et al.  Supporting Online Material Materials and Methods Som Text Figs. S1 to S6 References Evidence for a Collective Intelligence Factor in the Performance of Human Groups , 2022 .

[85]  William M. Tierney,et al.  Editorial Peer Reviewers' Recommendations at a General Medical Journal: Are They Reliable and Do Editors Care? , 2010, PloS one.

[86]  Gustavo Manso Motivating Innovation , 2010 .

[87]  T. Steen,et al.  The double-edged sword , 2011 .

[88]  Nicholas Graves,et al.  Funding grant proposals for scientific research: retrospective analysis of scores by members of grant review panel , 2011, BMJ : British Medical Journal.

[89]  Liv Langfeldt,et al.  How Professors Think: Inside the Curious World of Academic Judgment , 2011 .

[90]  Kathlyn E. Fletcher,et al.  The Validity of Peer Review in a General Medicine Journal , 2011, PloS one.

[91]  Cassidy R. Sugimoto,et al.  Citation gamesmanship: testing for evidence of ego bias in peer review , 2012, Scientometrics.

[92]  Carole J. Lee A Kuhnian Critique of Psychometric Research on Peer Review , 2012 .

[93]  Yu Xie,et al.  Is American Science in Decline , 2012 .

[94]  Paula E. Stephan How Economics Shapes Science , 2012 .

[95]  T. Kuhn,et al.  The Structure of Scientific Revolutions: 50th Anniversary Edition , 2012 .

[96]  Danielle Li Expertise vs . Bias in Evaluation : Evidence from the NIH ∗ , 2013 .

[97]  W. Myers,et al.  Atypical Combinations and Scientific Impact , 2013 .

[98]  Fatima M. Albar,et al.  Fast and frugal heuristics for new product screening – is managerial judgment ‘good enough?’ , 2013 .

[99]  Keyvan Vakili,et al.  The double-edged sword of recombination in breakthrough innovation , 2013 .

[100]  Cassidy R. Sugimoto,et al.  Bias in peer review , 2013, J. Assoc. Inf. Sci. Technol..

[101]  Linus Dahlander,et al.  Distant Search, Narrow Attention: How Crowding Alters Organizations’ Filtering of Suggestions in Crowdsourcing , 2014 .

[102]  K. Boudreau,et al.  'Open' Disclosure of Innovations, Incentives and Follow-on Reuse: Theory on Processes of Cumulative Innovation and a Field Experiment in Computational Biology , 2015 .

[103]  Sarah Kaplan,et al.  The double-edged sword of recombination in breakthrough innovation: The Double-Edged Sword of Recombination , 2015 .

[104]  Mooweon Rhee,et al.  Exploration and Exploitation , 2016 .

[105]  Ethan Mollick,et al.  Wisdom or Madness? Comparing Crowds with Expert Evaluation in Funding the Arts , 2015, Manag. Sci..

[106]  Danielle Li Expertise versus Bias in Evaluation: Evidence from the NIH , 2017 .