Science and Technology Advance through Surprise

Breakthrough discoveries and inventions involve unexpected combinations of contents including problems, methods, and natural entities, and also diverse contexts such as journals, subfields, and conferences. Drawing on data from tens of millions of research papers, patents, and researchers, we construct models that predict next year's content and context combinations with an AUC of 95% based on embeddings constructed from high-dimensional stochastic block models, where the improbability of new combinations itself predicts up to 50% of the likelihood that they will gain outsized citations and major awards. Most of these breakthroughs occur when problems in one field are unexpectedly solved by researchers from a distant other. These findings demonstrate the critical role of surprise in advance, and enable evaluation of scientific institutions ranging from education and peer review to awards in supporting it.

[1]  Edoardo M. Airoldi,et al.  Mixed Membership Stochastic Blockmodels , 2007, NIPS.

[2]  Michael Szell,et al.  A Nobel opportunity for interdisciplinarity , 2018, Nature Physics.

[3]  L. Fleming.  Breakthroughs and the ¿Long Tail¿ of Innovation , 2007 .

[4]  Olga Kononova,et al.  Unsupervised word embeddings capture latent knowledge from materials science literature , 2019, Nature.

[5]  S. Haustein,et al.  Long-Distance Interdisciplinarity Leads to Higher Scientific Impact , 2015, PloS one.

[6]  T. Kuhn,et al.  The Structure of Scientific Revolutions , 1963 .

[7]  Luís M. A. Bettencourt,et al.  Invention as a combinatorial process: evidence from US patents , 2014, Journal of The Royal Society Interface.

[8]  W. Brian Arthur,et al.  The Nature of Technology: What it Is and How it Evolves , 2009 .

[9]  C. Peirce,et al.  Prolegomena to a Science of Reasoning: Phaneroscopy, Semeiotic, Logic , 2015 .

[10]  F. Sanger,et al.  DNA sequencing with chain-terminating inhibitors. , 1977, Proceedings of the National Academy of Sciences of the United States of America.

[11]  Mark E. J. Newman,et al.  Stochastic blockmodels and community structure in networks , 2010, Physical review. E, Statistical, nonlinear, and soft matter physics.

[12]  J. Grilli,et al.  Higher-order interactions stabilize dynamics in competitive network models , 2017, Nature.

[13]  Jordan L. Boyd-Graber,et al.  Measuring discursive influence across scholarship , 2018, Proceedings of the National Academy of Sciences.

[14]  Erin Leahey,et al.  Prominent but Less Productive , 2015, ArXiv.

[15]  T. Kuhn.  The Structure of Scientific Revolutions 2nd edition , 1970 .

[16]  Dale S. Niederhauser,et al.  The Nature of Technology , 2013 .

[17]  Pierre Azoulay,et al.  Toward a more scientific science , 2018, Science.

[18]  D. C. Tsui,et al.  Two-Dimensional Magnetotransport in the Extreme Quantum Limit , 1982 .

[19]  David F. Channell.  Pasteur's Quadrant: Basic Science and Technological Innovation , 1999 .

[20]  Santo Fortunato,et al.  A dataset of publication records for Nobel laureates , 2019, Scientific Data.

[21]  Vincent Larivière,et al.  On the relationship between interdisciplinarity and scientific impact , 2009, J. Assoc. Inf. Sci. Technol..

[22]  Robert K. Merton,et al.  The travels and adventures of serendipity , 2004 .

[23]  Stefano Allesina,et al.  Beyond pairwise mechanisms of species coexistence in complex communities , 2017, Nature.

[24]  John E. Brandl,et al.  Pasteur's Quadrant: Basic science and technological innovation , 1998 .

[25]  H. Chesbrough,et al.  Open Innovation: A New Paradigm for Understanding Industrial Innovation , 2006 .

[26]  James A. Evans.  Industry Induces Academic Science to Know Less about More1 , 2010, American Journal of Sociology.

[27]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[28]  W. Myers,et al.  Atypical Combinations and Scientific Impact , 2013 .

[29]  Jure Leskovec,et al.  Higher-order organization of complex networks , 2016, Science.

[30]  T. Pinch,et al.  The Social Construction of Facts and Artefacts: or How the Sociology of Science and the Sociology of Technology might Benefit Each Other , 1984 .

[31]  Jacob G Foster,et al.  Choosing experiments to accelerate collective discovery , 2015, Proceedings of the National Academy of Sciences.