What do global university rankings really measure? The search for the X factor and the X entity

Most academic rankings attempt to measure the quality of university education and research. However, previous studies that examine the most influential rankings conclude that the variables they use could be an epiphenomenon of an X factor that has little to do with quality. The aim of this study is to investigate the existence of this hidden factor or profile in the two most influential global university rankings in the world: the Academic Ranking of World Universities (ARWU) of the University of Shanghai Jiao Tong, and the Times Higher Education (THE) ranking. Results support the existence of an underlying entity profile, characterized by institutions normally from the US that enjoy a high reputation. Results also support the idea that rankings lack the capacity to assess university quality in all its complexity, and two strategies are suggested in relation to the vicious circle created between institutional reputation and rankings.

[1]  Massimo Savino,et al.  A Global Survey of Rankings and League Tables/ College and University Ranking Systems Global Perspectives and American Challenges / Institute for Higher Education Policy (editor). April 2007. P. 23–34 , 2007 .

[2]  Cinzia Daraio,et al.  Universities and Strategic Knowledge Creation. Specialization and Performance in Europe , 2007 .

[3]  L. Amaral,et al.  Challenges in Ranking of Universities , 2005 .

[4]  D. B. Montgomery,et al.  First‐mover advantages , 1988 .

[5]  David J. Smyth.  The Determinants of the Reputations of Economics Departments: Pages Published, Citations and the Andy Rooney Effect , 1999 .

[6]  J. Salmi,et al.  League Tables as Policy Instruments: Uses and Misuses , 2007 .

[7]  Alex Usher,et al.  A World of Difference: A Global Survey of University League Tables. Canadian Education Report Series. , 2006 .

[8]  Kuang-hua Chen,et al.  A comparative study on world university rankings: a bibliometric survey , 2012, Scientometrics.

[9]  Lutz Bornmann,et al.  Peer Review and Bibliometric: Potentials and Problems , 2011 .

[10]  Gregory S. Carpenter,et al.  Consumer Preference Formation and Pioneering Advantage , 1989 .

[11]  Nick von Tunzelmann,et al.  The Effects of Size on Research Performance: A SPRU Review , 2003 .

[12]  Bernard Longden,et al.  Ranking Indicators and Weights , 2011 .

[13]  J. Gómez-Sancho,et al.  International Higher Education Rankings at a Glance: How to Valorise the Research in Social Sciences and Humanities? , 2012 .

[14]  Dan Johnson.  Getting Noticed in Economics: The Determinants of Academic Citations , 1997 .

[15]  A. Usher,et al.  A Global Survey of University Ranking and League Tables , 2007 .

[16]  K. Webber,et al.  Measuring the Research Performance of Postsecondary Institutions , 2011 .

[17]  Anthony F. J. van Raan.  Reply to the comments of Liu et al. , 2005, Scientometrics.

[18]  Franciscus A. van Vught,et al.  U-Multirank: design and testing the feasability of a multidimensional global university ranking: final report , 2011 .

[19]  D. Dill,et al.  Academic quality, league tables, and public policy: A cross-national analysis of university ranking systems , 2005 .

[20]  J. S. Katz,et al.  The self-similar science system , 1999 .

[21]  C. Fombrun,et al.  What's in a Name? Reputation Building and Corporate Strategy , 1990 .

[22]  J. Scott Armstrong,et al.  Business School Prestige - Research Versus Teaching , 2005 .

[23]  N. Adler,et al.  When Knowledge Wins: Transcending the Sense and Nonsense of Academic Rankings , 2009 .

[24]  Jean-Charles Billaut,et al.  Should you believe in the Shanghai ranking? , 2010, Scientometrics.

[25]  Kevin G. Corley,et al.  Being Good Versus Looking Good: Business School Rankings and the Circean Transformation From Substance to Image , 2002 .

[26]  W. C. Patterson,et al.  FIRST-MOVER ADVANTAGE: THE OPPORTUNITY CURVE , 1993 .

[27]  D. Rigney,et al.  The Matthew Effect: How Advantage Begets Further Advantage , 2011 .

[28]  Mu-Hsuan Huang,et al.  Opening the black box of QS World University Rankings , 2012 .

[29]  Roberto Rodríguez Gómez,et al.  El ranking Times en el mercado del prestigio universitario , 1969 .

[30]  Anthony F. J. van Raan,et al.  Statistical properties of bibliometric indicators: Research group indicator distributions and correlations , 2006, J. Assoc. Inf. Sci. Technol..

[31]  Sriram Shankar,et al.  Catching up with Harvard: results from regression analysis of world universities league tables , 2011 .

[32]  Vicente Safón,et al.  Can the reputation of an established business school change? , 2012 .

[33]  Heike Wetzel,et al.  Competing in the Higher Education Market: Empirical Evidence for Economies of Scale and Scope in German Higher Education Institutions , 2014 .

[34]  N. Dyke.  Twenty Years of University Report Cards , 2005 .

[35]  Rachelle L. Brooks.  Measuring University Quality , 2005 .

[36]  Salih Murat Akkın,et al.  Kitap Değerlendirmesi: Global University Rankings and Their Impact / A. Rauhvargers, 2011 , 2011 .

[37]  I V Trotsuk,et al.  Ranking of Higher Education Institutions: Ideology and Methodology of Development (Russian Practice) , 2009 .

[38]  A. Oswald,et al.  Can We Test for Bias in Scientific Peer-Review? , 2008 .

[39]  R. Braddock,et al.  International University Ranking Systems and the Idea of University Excellence , 2007 .

[40]  Jung Cheol Shin,et al.  The Past, Present, and Future of University Rankings , 2011 .

[41]  Simon Marginson,et al.  Global University Rankings: Implications in general and for Australia , 2007 .

[42]  Catherine O’Connell,et al.  Research discourses surrounding global university rankings: exploring the relationship with policy and practice recommendations , 2013 .

[43]  魏屹东,et al.  Scientometrics , 2018, Encyclopedia of Big Data.

[44]  Ellen Hazelkorn,et al.  Impact of Global Rankings on Higher Education Research and the Production of Knowledge , 2009 .

[45]  Eric L. Dey,et al.  Changing Patterns of Publication Productivity: Accumulative Advantage or Institutional Isomorphism?. , 1997 .

[46]  Sriram Shankar,et al.  Why does the USA dominate university league tables? , 2011 .

[47]  Luigi Zingales,et al.  The Corporate Governance Role of the Media , 2002 .

[48]  Gualberto Buela-Casal,et al.  Comparative study of international academic rankings of universities , 2007, Scientometrics.

[49]  Robert T. Blackburn,et al.  Changes in academic research performance over time: A study of institutional accumulative advantage , 1989 .

[50]  Fernando F. Suarez,et al.  The Role of Environmental Dynamics in Building a First Mover Advantage Theory , 2007 .

[51]  Domingo Docampo,et al.  On using the Shanghai ranking to assess the research performance of university systems , 2010, Scientometrics.

[52]  K. Henkens,et al.  Signals in Science - on the Importance of Signaling in Gaining Attention in Science , 2004 .

[53]  Jung Cheol Shin,et al.  Organizational Effectiveness and University Rankings , 2011 .

[54]  Philip G. Altbach,et al.  The Dilemmas of Ranking , 2015 .

[55]  Gustavo Crespi.  The UK Knowledge Production Function , 2007 .

[56]  Thed N. van Leeuwen,et al.  Language biases in the coverage of the Science Citation Index and its consequencesfor international comparisons of national research performance , 2001, Scientometrics.

[57]  Ulrich Teichler,et al.  Social Contexts and Systemic Consequence of University Rankings: A Meta-Analysis of the Ranking Literature , 2011 .

[58]  Juha Hedman,et al.  Productivity analysis of research in Natural Sciences, Technology and Clinical Medicine: an input–output model applied in comparison of Top 300 ranked universities of 4 North European and 4 East Asian countries , 2013, Scientometrics.

[59]  B. Keith,et al.  Organizational Contexts and University Performance Outcomes: The Limited Role of Purposive Action in the Management of Institutional Status , 2001 .

[60]  Michael Power,et al.  The audit society : rituals of verification , 1997 .

[61]  Sara L. Rynes,et al.  Who moved our cheese? Reclaiming professionalism in business education , 2003 .

[62]  Li Liu,et al.  Academic ranking of world universities using scientometrics  - A comment to the “Fatal Attraction” , 2005, Scientometrics.

[63]  A. Saltelli,et al.  Rickety numbers: Volatility of university rankings and policy implications , 2011 .

[64]  Nicholas A. Bowman,et al.  Anchoring effects in world university rankings: exploring biases in reputation scores , 2011 .

[65]  John T. E. Richardson,et al.  Counting What Is Measured or Measuring What Counts? League Tables and Their Impact On Higher Education Institutions in England , 2008 .

[66]  J. Scott Armstrong,et al.  The Ombudsman: Business School Prestige—Research versus Teaching , 1994 .

[67]  Subhash C. Ray,et al.  MBA Program Reputation: Quantitative Rankings for Students, Employers, and Program Administrators , 2005 .

[68]  Saisana Michaela,et al.  Higher Education Rankings: Robustness Issues and Critical Assessment , 2008 .

[69]  Nicholas A. Bowman,et al.  U.S. News & World Report College Rankings: Modeling Institutional Effects on Organizational Reputation , 2009, American Journal of Education.

[70]  Frederick P. Morgeson,et al.  Same as It Ever Was: Recognizing Stability in the BusinessWeek Rankings , 2008 .

[71]  Anthony F. J. van Raan,et al.  Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods , 2005, Scientometrics.

[72]  Ilia D. Dichev.  How Good Are Business School Rankings , 1999 .

[73]  Nian Cai Liu,et al.  The world-class university and ranking : aiming beyond status , 2007 .

[74]  Ross Williams,et al.  Broadening the Criteria: Lessons from the Australian Rankings , 2005 .

[75]  Isabel Gómez,et al.  Structure and research performance of Spanish universities , 2009, Scientometrics.

[76]  R. Merton.  The Matthew Effect in Science , 1968, Science.

[77]  Ulrich Teichler.  The Future of University Rankings , 2011 .

[78]  D. Eastwood,et al.  Counting What Is Measured or Measuring What Counts? League Tables and Their Impact On Higher Education Institutions in England , 2008 .

[79]  J. R. Cole,et al.  Scientific output and recognition: a study in the operation of the reward system in science. , 1967, American sociological review.

[80]  J. Hair.  Multivariate data analysis , 1972 .