Challenges for scientometric indicators: data demining, knowledge-flow measurements and diversity issues

Scientometrics and bibliometrics are being forced to respond to a strong increase in demand (e.g. research assessment practices, economics of science & technology, and innovation) and new forms of supply (e.g. availability of publication sources and statistics, Internet developments and online tools). This situation results in contrasting perspectives: on the one hand, it can favour spectac- ular 'hit-parades' and some veneration of numbers; on the other hand, it paves the way for more cau- tious and sophisticated evaluation systems, rooted in a better understanding of the dynamics of sci- ence. This paper describes some of the challenges for bibliometric indicators (data 'demining', knowledge-flow measurements and diversity issues) underlying, among other applications, reliable evaluation procedures. Responding to these challenges is necessary to promote a better use of scien- tometrics, although there are no guarantees against misuse in decision-making contexts. A few open issues are outlined on the dynamics of science, challenges of the web age, and interactions between scientometrics and scientific communities.

[1]  A. Barabasi,et al.  Evolution of the social network of scientific collaborations , 2001, cond-mat/0104162.

[2]  Paula E. Stephan.  The Economics of Science , 1996 .

[3]  Michael J. Moravcsik,et al.  Variation of the nature of citation measures with journals and scientific specialties , 1978, J. Am. Soc. Inf. Sci..

[4]  Judit Bar-Ilan,et al.  The sources used by bibliometrics-scientometrics as reflected in references , 2002, Scientometrics.

[5]  C. M. Noyons.  Science Maps Within a Science Policy Context , 2004 .

[6]  M. Jackson,et al.  A Strategic Model of Social and Economic Networks , 1996 .

[7]  José Luis Ortega,et al.  Scientific research activity and communication measured with cybermetrics indicators , 2006, J. Assoc. Inf. Sci. Technol..

[8]  Thed N. van Leeuwen,et al.  The Holy Grail of science policy: Exploring and combining bibliometric tools in search of scientific excellence , 2003, Scientometrics.

[9]  Michel Zitt,et al.  Mapping nanosciences by citation flows: A preliminary analysis , 2007, Scientometrics.

[10]  Wolfgang Glänzel,et al.  The decline of Swedish neuroscience: Decomposing a bibliometric national science indicator , 2004, Scientometrics.

[11]  W. Cheung.  The economics of post-doc publishing , 2008 .

[12]  B. Latour,et al.  Laboratory Life: The Construction of Scientific Facts , 1979 .

[13]  B. C. Griffith,et al.  The Structure of Scientific Literatures I: Identifying and Graphing Specialties , 1974 .

[14]  Michel Zitt,et al.  Relativity of citation performance and excellence measures: From cross-field to cross-scale effects of field-normalisation , 2005, Scientometrics.

[15]  Stevan Harnad,et al.  Validating research performance metrics against peer rankings , 2008 .

[16]  Ronald Rousseau.  Relations between continuous versions of bibliometric laws , 1990, J. Am. Soc. Inf. Sci..

[17]  Loet Leydesdorff,et al.  The Challenge of Scientometrics: The Development, Measurement, and Self-Organization of Scientific Communications , 2001 .

[18]  Rémi Barré.  Sense and nonsense of S&T productivity indicators , 2001 .

[19]  M. Callon.  The Sociology of an Actor-Network: The Case of the Electric Vehicle , 1986 .

[20]  Mark S. Granovetter.  The Strength of Weak Ties , 1973, American Journal of Sociology.

[21]  Léopold Simar,et al.  Introducing Environmental Variables in Nonparametric Frontier Models: a Probabilistic Approach , 2005 .

[22]  Blaise Cronin,et al.  Normative shaping of scientific practice: The magic of Merton , 2004, Scientometrics.

[23]  Liwen Vaughan,et al.  Webometrics , 2005, Annu. Rev. Inf. Sci. Technol..

[24]  G. Lewison,et al.  Outputs and expenditures on health research in eight disease areas using a bibliometric approach, 1996–2001 , 2004 .

[25]  Ismael Rafols,et al.  Diversity measures and network centralities as indicators of interdisciplinarity: Case studies in BionanoScience , 2007 .

[26]  Cláudia S. Sarrico,et al.  Pitfalls and protocols in DEA , 2001, Eur. J. Oper. Res..

[27]  Michel Zitt,et al.  Correcting glasses help fair comparisons in international science landscape: Country indicators as a function of ISI database delineation , 2003, Scientometrics.

[28]  L. Butler,et al.  Explaining Australia’s increased share of ISI publications—the effects of a funding formula based on publication counts , 2003 .

[29]  Andrea Bonaccorsi.  Matching properties. Search regimes and institutional systems of science , 2002 .

[30]  Michel Zitt,et al.  Shadows of the Past in International Cooperation: Collaboration Profiles of the Top Five Producers of Science , 2000, Scientometrics.

[31]  R. Cowan,et al.  Evolving networks of inventors , 2004 .

[32]  Henk F. Moed,et al.  Handbook of Quantitative Science and Technology Research: The Use of Publication and Patent Statistics in Studies of S&T Systems , 2004 .

[33]  Anton J. Nederhof,et al.  Development of bibliometric indicators for utility of research to users in society: Measurement of external knowledge transfer via publications in trade journals , 2005, Scientometrics.

[34]  Linda Butler,et al.  Using a balanced approach to bibliometrics: quantitative performance measures in the Australian Research Quality Framework , 2008 .

[35]  Duncan J. Watts,et al.  Collective dynamics of ‘small-world’ networks , 1998, Nature.

[36]  Peter Weingart,et al.  Impact of bibliometrics upon the science system: Inadvertent consequences? , 2005, Scientometrics.

[37]  Bart De Moor,et al.  A hybrid mapping of information science , 2008, Scientometrics.

[38]  D. Hicks.  The Four Literatures of Social Science , 2004 .

[39]  Loet Leydesdorff,et al.  Mapping interdisciplinarity at the interfaces between the Science Citation Index and the Social Science Citation Index , 2007, Scientometrics.

[40]  M. Callon,et al.  From translations to problematic networks: An introduction to co-word analysis , 1983 .

[41]  Thed N. van Leeuwen,et al.  Citation delay in interdisciplinary knowledge exchange , 2001, Scientometrics.

[42]  Peter Ingwersen,et al.  Toward a basic framework for webometrics , 2004, J. Assoc. Inf. Sci. Technol..

[43]  Jean-Michel Dalle,et al.  Sequential problem choice and the reward system in Open Science , 2007 .

[44]  D. Aksnes,et al.  Peer reviews and bibliometric indicators: a comparative study at a Norwegian university , 2004 .

[45]  Tibor Braun,et al.  AGAINST ABSOLUTE METHODS: RELATIVE SCIENTOMETRIC INDICATORS AND RELATIONAL CHARTS AS EVALUATION TOOLS , 1988 .

[46]  Paul Wouters,et al.  Citation cycles and peer review cycles , 2006, Scientometrics.

[47]  Michel Zitt,et al.  Development of a method for detection and trend analysis of research fronts built by lexical or cocitation analysis , 1994, Scientometrics.

[48]  Kevin W Boyack,et al.  Mapping knowledge domains: Characterizing PNAS , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[49]  Ed J. Rinia,et al.  COMPARATIVE ANALYSIS OF A SET OF BIBLIOMETRIC INDICATORS AND CENTRAL PEER REVIEW CRITERIA. EVALUATION OF CONDENSED MATTER PHYSICS IN THE NETHERLANDS , 1998 .

[50]  Marc Luwel,et al.  Is the science citation index US-biased? , 1999, Scientometrics.

[51]  Jane M. Russell,et al.  World distribution of social science journals:A view from the periphery , 2001, Scientometrics.

[52]  Howard D. White,et al.  Author cocitation: A literature measure of intellectual structure , 1981, J. Am. Soc. Inf. Sci..

[53]  A. Stirling.  A general framework for analysing diversity in science, technology and society , 2007, Journal of The Royal Society Interface.

[54]  J. S. Katz,et al.  The self-similar science system , 1999 .

[55]  Paul Wouters,et al.  Aux origines de la scientométrie. La naissance du Science Citation Index , 2006 .

[56]  P. Bourdieu.  The specificity of the scientific field and the social conditions of the progress of reason , 1975 .

[57]  Jochen Gläser,et al.  Impact of evaluation-based funding on the production of scientific knowledge: What to worry about, and how to find out (Expertise for the German Ministry for Education and Research) , 2002 .

[58]  Grant Lewison,et al.  The effect of funding on the outputs of biomedical research , 2006, Scientometrics.

[59]  D. Price.  Little Science, Big Science , 1965 .

[60]  Leo Egghe,et al.  The duality of informetric systems with applications to the empirical laws , 1990, J. Inf. Sci..

[61]  Michel Zitt,et al.  Delineating complex scientific fields by an hybrid lexical-citation method: An application to nanosciences , 2006, Inf. Process. Manag..

[62]  Anton J. Nederhof,et al.  Bibliometric monitoring of research performance in the Social Sciences and the Humanities: A Review , 2006, Scientometrics.

[63]  J. ANTHONYF..  Competition amongst scientists for publication status: Toward a model of scientific publication and citation distributions , 2001 .

[64]  Anthony F. J. van Raan,et al.  Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises , 1996, Scientometrics.

[65]  Wolfgang Glänzel,et al.  Two decades of "Scientometrics". An interdisciplinary field represented by its leading journal , 2004, Scientometrics.

[66]  Leo Egghe,et al.  The exact place of Zipf's and Pareto's law amongst the classical informetric laws , 2005, Scientometrics.

[67]  Isabel Gómez,et al.  An approach to interdisciplinarity through bibliometric indicators , 2001, Scientometrics.

[68]  Henry G. Small,et al.  On the shoulders of Robert Merton: Towards a normative theory of citation , 2004, Scientometrics.

[69]  E GARFIELD,et al.  Citation indexes for science; a new dimension in documentation through association of ideas. , 2006, Science.

[70]  Gabriel Pinski,et al.  Citation influence for journal aggregates of scientific publications: Theory, with application to the literature of physics , 1976, Inf. Process. Manag..

[71]  Wolfgang Glänzel,et al.  Cross-national preference in co-authorship, references and citations , 2006, Scientometrics.

[72]  Gaston Heimeriks,et al.  Mapping research topics using word-reference co-occurrences: A method and an exploratory case study , 2006, Scientometrics.

[73]  K. Debackere,et al.  Institutional variations in problem choice and persistence among scientists in an emerging field , 1994 .

[74]  J. E. Hirsch,et al.  An index to quantify an individual's scientific research output , 2005, Proc. Natl. Acad. Sci. USA.

[75]  N. Mohaghegh,et al.  WHY THE IMPACT FACTOR OF JOURNALS SHOULD NOT BE USED FOR EVALUATING RESEARCH , 2005 .

[76]  J. S. Katz,et al.  What is research collaboration , 1997 .