How to standardize (if you must)

In many situations we are interested in appraising the value of a certain characteristic for a given individual relative to the context in which this value is observed. In recent years this problem has become prominent in the evaluation of scientific productivity and impact. A popular approach to such relative valuations consists in using percentile ranks. This is a purely ordinal method that may sometimes lead to counterintuitive appraisals, in that it discards all information about the distance between the raw values within a given context. By contrast, this information is partly preserved by using standardization, i.e., by transforming the absolute values in such a way that, within the same context, the distance between the relative values is monotonically related to the distance between the absolute ones. While there are many practically useful alternatives for standardizing a given characteristic across different contexts, the general problem seems to have never been addressed from a theoretical and normative viewpoint. The main aim of this paper is to fill this gap and provide a conceptual framework that allows for this kind of systematic investigation. We then use this framework to prove that, under some rather weak assumptions, the general format of a standardization function can be determined quite sharply.

[1]  Tom Tullis,et al.  Measuring the User Experience, Second Edition: Collecting, Analyzing, and Presenting Usability Metrics , 2013 .

[2]  Péter Vinkler,et al.  The case of scientometricians with the "absolute relative" impact indicator , 2012, J. Informetrics.

[3]  Filippo Radicchi,et al.  Quantitative evaluation of alternative field normalization procedures , 2013, J. Informetrics.

[4]  H. Moed CWTS crown indicator measures citation impact of a research group's publication oeuvre , 2010, J. Informetrics.

[5]  Claudio Castellano,et al.  Universality of citation distributions: Toward an objective measure of scientific impact , 2008, Proceedings of the National Academy of Sciences.

[6]  Jonas Lundberg,et al.  Lifting the crown - citation z-score , 2007, J. Informetrics.

[7]  Tindaro Cicero,et al.  How important is choice of the scaling factor in standardizing citations? , 2012, J. Informetrics.

[8]  Montfort Mlachila,et al.  A Quality of Growth Index for Developing Countries: A Proposal , 2014, Social Indicators Research.

[9]  Ludo Waltman,et al.  A review of the literature on citation impact indicators , 2015, J. Informetrics.

[10]  Montfort Mlachila,et al.  A Quality of Growth Index for Developing Countries: A Proposal , 2014, Social Indicators Research.

[11]  Thed N. van Leeuwen,et al.  Towards a new crown indicator: Some theoretical considerations , 2010, J. Informetrics.

[12]  W. Sturm,et al.  Neuropsychological assessment , 2007, Journal of Neurology.

[13]  Yunlong Wang Chapter 2 Use of Percentiles and Z-Scores in Anthropometry , 2011 .

[14]  Ludo Waltman,et al.  On the calculation of percentile-based bibliometric indicators , 2012, J. Assoc. Inf. Sci. Technol..

[15]  Pedro Albarrán,et al.  The skewness of science in 219 sub-fields and a number of aggregates , 2010, Scientometrics.

[16]  G. W. Milligan,et al.  A study of standardization of variables in cluster analysis , 1988 .

[17]  D. Saccuzzo,et al.  Psychological Testing: Principles, Applications, and Issues , 1982 .

[18]  Thed N. van Leeuwen,et al.  The Holy Grail of science policy: Exploring and combining bibliometric tools in search of scientific excellence , 2003, Scientometrics.

[19]  A. M. Stoddard,et al.  Standardization of measures prior to cluster analysis. , 1979, Biometrics.

[20]  Stefano Tarantola,et al.  Handbook on Constructing Composite Indicators: Methodology and User Guide , 2005 .

[21]  Ying Cheng,et al.  Comparison of the effect of mean-based method and z-score for field normalization of citations at the level of Web of Science subject categories , 2014, Scientometrics.

[22]  Loet Leydesdorff,et al.  Turning the tables in citation analysis one more time: Principles for comparing sets of documents by using an “Integrated Impact Indicator” (I3) , 2011 .

[23]  D. Varberg Convex Functions , 1973 .

[24]  Thed N. van Leeuwen,et al.  Rivals for the crown: Reply to Opthof and Leydesdorff , 2010, J. Informetrics.

[25]  Jeff Sauro,et al.  A method to standardize usability metrics into a single score , 2005, CHI.

[26]  Thed N. van Leeuwen,et al.  Benchmarking international scientific excellence: Are highly cited research papers an appropriate frame of reference? , 2002, Scientometrics.