Pairwise Comparison Based Ranking Vector Approach to Estimation Performance Ranking

Given multiple statistical estimators, how to rank their performance is a worthwhile problem. We propose an approach to estimation performance ranking based on pairwise comparison. Because ranking is about two or more estimators and is relative, pairwise comparison (e.g., Pitman’s measure of closeness) suits the needs. However, pairwise comparison cannot guarantee transitivity, which is needed for ranking. To get around this problem, we propose a ranking vector (RV) approach based on pairwise comparison. Here, an RV is obtained by using pairwise comparison results without using pairwise ranks directly. An RV provides ordinal information determining the rank and also supplementary cardinal information exhibiting how much one estimator is better than another. Ordinal information is more important and thus is guaranteed by using an order-preserving mapping in obtaining an RV. Our RV approach based on pairwise comparison is also applied to multiple-attribute decision problems. The approach is easily applicable and it does not need data normalization.

[1]  Slava Kisilevich,et al.  Initial Profile Generation in Recommender Systems Using Pairwise Comparison , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[2]  Carl D. Meyer,et al.  Matrix Analysis and Applied Linear Algebra , 2000 .

[3]  Francisco Herrera,et al.  Aggregation operators for linguistic weighted information , 1997, IEEE Trans. Syst. Man Cybern. Part A.

[4]  E. J. G. Pitman Hobart.  The “closest” estimates of statistical parameters , 1991 .

[5]  Sudip Bose,et al.  On the transitivity of the posterior Pitman closeness criterion , 1998 .

[6]  Erik Blasch,et al.  Track purity and current assignment ratio for target tracking and identification evaluation , 2011, 14th International Conference on Information Fusion.

[7]  David de Andrés,et al.  From Measures to Conclusions Using Analytic Hierarchy Process in Dependability Benchmarking , 2014, IEEE Transactions on Instrumentation and Measurement.

[8]  Chee-Yee Chong.  Problem characterization in tracking/fusion algorithm evaluation , 2000, Proceedings of the Third International Conference on Information Fusion.

[9]  X. Rong Li,et al.  Iterative Mid-Range with Application to Estimation Performance Evaluation , 2015, IEEE Signal Processing Letters.

[10]  A. Granas,et al.  Fixed Point Theory , 2003 .

[11]  Ronald R. Yager,et al.  On ordered weighted averaging aggregation operators in multicriteria decisionmaking , 1988, IEEE Trans. Syst. Man Cybern..

[12]  U. Krause.  Concave Perron–Frobenius Theory and applications , 2001 .

[13]  LI X.RONG,et al.  Evaluation of estimation algorithms part I: incomprehensive measures of performance , 2006, IEEE Transactions on Aerospace and Electronic Systems.

[14]  X. Rong Li,et al.  Two classes of relative measures of estimation performance , 2007, 2007 10th International Conference on Information Fusion.

[15]  Ernest H. Forman,et al.  Promoting shared health care decision making using the analytic hierarchy process , 1999 .

[16]  Jian Chen,et al.  MAGDM Linear-Programming Models With Distinct Uncertain Preference Structures , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[17]  R. L. Mason,et al.  Pitman's Measure of Closeness: A Comparison of Statistical Estimators , 1987 .

[18]  Zeshui Xu,et al.  Multiple-Attribute Group Decision Making With Different Formats of Preference Information on Attributes , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[19]  Tian Zhi,et al.  Performance Evaluation of Track Fusion with Information , 2002 .

[20]  J. Gillis,et al.  Matrix Iterative Analysis , 1961 .

[21]  Genshe Chen,et al.  Information theoretic measures for performance evaluation and comparison , 2009, 2009 12th International Conference on Information Fusion.

[22]  Alan C. Bovik,et al.  Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures , 2009, IEEE Signal Processing Magazine.

[23]  X. Rong Li,et al.  Evaluation of Estimation Algorithms: Credibility Tests , 2012, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[24]  Ching-Lai Hwang,et al.  Multiple Attribute Decision Making — An Overview , 1992 .

[25]  Abraham Seidmann,et al.  Performance evaluation of flexible manufacturing systems , 1984, IEEE Transactions on Systems, Man, and Cybernetics.

[26]  Kamal M. Al‐Subhi Al‐Harbi,et al.  Application of the AHP in project management , 2001 .

[27]  X. Rong Li,et al.  Measures for ranking estimation performance based on single or multiple performance metrics , 2013, Proceedings of the 16th International Conference on Information Fusion.

[28]  T. Saaty,et al.  The Analytic Hierarchy Process , 1985 .