Measures for ranking estimation performance based on single or multiple performance metrics

There are several error metrics for estimation performance evaluation. To rank the performance of estimators, a popular method is using the same error metric of performance. It is not without controversy. First, this ranking method depending on the “marginal” information without considering the “joint” information among the estimators is one-sided since different error metrics reflect different aspects of performance. Second, ranking according to different error metrics may lead to different results. Thus, we propose to use the “joint” information just like Pitman's closeness measure (PCM) to rank the performance of estimators. However, one drawback of PCM, named nontransitivity, brings big trouble for estimation performance ranking. To rank estimators utilizing the “joint” information, we propose a new approach using a so-called estimator ranking vector (ERV). The elements of ERV reflect how good the corresponding estimators are. Order-preserving mappings are proposed to obtain ERV, which, however, may not be unique. Then we use three specific mappings (i.e., linear, contraction, and concave, respectively) to solve this problem. Linear mappings can be easily applied and the other two mappings broaden the application domain of ERV. The ranking vector can also be used in multiple-attribute ranking problem. It does not need data normalization.

[1]  X. Rong Li,et al.  Evaluation of Estimation Algorithms: Credibility Tests , 2012, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[2]  Ye.G. D'yakonov.  Matrix iterative analysis: Varga, R. S., 1963☆ , 1965 .

[3]  Branko Ristic,et al.  A Metric for Performance Evaluation of Multi-Target Tracking Algorithms , 2011, IEEE Transactions on Signal Processing.

[4]  LI X.RONG,et al.  Evaluation of estimation algorithms part I: incomprehensive measures of performance , 2006, IEEE Transactions on Aerospace and Electronic Systems.

[5]  Tian Jie,et al.  Notice of RetractionResearch on Chinese cities comprehensive competitiveness based on principal component analysis and factor analysis in SPSS , 2011, 2011 IEEE 2nd International Conference on Software Engineering and Service Science.

[6]  U. Krause.  Concave Perron–Frobenius Theory and applications , 2001 .

[7]  X. Rong Li,et al.  Two classes of relative measures of estimation performance , 2007, 2007 10th International Conference on Information Fusion.

[8]  Carl D. Meyer,et al.  Matrix Analysis and Applied Linear Algebra , 2000 .

[9]  C. R. Rao,et al.  The pitman nearness criterion and its determination , 1986 .

[10]  X.R. Li,et al.  Relative error measures for evaluation of estimation algorithms , 2005, 2005 7th International Conference on Information Fusion.

[11]  Erik Blasch,et al.  Track purity and current assignment ratio for target tracking and identification evaluation , 2011, 14th International Conference on Information Fusion.

[12]  X. Rong Li,et al.  New robust metrics of central tendency for estimation performance evaluation , 2012, 2012 15th International Conference on Information Fusion.

[13]  Lennart Svensson,et al.  Performance evaluation of MHT and GM-CPHD in a ground target tracking scenario , 2009, 2009 12th International Conference on Information Fusion.

[14]  Edmund Førland Brekke,et al.  Performance of PDAF-based tracking methods in heavy-tailed clutter , 2009, 2009 12th International Conference on Information Fusion.

[15]  Sai-jing Zhu.  Comprehensive evaluation of green construction based on AHP and grey theory , 2012, 2012 International Conference on Information Management, Innovation Management and Industrial Engineering.

[16]  Zhou Hui,et al.  Notice of RetractionApplication of weighted Principal Component Analysis in comprehensive evaluation for power quality , 2011, 2011 IEEE Power Engineering and Automation Conference.

[17]  Chee-Yee Chong.  Problem characterization in tracking/fusion algorithm evaluation , 2000, Proceedings of the Third International Conference on Information Fusion.

[18]  Yaakov Bar-Shalom,et al.  Multisensor target-tracking performance with bias compensation , 2005, SPIE Optics + Photonics.

[19]  E. J. G. Pitman Hobart.  The “closest” estimates of statistical parameters , 1991 .

[20]  Malay Ghosh,et al.  Bayesian pitman closeness , 1991 .

[21]  Ondrej Straka,et al.  Performance evaluation of local state estimation methods in bearings-only tracking problems , 2011, 14th International Conference on Information Fusion.

[22]  A. Granas,et al.  Fixed Point Theory , 2003 .

[23]  Genshe Chen,et al.  Information theoretic measures for performance evaluation and comparison , 2009, 2009 12th International Conference on Information Fusion.

[24]  Matthieu Geist,et al.  Performance evaluation for particle filters , 2011, 14th International Conference on Information Fusion.

[25]  Bin Zhang,et al.  Application of Entropy Weight Fuzzy Comprehensive Evaluation in Optimal Selection of Engineering Machinery , 2008, 2008 ISECS International Colloquium on Computing, Communication, Control, and Management.

[26]  R. L. Mason,et al.  Pitman's Measure of Closeness: A Comparison of Statistical Estimators , 1987 .

[27]  Tian Zhi,et al.  Performance Evaluation of Track Fusion with Information , 2002 .

[28]  Sudip Bose,et al.  On the transitivity of the posterior Pitman closeness criterion , 1998 .

[29]  Thia Kirubarajan,et al.  Performance measures for multiple target tracking problems , 2011, 14th International Conference on Information Fusion.

[30]  N. Rescher.  Discourse on a Method , 1968 .

[31]  Li-Fang Shi.  Entropy Based Fuzzy Comprehensive Evaluation of University Teachers , 2010, 2010 First International Conference on Pervasive Computing, Signal Processing and Applications.

[32]  X. R. Li,et al.  Measures of performance for evaluation of estimators and filters , 2001 .

[33]  Y. Bar-Shalom,et al.  Performance evaluation of multisensor track-to-track fusion , 1996, 1996 IEEE/SICE/RSJ International Conference on Multisensor Fusion and Integration for Intelligent Systems (Cat. No.96TH8242).

[34]  Oliver E. Drummond.  Methodologies for performance evaluation of multitarget multisensor tracking , 1999, Optics & Photonics.