Expertise-based ranking of experts: An assessment level approach

The quality of a formal decision is influenced by the level of expertise of the decision makers (DMs). The composition of a team of DMs can change when new members join or old members leave, based on their ranking. In order to improve the quality of decisions, this ranking should be based on their demonstrated expertise. This paper proposes using the experts' expertise levels, in terms of the ability to differentiate consistently, to determine their ranking, according to the level at which they assess alternatives. The expertise level is expressed using the CWS-Index (CochranWeissShanteau), a ratio between Discrimination and Inconsistency. The experts give their evaluations using pairwise comparisons of Fuzzy Preference Relations with an Additive Consistency property. This property can be used to generate estimators, and replaces the repetition needed to obtain the CWS-Index. Finally, a numerical example is discussed to illustrate the model for producing expertise-based ranking of experts.

[1]  Suparno Suparno,et al.  EXPERTISE-BASED EXPERTS IMPORTANCE WEIGHTS INADVERSE JUDGMENT , 2014 .

[2]  Francisco Herrera,et al.  Group decision making with incomplete fuzzy linguistic preference relations , 2009, Int. J. Intell. Syst..

[3]  T. Tanino.  Fuzzy preference orderings in group decision making , 1984 .

[4]  John R. Anderson.  Cognitive Psychology and Its Implications , 1980 .

[5]  Francisco Herrera,et al.  A consistency‐based procedure to estimate missing pairwise preference values , 2008, Int. J. Intell. Syst..

[6]  J. S. Hunter,et al.  Statistics for Experimenters: Design, Innovation, and Discovery , 2006 .

[7]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[8]  José Luis García-Lapresta,et al.  Modeling rationality in a linguistic framework , 2009, Fuzzy Sets Syst..

[9]  Enrique Herrera-Viedma,et al.  A Note on Two Methods for Estimating Missing Pairwise Preference Values , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[10]  Bernard De Baets,et al.  Cyclic Evaluation of Transitivity of Reciprocal Relations , 2006, Soc. Choice Welf..

[11]  Bernard De Baets,et al.  Aggregation of monotone reciprocal relations with application to group decision making , 2011, Fuzzy Sets Syst..

[12]  Francisco Herrera,et al.  Some induced ordered weighted averaging operators and their use for solving group decision-making problems based on fuzzy preference relations , 2007, Eur. J. Oper. Res..

[13]  Michael D. Lee,et al.  Inferring Expertise in Knowledge and Prediction Ranking Tasks , 2012, Top. Cogn. Sci..

[14]  Francisco Herrera,et al.  Some issues on consistency of fuzzy preference relations , 2004, Eur. J. Oper. Res..

[15]  B. De Baets,et al.  Transitivity frameworks for reciprocal relations: cycle-transitivity versus FG-transitivity , 2005, Fuzzy Sets Syst..

[16]  J. Shanteau.  Competence in experts: The role of task characteristics , 1992 .

[17]  John S. Gero,et al.  Modelling expertise of temporary design teams , 2004 .

[18]  G. A. Miller.  THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[19]  T. A. Wheeler,et al.  Use of expert judgment in NUREG--1150 , 1991 .

[20]  K. R. Hammond.  Human judgment and social policy , 1980 .

[21]  Francisco Herrera,et al.  Group Decision-Making Model With Incomplete Fuzzy Preference Relations Based on Additive Consistency , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[22]  T. Saaty,et al.  Why the magic number seven plus or minus two , 2003 .

[23]  M. Chi,et al.  The Nature of Expertise , 1988 .

[24]  Fei Yu,et al.  Assigning Method for Decision Power Based on Linguisitc 2-tuple Judgment Matrices , 2011, J. Softw..

[25]  Zhi-ping Fan,et al.  Study on Assessment Level of Experts based on Difference Preference Information , 2007 .

[26]  David J. Weiss,et al.  Performance-based assessment of expertise: How to decide if someone is an expert or not , 2002, Eur. J. Oper. Res..

[27]  Bernard De Baets,et al.  Closing reciprocal relations w.r.t. stochastic transitivity , 2014, Fuzzy Sets Syst..

[28]  Yin-Feng Xu,et al.  Linear optimization modeling of consistency issues in group decision making based on fuzzy preference relations , 2012, Expert Syst. Appl..

[29]  James Shanteau,et al.  Who's the Best? A Relativistic View of Expertise , 2014 .

[30]  Thomas L. Saaty,et al.  Making decisions in hierarchic and network systems , 2008, Int. J. Appl. Decis. Sci..

[31]  J. Kacprzyk,et al.  Multiperson decision making models using fuzzy sets and possibility theory , 1990 .

[32]  Francisco Herrera,et al.  A Consensus Model for Group Decision Making With Incomplete Fuzzy Preference Relations , 2007, IEEE Transactions on Fuzzy Systems.

[33]  Jing Liu,et al.  Method on the Assessment Level of Experts in Group Decision Making Based on Multi-Granularity Linguistic Judgment Matrices , 2009, 2009 International Conference on Information Engineering and Computer Science.

[34]  David J. Weiss,et al.  Empirical Assessment of Expertise , 2003, Hum. Factors.

[35]  J. Bezdek,et al.  Fuzzy relation spaces for group decision theory: An application , 1979 .

[36]  Francisco Herrera,et al.  A web based consensus support system for group decision making problems and incomplete preferences , 2010, Inf. Sci..

[37]  James Shanteau,et al.  Decloaking the privileged expert , 2012 .

[38]  Hillel J. Einhorn,et al.  Expert measurement and mechanical combination , 1972 .

[39]  Joseph G. Pigeon,et al.  Statistics for Experimenters: Design, Innovation and Discovery , 2006, Technometrics.

[40]  J. Kacprzyk.  Group decision making with a fuzzy linguistic majority , 1986 .

[41]  Zbigniew Switalski,et al.  General transitivity conditions for fuzzy reciprocal preference matrices , 2003, Fuzzy Sets Syst..

[42]  F. Herrera,et al.  A consistency-based procedure to estimate missing pairwise preference values , 2008 .

[43]  Vinod Kumar Malhotra,et al.  Domain experts influence decision quality: Towards a robust method for their identification , 2007 .

[45]  Chris Horn.  How Experience Affects Perception in Expert Decision-Making , 2006 .

[46]  J. Bring,et al.  How do GPs use clinical information in their judgements of heart failure? A clinical judgement analysis study. , 1998, Scandinavian journal of primary health care.

[47]  Francisco Herrera,et al.  Integrating three representation models in fuzzy multipurpose decision making based on fuzzy preference relations , 1998, Fuzzy Sets Syst..

[48]  Enrique Herrera-Viedma,et al.  Preferences and Consistency Issues in Group Decision Making , 2008, Fuzzy Sets and Their Extensions: Representation, Aggregation and Models.

[49]  R. Cooke.  Experts in Uncertainty: Opinion and Subjective Probability in Science , 1991 .

[50]  H. J. Einhorn.  Expert judgment: Some necessary conditions and an example. , 1974 .

[51]  Müjgan Sagir Ozdemir,et al.  Validity and inconsistency in the analytic hierarchy process , 2005 .