Agreement of expert estimates

Abstract The application of multicriteria methods largely depends on the calculation of the criteria weights based on expert evaluation. The results obtained can be used for practical purposes if expert judgments are in good agreement. This may be determined by the concordance of a coefficient obtained by ranking the available alternatives. The paper considers a possibility to apply the concordance coefficient in cases when expert evaluation is not based on ranking. The calculations reveal the dependence of the agreement of expert estimates on a particular method used. The highest degree of agreement has been obtained by using the direct ranking method. The effect of equally assessed criteria, i. e. the tied ranks on the concordance coefficient and thereby on the level of expert judgments agreement is usually insignificant and cannot change the results of rating.