Comparing MCDA Aggregation Methods in Constructing Composite Indicators Using the Shannon-Spearman Measure

Composite indicators have been increasingly recognized as a useful tool for performance monitoring, benchmarking comparisons and public communication in a wide range of fields. The usefulness of a composite indicator depends heavily on the underlying data aggregation scheme where multiple criteria decision analysis (MCDA) is commonly used. A problem in this application is the determination of an appropriate MCDA aggregation method. Of the many criteria for comparing MCDA methods, the Shannon-Spearman measure (SSM) is one that compares alternative MCDA aggregation methods in constructing composite indicators based on the information loss concept. This paper assesses the effectiveness of the SSM using Monte Carlo approach-based uncertain analysis and variance-based sensitivity analysis techniques. It is found that most of the variation in the SSM arises from the uncertainty in choosing an aggregation method. Therefore, the SSM can be considered as an effective measure for comparing MCDA aggregation methods in constructing composite indicators. We also use the SSM to evaluate five MCDA aggregation methods in constructing composite indicators and present the findings.

[1]  David L. Olson,et al.  Comparison of three multicriteria methods to predict known outcomes , 2001, Eur. J. Oper. Res..

[2]  Rolf Färe,et al.  An Economic Approach to Achievement and Improvement Indexes , 2001 .

[3]  J. Church.  Human Development Report , 2001 .

[4]  Risto Lahdelma,et al.  Comparing multicriteria methods in the context of environmental problems , 1998 .

[5]  E. Chan,et al.  The Analytic Hierarchy Process (AHP) Approach for Assessment of Urban Renewal Proposals , 2008 .

[6]  Adel Guitouni,et al.  Tentative guidelines to help choosing an appropriate MCDA method , 1998, Eur. J. Oper. Res..

[7]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[8]  Stefano Tarantola,et al.  Handbook on Constructing Composite Indicators: Methodology and User Guide , 2005 .

[9]  G. Mavrotas,et al.  Determining objective weights in multiple criteria problems: The critic method , 1995, Comput. Oper. Res..

[10]  F. Booysen.  An Overview and Evaluation of Composite Indices of Development , 2002 .

[11]  Barry Turner,et al.  United Nations (UN) , 2009 .

[12]  Andrea Bettini.  Il Sole 24 Ore , 2009 .

[13]  Dimitris K. Despotis,et al.  Measuring human development via data envelopment analysis: the case of Asia and the Pacific , 2005 .

[14]  B. W. Ang,et al.  Comparing aggregating methods for constructing the composite environmental index: An objective measure , 2006 .

[15]  G. Munda.  “Measuring Sustainability”: A Multi-Criterion Framework , 2005 .

[16]  L. Duckstein,et al.  A procedure for selection of a multiobjective technique with application to water and mineral resources , 1984 .

[17]  C. Romero,et al.  In search of a natural systems sustainability index , 2004 .

[18]  Benjamin F. Hobbs,et al.  Does choice of multicriteria method matter? An experiment in water resources planning , 1992 .

[19]  P. Murias,et al.  An Economic Wellbeing Index for the Spanish Provinces: A Data Envelopment Analysis Approach , 2006 .

[20]  H. Welsch,et al.  Meaningful environmental indices: a social choice approach , 2004 .

[21]  B. W. Ang,et al.  Indicators for Assessing Sustainability Performance , 2008 .

[22]  P. Murias,et al.  A Composite Indicator for University Quality Assesment: The Case of Spanish Higher Education System , 2008 .

[23]  Benjamin F. Hobbs,et al.  What Can We Learn from Experiments in Multiobjective Decision Analysis? , 1986, IEEE Transactions on Systems, Man, and Cybernetics.

[24]  Peng Zhou,et al.  A non-radial DEA approach to measuring environmental performance , 2007, Eur. J. Oper. Res..

[25]  Stelios H. Zanakis,et al.  Multi-attribute decision making: A simulation comparison of select methods , 1998, Eur. J. Oper. Res..

[26]  Ching-Lai Hwang,et al.  Multiple attribute decision making : an introduction , 1995 .

[27]  Mark Gershon,et al.  Model selection in multiobjective decision making for river basin planning , 1982 .

[28]  Stefano Tarantola,et al.  Variance‐Based Methods , 2008 .

[29]  Krishna B. Misra,et al.  Handbook of Performability Engineering , 2008 .

[30]  U. Tappeiner,et al.  The Stability of Rankings Derived From Composite Indicators: Analysis of the “IL Sole 24 Ore” Quality of Life Report , 2006 .

[31]  Laurens Cherchye,et al.  An Introduction to ‘Benefit of the Doubt’ Composite Indicators , 2007 .

[32]  Dimitris K. Despotis,et al.  A reassessment of the human development index via data envelopment analysis , 2005, J. Oper. Res. Soc..

[33]  Evangelos Triantaphyllou,et al.  Multi-criteria Decision Making Methods: A Comparative Study , 2000 .

[34]  C. Lovell,et al.  One Market, One Number? A Composite Indicator Assessment of EU Internal Market Dynamics , 2005 .

[35]  A. Saltelli,et al.  Composite Indicators between Analysis and Advocacy , 2007 .

[36]  B. W. Ang,et al.  A mathematical programming approach to constructing composite indicators , 2007 .

[37]  Stefano Tarantola,et al.  Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models , 2004 .

[38]  E. Choo,et al.  Interpretation of criteria weights in multicriteria decision making , 1999 .

[39]  Abraham Charnes,et al.  Measuring the efficiency of decision making units , 1978 .

[40]  Matthias Ehrgott,et al.  Multiple criteria decision analysis: state of the art surveys , 2005 .

[41]  Bryony Hoskins,et al.  Measuring Active Citizenship through the Development of a Composite Indicator , 2009 .

[42]  Laurens Cherchye,et al.  Creating composite indicators with DEA and robustness analysis: the case of the Technology Achievement Index , 2006, J. Oper. Res. Soc..

[43]  Stefano Tarantola,et al.  Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators , 2005 .

[44]  Bikas K. Sinha,et al.  On some aspects of data integration techniques with environmental applications , 2003 .