Expert Panel Opinion and Global Sensitivity Analysis for Composite Indicators

Composite indicators aggregate multi-dimensional processes into simplified concepts often aiming at underpinning the development of data-driven narratives for policy consumption. Due to methodological issues, doubts are often raised about the robustness of the composite indicators and the significance of the associated policy messages. In this paper we use expert panel information (derived from budget allocation and analytic hierarchy process) on the relative importance of the underlying indicators included in a composite indicator and run in tandem uncertainty and sensitivity analysis to gain useful insights during the process of composite indicators building. We discuss the extent to which variance-based sensitivity analysis may increase transparency or make policy inference more defensible by using the United Nation’s Technology Achievement Index as an illustration.

[1]  I. Sobol.  On the distribution of points in a cube and the approximate evaluation of integrals , 1967 .

[2]  I. Sobol.  Uniformly distributed sequences with an additional uniform property , 1976 .

[3]  Edward E. Leamer,et al.  Specification Searches: Ad Hoc Inference with Nonexperimental Data , 1980 .

[4]  Peter E. Kennedy.  A Guide to Econometrics , 1979 .

[5]  R. W. Saaty,et al.  The analytic hierarchy process—what it is and how it is used , 1987 .

[6]  Jerome R. Ravetz,et al.  Uncertainty and Quality in Science for Policy , 1990 .

[7]  David R. Cox,et al.  Quality‐Of‐Life Assessment: Can We Keep it Simple? , 1992 .

[8]  N Oreskes,et al.  Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences , 1994, Science.

[9]  A. Saltelli,et al.  Importance measures in global sensitivity analysis of nonlinear models , 1996 .

[10]  Bedřich Moldan,et al.  Sustainability indicators : a report on the project on indicators of sustainable development , 1997 .

[11]  Clair Gough,et al.  Integrated Assessment: an emerging methodology for complex issues , 1998 .

[12]  Stefano Tarantola,et al.  Sensitivity Analysis as an Ingredient of Modeling , 2000 .

[13]  A. Saltelli,et al.  Sensitivity Anaysis as an Ingredient of Modeling , 2000 .

[14]  J. Church.  Human Development Report , 2001 .

[15]  Saltelli Andrea,et al.  Statistical Techniques and Participatory Approaches for the Compilation of the European Internal Market Index 1992-2001 , 2002 .

[16]  S. Tarantola,et al.  State-of-the-art Report on Current Methodologies and Practices for Composite Indicator Development , 2002 .

[17]  A. Saltelli,et al.  Making best use of model evaluations to compute sensitivity indices , 2002 .

[18]  A. Saltelli,et al.  On the Relative Importance of Input Factors in Mathematical Models , 2002 .

[19]  Elsie M. Sunderland,et al.  Draft Guidance on the Development, Evaluation, and Application of Regulatory Environmental Models , 2003 .

[20]  Michael Freudenberg,et al.  Composite Indicators of Country Performance: A Critical Assessment , 2003 .

[21]  M. Bohanec,et al.  The Analytic Hierarchy Process , 2004 .

[22]  Stefano Tarantola,et al.  Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models , 2004 .

[23]  A. Sharpe.  Literature Review of Frameworks for Macro-indicators , 2004 .

[24]  Giuseppe Munda,et al.  Social multi-criteria evaluation: Methodological foundations and operational consequences , 2004, Eur. J. Oper. Res..

[25]  Liliana B. Andonova,et al.  2005 Environmental Sustainability Index Benchmarking National Environmental Stewardship Appendix A Methodology , 2005 .

[26]  Stefano Tarantola,et al.  Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators , 2005 .

[27]  Stefano Tarantola,et al.  Tools for Composite Indicators Building , 2005 .

[28]  Giuseppe Munda,et al.  Non-Compensatory Composite Indicators for Ranking Countries : A Defensible Setting , 2006 .

[29]  A. Saltelli,et al.  Pilot 2006 Environmental Performance Index , 2006 .

[30]  P. Conceição,et al.  The New Public Finance: Responding to Global Challenges , 2006 .

[31]  C. Lovell,et al.  One Market, One Number? A Composite Indicator Assessment of EU Internal Market Dynamics , 2005 .

[32]  Fulvia Pennoni,et al.  Comparative Analysis of Alcohol Control Policies in 30 Countries , 2007, PLoS medicine.

[33]  A. Saltelli,et al.  Composite Indicators between Analysis and Advocacy , 2007 .

[34]  David Hall.  The New Public Finance – Responding to Global Challenges , 2008 .

[35]  Stefano Tarantola,et al.  Variance‐Based Methods , 2008 .