Assessing Algorithmic Performance by Frontier Analysis: A DEA Approach

InCombinatorialOptimizationtheevaluationofheuristicalgorithmsoftenrequirestheconsideration ofmultipleperformancemetricsthatarerelevantfortheapplicationofinterest.Traditionalempirical analysisofalgorithmsreliesonevaluatingindividualperformancemetricswheretheoverallassessment isconductedbysubjectivejudgmentwithoutthesupportofrigorousscientificmethods.Theauthors proposeananalyticalapproachbasedondataenvelopmentanalysis(DEA)torankalgorithmsbytheir relativeefficiencyscoresthatresultfromcombiningmultipleperformancemetrics.Toevaluatetheir approach,theyperformapilotstudyexaminingtherelativeperformanceoftensurrogateconstraint algorithmsfordifferentclassesofthesetcoveringproblem.TheanalysisshowstheirDEA-based approachishighlyeffective,establishingacleardifferencebetweenthealgorithms’performances atappropriatestatisticalsignificancelevels,andinconsequenceprovidingusefulinsightsintothe selectionofalgorithmstoaddresseachclassofinstances.Theirapproachisgeneralandcanbeused withalltypesofperformancemetricsandalgorithms. KEywoRDS Algorithm Performance, Algorithm Selection, Constraint Normalization, Data Envelopment Analysis, Metaheuristics, Performance Metrics, Set Covering Problem, Surrogate Constraints

[1]  Marjan Mernik,et al.  Is a comparison of results meaningful from the inexact replications of computational experiments? , 2016, Soft Comput..

[2]  Éric D. Taillard,et al.  Few statistical tests for proportions comparison , 2008, Eur. J. Oper. Res..

[3]  D. Mckillop,et al.  Best-practice benchmarking in UK higher education: new nonparametric approaches using financial ratios and profit efficiency methodologies , 2009 .

[4]  Gholam Reza Jahanshahloo,et al.  Finding the Most Preferred Decision-Making Unit in Data Envelopment Analysis , 2016, Adv. Oper. Res..

[5]  Bruce Hollingsworth,et al.  Use of ratios in data envelopment analysis , 2003 .

[6]  A. S. Fernández-Castro,et al.  Towards a general non-parametric model of corporate performance , 1994 .

[7]  Jian-Bo Yang,et al.  Using interactive multiobjective methods to solve DEA problems with value judgements , 2009, Comput. Oper. Res..

[8]  Antonio Sassano,et al.  A Lagrangian-based heuristic for large-scale set covering problems , 1998, Math. Program..

[9]  Marco Laumanns,et al.  Performance assessment of multiobjective optimizers: an analysis and review , 2003, IEEE Trans. Evol. Comput..

[10]  Andrew C. Ho,et al.  Set covering algorithms using cutting planes, heuristics, and subgradient optimization: A computational study , 1980 .

[11]  José Humberto Ablanedo-Rosas,et al.  Surrogate constraint normalization for the set covering problem , 2010, Eur. J. Oper. Res..

[12]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[13]  Lawrence M. Seiford,et al.  Data envelopment analysis (DEA) - Thirty years on , 2009, Eur. J. Oper. Res..

[14]  F. Glover.  A Multiphase-Dual Algorithm for the Zero-One Integer Programming Problem , 1965 .

[15]  J. Beasley.  An algorithm for set covering problem , 1987 .

[16]  Fred W. Glover,et al.  Contributions of Professor William W. Cooper in Operations Research and Management Science , 2009, Eur. J. Oper. Res..

[17]  T. Grossman,et al.  Computational Experience with Approximation Algorithms for the Set Covering Problem , 1994 .

[18]  A. Charnes,et al.  Some Models for Estimating Technical and Scale Inefficiencies in Data Envelopment Analysis , 1984 .

[19]  H. A. Al-Tamimi,et al.  Evaluating the operational and profitability efficiency of a UAE-based commercial bank , 2007 .

[20]  William W. Cooper,et al.  Origins, uses of, and relations between goal programming and data envelopment analysis , 2005 .

[21]  J. Beasley.  A lagrangian heuristic for set‐covering problems , 1990 .

[22]  Fred W. Glover,et al.  Surrogate constraint analysis-new heuristics and learning schemes for satisfiability problems , 1996, Satisfiability Problem: Theory and Applications.

[23]  P. Andersen,et al.  A procedure for ranking efficient units in data envelopment analysis , 1993 .

[24]  C. K. Y. Lin,et al.  Multi-objective metaheuristics for a location-routing problem with multiple use of vehicles on real data and simulated data , 2006, Eur. J. Oper. Res..

[25]  Ali Emrouznejad,et al.  Evaluation of research in efficiency and productivity: A survey and analysis of the first 30 years , 2008 .

[26]  Marjan Mernik,et al.  Replication and comparison of computational experiments in applied evolutionary computing: Common pitfalls and guidelines to avoid them , 2014, Appl. Soft Comput..

[27]  Rajiv D. Banker,et al.  Hypothesis tests using data envelopment analysis , 1996 .

[28]  Francisco Herrera,et al.  A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms , 2011, Swarm Evol. Comput..

[29]  Abraham Charnes,et al.  Measuring the efficiency of decision making units , 1978 .

[30]  Jian-Bo Yang,et al.  Integrating DEA-oriented performance assessment and target setting using interactive MOLP methods , 2009, Eur. J. Oper. Res..

[31]  John N. Hooker,et al.  Needed: An Empirical Science of Algorithms , 1994, Oper. Res..