A server for automated performance analysis of benchmarking data

As part of Performance World, we describe an automation server (PAVER: http://www.gamsworld.org/performance/paver ) to help facilitate reproducible performance analysis of benchmarking data for optimization software. Although PAVER does not solve optimization problems, it automates the task of performance data analysis and visualization, taking into account various performance metrics. These include not only robustness and efficiency, but also quality of solution. This paper discusses the tools and the performance metrics used, as well as the design of the server. We give examples of performance data analysis using an instance of the COPS test case for nonlinear programming to illustrate the features of the server.

[1]  John M. Mulvey,et al.  On Reporting Computational Experiments with Mathematical Software , 1979, TOMS.

[2]  Robert E. Bixby,et al.  Solving Real-World Linear Programs: A Decade and More of Progress , 2002, Oper. Res..

[3]  Michael R. Bussieck,et al.  Quality Assurance and Global Optimization , 2002, COCOS.

[4]  Leon S. Lasdon,et al.  Intopt: an interior point algorithm for large scale nonlinear optimization , 1995 .

[5]  David Kendrick,et al.  GAMS, a user's guide , 1988, SGNM.

[6]  Nicholas I. M. Gould,et al.  A numerical comparison between the LANCELOT and MINOS packages for large-scale constrained optimization , 1997 .

[7]  Michael A. Saunders,et al.  SNOPT: An SQP Algorithm for Large-Scale Constrained Optimization , 2005, SIAM Rev..

[8]  Michael R. Bussieck,et al.  MINLPLib - A Collection of Test Models for Mixed-Integer Nonlinear Programming , 2003, INFORMS J. Comput..

[9]  Brian W. Kernighan,et al.  AMPL: A Modeling Language for Mathematical Programming , 1993 .

[10]  Jorge Nocedal,et al.  A Numerical Study of the Limited Memory BFGS Method and the Truncated-Newton Method for Large Scale Optimization , 1991, SIAM J. Optim..

[11]  Martin W. P. Savelsbergh,et al.  An Updated Mixed Integer Programming Library: MIPLIB 3.0 , 1998 .

[12]  Hande Y. Benson,et al.  INTERIOR-POINT METHODS FOR NONCONVEX NONLINEAR PROGRAMMING: JAMMING AND COMPARATIVE NUMERICAL TESTING , 2000 .

[13]  Michael C. Ferris,et al.  A Comparison of Large Scale Mixed Complementarity Problem Solvers , 1997, Comput. Optim. Appl..

[14]  Nikolaos V. Sahinidis,et al.  Convexification and Global Optimization in Continuous and Mixed-Integer Nonlinear Programming , 2002 .

[15]  Jorge J. Moré,et al.  Digital Object Identifier (DOI) 10.1007/s101070100263 , 2001 .

[16]  William Gropp,et al.  Optimization environments and the NEOS server , 1997 .

[17]  Todd Munson,et al.  Benchmarking optimization software with COPS. , 2001 .

[18]  Elizabeth D. Dolan,et al.  NEOS Server 4.0 Administrative Guide , 2001, ArXiv.

[19]  Todd S. Munson,et al.  Optimality Measures for Performance Profiles , 2006, SIAM J. Optim..

[20]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[21]  John C. Plummer,et al.  A Multistart Scatter Search Heuristic for Smooth NLP and MINLP Problems , 2005 .

[22]  Sven Leyffer,et al.  User manual for filterSQP , 1998 .

[23]  R. Vanderbei.  LOQO user's manual — version 3.10 , 1999 .

[24]  H. Mittelmann.  Benchmarking interior point Lp/Qp solvers , 1999 .

[25]  Hans D. Mittelmann,et al.  An independent benchmarking of SDP and SOCP solvers , 2003, Math. Program..

[26]  Jorge J. Moré,et al.  The NEOS Server , 1998 .