Joint Analysis of Multiple Algorithms and Performance Measures

There has been an increasing interest in the development of new methods using Pareto optimality to deal with multi-objective criteria (for example, accuracy and time complexity). Once one has developed an approach to a problem of interest, the problem is then how to compare it with the state of art. In machine learning, algorithms are typically evaluated by comparing their performance on different data sets by means of statistical tests. Standard tests used for this purpose are able to consider jointly neither performance measures nor multiple competitors at once. The aim of this paper is to resolve these issues by developing statistical procedures that are able to account for multiple competing measures at the same time and to compare multiple algorithms altogether. In particular, we develop two tests: a frequentist procedure based on the generalized likelihood ratio test and a Bayesian procedure based on a multinomial-Dirichlet conjugate model. We further extend them by discovering conditional independences among measures to reduce the number of parameters of such models, as usually the number of studied cases is very reduced in such comparisons. Data from a comparison among general purpose classifiers are used to show a practical application of our tests.

[1]  Changhe Yuan,et al.  Learning Optimal Bayesian Networks: A Shortest Path Perspective , 2013, J. Artif. Intell. Res..

[2]  Alfred O. Hero,et al.  Multi-criteria Anomaly Detection using Pareto Depth Analysis , 2011, NIPS.

[3]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[4]  Gregory F. Cooper,et al.  A Bayesian method for the induction of probabilistic networks from data , 1992, Machine Learning.

[5]  Qiang Ji,et al.  Constrained Maximum Likelihood Learning of Bayesian Networks for Facial Action Recognition , 2008, ECCV.

[6]  Alessio Benavoli,et al.  Statistical Tests for Joint Analysis of Performance Measures , 2015, AMBN@JSAI-isAI.

[7]  Qiang Ji,et al.  Efficient Structure Learning of Bayesian Networks using Constraints , 2011, J. Mach. Learn. Res..

[8]  P. Walley.  Inferences from Multinomial Data: Learning About a Bag of Marbles , 1996 .

[9]  S. S. Wilks.  The Large-Sample Distribution of the Likelihood Ratio for Testing Composite Hypotheses , 1938 .

[10]  James Cussens,et al.  Advances in Bayesian Network Learning using Integer Programming , 2013, UAI.

[11]  Philip S. Yu,et al.  Multi-Objective Multi-Label Classification , 2012, SDM.

[12]  Tomi Silander,et al.  A Simple Approach for Finding the Globally Optimal Bayesian Network Structure , 2006, UAI.

[13]  J. Rice.  Mathematical Statistics and Data Analysis , 1988 .

[14]  A. Dasgupta.  Asymptotic Theory of Statistics and Probability , 2008 .

[15]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[16]  Wray L. Buntine.  Theory Refinement on Bayesian Networks , 1991, UAI.

[17]  David Maxwell Chickering,et al.  Learning Bayesian Networks: The Combination of Knowledge and Statistical Data , 1994, Machine Learning.

[18]  Sung-Bae Cho,et al.  Multi-criterion Pareto based particle swarm optimized polynomial neural network for classification: A review and state-of-the-art , 2009, Comput. Sci. Rev..

[19]  Daoqiang Zhang,et al.  A Multiobjective Simultaneous Learning Framework for Clustering and Classification , 2010, IEEE Transactions on Neural Networks.

[20]  Qiang Ji,et al.  Properties of Bayesian Dirichlet Scores to Learn Bayesian Network Structures , 2010, AAAI.