Validity of Normality Assumption in CSP Research

There are many new methods for solving constraint satisfaction problems proposed in recent years. Due to their complexity, a theoretical analysis on their average-case behaviours seems to be very difficult. Researchers tend to adopt an empirical approach to evaluate constraint satisfaction techniques. When empirical results are ready, statistical techniques are often employed for analysis. The question is which statistics to use. Some recent research uses parametric tests such as t-test and ANOVA. However those tests assume that the characteristic of the normal curve can be applied. In this paper, we provide evidence that the normality assumption is often not valid in the results produced by a range of constraint satisfaction algorithmheuristic combinations on random binary constraint satisfaction problems and 3-colouring problems, particularly when a problem is within the "mushy region", which are popular benchmark problems for evaluating CSP methods. The failure of normality assumption highlights the need for some statistics which do not rely on the normality assumption to analyse empirical results from CSP research. We believe that non-parametric techniques could be the right tools for that purpose.

[1]  Eugene C. Freuder,et al.  Selective relaxation for constraint satisfaction problems , 1991, [Proceedings] Third International Conference on Tools for Artificial Intelligence - TAI 91.

[2]  Patrick Prosser,et al.  Forward Checking with Backmarking , 1995, Constraint Processing, Selected Papers.

[3]  Peter van Beek,et al.  A Theoretical Evaluation of Selected Backtracking Algorithms , 1995, IJCAI.

[4]  Barbara M. Smith,et al.  The Phase Transition and the Mushy Region in Constraint Satisfaction Problems , 1994, ECAI.

[5]  Peter C. Cheeseman,et al.  Where the Really Hard Problems Are , 1991, IJCAI.

[6]  Rina Dechter,et al.  Look-Ahead Value Ordering for Constraint Satisfaction Problems , 1995, IJCAI.

[7]  Patrick Prosser,et al.  HYBRID ALGORITHMS FOR THE CONSTRAINT SATISFACTION PROBLEM , 1993, Comput. Intell..

[8]  Robert M. Haralick,et al.  Increasing Tree Search Efficiency for Constraint Satisfaction Problems , 1979, Artif. Intell..

[9]  Edward P. K. Tsang,et al.  Foundations of constraint satisfaction , 1993, Computation in cognitive science.

[10]  James M. Crawford,et al.  Experimental Results on the Crossover Point inSatis ability , 1993 .

[11]  Daniel Br.  New Methods to Color the Vertices of a Graph , 1979 .

[12]  W. Vincent.  Statistics In Kinesiology , 1994 .

[13]  Tad Hogg,et al.  The Hardest Constraint Problems: A Double Phase Transition , 1994, Artif. Intell..

[14]  Daniel Brélaz,et al.  New methods to color the vertices of a graph , 1979, CACM.

[15]  Eugene C. Freuder.  A Sufficient Condition for Backtrack-Free Search , 1982, JACM.

[16]  Alvin C. M. Kwan,et al.  Minimal Forward Checking with Backmarking , 1995 .

[17]  Toby Walsh,et al.  Easy Problems are Sometimes Hard , 1994, Artif. Intell..