Towards a psychophysical evaluation of colour constancy algorithms

Computational colour constancy tries to solve the problem of recovering the illuminant of a scene from an acquired image. The most popular algorithms developed to deal with this problem use heuristics to select a unique solution from within the feasible set. Their performance has shown that there is still a long way to go to globally solve this problem as a preliminary step in computer vision. Recent works tried to insert high-level constraints to improve the selection step, whose plausibility could be evaluated according to their performance on the final visual task. To allow comparisons of constraints independently of the task, in this work we present a new performance measure, the perceptual angular error. It tries to evaluate the performance of a colour constancy algorithm according to the perceptual preferences of humans instead of the actual optimal solution. To this end, we present a new version of our “MaxName” algorithm, which aims at solving the illuminant problem using high-level information such as the number of identifiably colours on a scene. Afterwards, we show the results of a psychophysical experiment comparing three colour constancy algorithms. Our results show that in more than half of the judgements the preferred solution is not the one closest to the optimal solution. This makes us conclude that such a perceptual comparison is feasible, and we could benefit from the construction of a large colour constancy database of calibrated images, labelled according to the illuminant preferred by human observers. Introduction Colour Constancy is the ability of the human visual system to perceive a stable representation of colour despite illumination changes. Like other perceptual constancy capabilities of the visual system, colour constancy is crucial to succeed in many ecologically relevant visual tasks such as food collection, detection of predators, etc. The importance of colour constancy in biological vision is mirrored in computer vision applications, where success in a wide range of visual tasks relies on achieving a high degree illuminant invariance degree. In the last twenty years, research in computational colour constancy has tried to solve the problem of recovering the illuminant of a scene from an acquired image. Although this is a problem effortless solved by the visual system, it has been shown to be a mathematically ill-posed problem which therefore does not have a unique solution. A common computational approach to illuminant recovery (and colour constancy in general) is to produce a list of possible illuminants (feasible solutions) and then use some assumptions, based on the interactions of scene surfaces and illuminants to select the most appropriate solution among all possible illuminants. A recent extended review of computational colour constancy methods was provided by Hordley in [1]. In this review, computational algorithms were classified in five different groups according to how they approach the problem. These were (a) simple statistical methods [2], (b) neural networks [3], (c) gamut mapping [4,5], (d) probabilistic methods [6] and (e) physics-based methods [7]. Comparison studies ([8], [9]) have ranked the performance of these algorithms, which usually depend on the properties of the image dataset and the statistical measures used for the evaluation. It is generally agreed that, although some algorithms may perform well in average, they may also perform poorly for specific images. This is the reason why some authors [10] have proposed a one-to-one evaluation of the algorithms on individual images. In this way, comparisons become more independent of the chosen image dataset. However, the general conclusion is that more research should be directed towards a combination of different methods, since the performance of a method usually depends on the type of scene it deals with [11]. Recently, some interesting studies have pointed out towards this direction [12], i.e. trying to find which statistical properties of the scenes determine the best colour constancy method to use. In all these previous approaches, the evaluation of the performance of the algorithms has been based on computing the angular error between the selected solution and the actual solution that is provided by the acquisition method. Other recent proposals [13, 14] turn away from the usual approach and deal instead with multiple solutions “delegating” the selection of a unique solution to a subsequent step that depends on high-level, task-related interpretations, such as the ability to annotate the image content. In this example, the best solution would be the one giving the best semantic annotation of the image content. It is in this kind of approaches where the need for a different evaluation emerges, since the performance depends on the visual task and this can lead to an inability to compare different methods. Hence, to be able to evaluate this performance and to compare it with other high-level methods, in this paper we propose to explore a new evaluation procedure. Thus, the goal of this paper is twofold, firstly we address the problem of evaluating colour constancy methods using psychophysical data instead of the usual angular error from the optimal solution, and secondly we present a simpler version of the algorithm of Tous [13], MaxName, and its evaluation with this new approach. In the last section we discuss the results and we outline how a global dataset should be built in order to be able to achieve this perceptual evaluation of colour constancy algorithms. Perceptual performance evaluation Assuming the ill-posed nature of the problem and the difficulty of finding the optimal solution, we propose to bring the computational colour constancy algorithms towards a simulation of human colour constancy abilities by trying to match computational solutions to perceived solutions. Hence, in this paper we propose a new evaluation measurement, the Perceptual Angular Error, which is based on perceptual judgements of adequacy of a solution instead of the physical solution. This work gives a preliminary approach towards what we call a perceptual evaluation of computational colour constancy algorithms. The approach that we propose in this work does not try to give an alternative line research to the current trends, which focus on classifying scene contents to efficiently combine different methods. Here we try to complement these efforts from a different point of view we could consider as more on a top-down direction, instead of the bottom-up nature of the usual research. Differences between colour constancy algorithms essentially rely on two different aspects: (a) the assumptions made on the scene properties (such as grey-mean content of the scene, existence of a white patch, or highlights, etc.) or (b) the constraints on the recovered image (maximum global intensity as in MV C-Rule, maximum number of identifiable colour names, etc.). In other cases, assumptions and constraints are combined providing interesting approaches based on the use of most likely surfaces and illuminants (as in color by correlation or Bayesian colour constancy). From this point of view, in this work we point out that performance evaluation of different algorithms is intimately related to specific considerations of the method nature. If we want to measure the relevance of an assumption made on the scenes, we will need to evaluate on what kind of scenes the algorithm performs better, meanwhile if we are trying to evaluate the plausibility of a constraint in the selection of the best solution, then other perceptual measures could be more suited. In this work we focus on this second approach, and we propose to evaluate the adequacy of topdown constraints on CC methods by evaluating their correlation with the human colour constancy preferences, instead of their agreement with the physical solutions. We will show in the results section that human preferred solutions do not clearly match with the optimal solutions. As mentioned before, the most common performance evaluation for colour constancy algorithms consists in measuring how close their proposed solution is to the optimal solution, independently of the goal they are trying to deal with. This has been computed as