Accurate mapping of natural scenes radiance to cone activation space: a new image dataset

The characterization of trichromatic cameras is usually done in terms of a device-independent color space, such as the CIE 1931 XYZ space. This is indeed convenient since it allows the testing of results against colorimetric measures. We have characterized our camera to represent human cone activation by mapping the camera sensor’s (RGB) responses to human (LMS) through a polynomial transformation, which can be “customized” according to the types of scenes we want to represent. Here we present a method to test the accuracy of the camera measures and a study on how the choice of training reflectances for the polynomial may alter the results. Introduction The last decade has seen an increasing interest in the interplay between the distinctive characteristics of biological sensory systems and those of the natural environment. In the case of vision, this interest reflects the growing evidence that the statistical properties (both spatial and chromatic) of the visual environment have contributed to shape the way in which our visual system (and that of other species) function. Consequently, much research is based on the analysis of the visual environment (considering the tasks that a living organism needs to perform in order to survive and its biological constraints) with the aim of learning about the statistical regularities that the visual system may have exploited in its development. In his review work on the relationships between visual perception and the statistical properties of natural scenes, Geisler 1 points out that measuring within-domain statistics is central to testing for “efficient coding” (the hypothesis that the response characteristics of visual neurons can be predicted from the statistics of natural images plus some biological constraints). Only after knowing the probability distribution of the property considered, we can determine which is the most efficient way of coding it. To this respect it is important to point out that there is an exponential relationship between the number of samples required to estimate a probability distribution and the number of properties considered. In other words, the more complex regularities of the visual environment we want to map, the larger the number of scenes we need to gather: this is the main reason why scientists have so far concentrated on only a small group of properties which need few images to compute. Other reason has to do with technical limitations: when natural scene regularities are unrelated to the chromatic responses of the visual system (multiscale analysis, contours, etc.), the use of uncalibrated imagery is justified. However when they involve the chromatic domain, a more sophisticated approach is needed. Two techniques and methods were initially tried to measure and compute the statistical regularities of nature in the chromatic domain: (a) spectroradiometric devices which measure spectral radiance (radiance as a function of wavelength) from a small patch of image at the time, obtaining information about illuminants and reflective material properties; (b) hyperspectral cameras which measure the same from a whole image at the time but require long exposures, etc. Both these methods are impractical for gathering large databases of in-the-field imagery: spectroradiometric devices do not capture the spatial properties of natural images and hyperspectral cameras are only useful for indoor environments or when there is little change in time (long distance shots, man-made structures, landscapes, etc.). A third method has been tried more recently to reach a compromise between speed, portability and accuracy: calibrated trichromatic cameras are fast and portable but do not provide the complete spectral information necessary to fully characterize the reflectance of every patch of the image, however they are the only way to record the statistics of large samples of the visual environment to date. The latest advances in digital imaging have turned trichromatic cameras into the most common device for estimating/measuring the properties of natural scenes. Commercial digital cameras are relatively cheap and if properly calibrated they can provide photometric information for every region of the scene (i.e. a measure of the radiant power absorbed by each of the camera’s sensors, for each pixel). Calibrating a digital camera generally involves converting the image captured in the camera (also called device-dependent) color space into a reference (or device-independent) color-space. There are currently several methods to produce this color space transformation (see Martinez-Verdú et al for a more detailed explanation). One of such methods (the spectroradiometric approach) consists of obtaining the camera sensor’s response to a narrowband monochromatic stimulus which is in turn varied to span the whole spectral sensitivity range of the camera. Since the stimulus’ radiometric characteristics are known, it is possible to reconstruct the sensor’s spectral sensitivity at each wavelength (and for each color sensor). It is also necessary to measure the sensor’s output dependency on radiant power to obtain a complete picture of the camera’s response to light. Once the sensor’s responses are known, it is possible to find an approximate way of transforming the camera’s RGB values to any device-independent color space (commonly the CIE 1931 XYZ color space). A second approach to the characterization of digital cameras is based on mathematical models (mathematical approach) which estimate the camera’s matching functions from the device RGB responses to a set of (known) spectral reflectances, such as the squares of the Macbeth ColorChecker card. While the first approach is quite precise, it is seldom used because of its complexity. The last method is easier to implement but it is very vulnerable to measurement noise. Some intermediate approaches rely on assigning an estimated function to the camera’s sensors and performing a mapping of the camera’s space by means of a “training set” of RGB responses and radiometric measurements. Our approach is a mixture of the two: it consists of measuring the camera sensitivities by means of photographing a white target through a set of spectrally narrowband interference filters (spectroradiometric approach) while using a training set to “match” the theoretical camera output to a device independent space (mathematical approach). 50 ©2010 Society for Imaging Science and Technology The problem of converting to cone activation space Whatever solution is chosen for characterizing the camera output in terms of a device-independent color space, the further transformation of these values into a cone activation space is not without difficulties. Cone activation spaces are physiologically realistic alternatives to the already ubiquitous systems of specifying color adopted by the Commission Internationale de l’Eclairage (CIE), being the best established of these systems the CIE 1931. When the CIE 1931 systems was adopted, the spectral sensitivities of the actual photoreceptors in the human retina were not indisputably known and instead, a set of hypothetical primaries was adopted, based on the experiments of Guild and Wright to determine the human color matching functions. The trichromatic values XYZ of the CIE 1931 system can be understood as the photon catches of three arbitrary photoreceptors with spectral sensitivities determined by the so called , , x y z functions. These functions are approximately point-by-point linear transformations of the cone spectral sensitivities of an average human observer (in fact z is actually very close to the spectral sensitivity of human short-wavelength -or “S” cones and y was chosen to have the same shape as the standard function of luminous sensitivity or V ). Despite the CIE 1931’s popularity and some obvious advantages, a chromaticity system that is not based on human physiology (or any other physiology, as in this case) is of limited use for researching the neural properties of a visual system. To amend this situation, a number of physiologically-plausible chromatic systems have been adopted by the neuroscience community, being one of the most popular the MacLeodBoynton space. In the MacLeod-Boynton space, the axes correspond to two of the chromatic channels identified physiologically by Derrington et al 17 in the early visual system. In this space, physiologically significant loci are represented by horizontal and vertical lines. To make the situation more complicated, MacLeod-Boynton system is derived from the Smith and Pokorny human cone sensitivities, which in turn are not exact point-by-point transformations of the CIE , , x y z , but of the slightly different set of primaries calculated by Judd in 1951 (and tabulated by Vos in 1978) known as the Judd-Vos response functions. These are favored in visual science because of its better estimate of luminosity at short wavelengths. There is a formula for transforming between the chromaticity coordinates of the CIE 1931 and the Judd 1951 system but it is valid only for monochromatic lights. This means that to use the MacLeod-Boynton system or any other cone activation space derived from the Smith and Pokorny (1975) sensitivities it is necessary to know the spectroradiometric properties of the stimulus. The most straightforward way of avoiding the inconvenience of a two-part chromatic conversion of the stimulus (from device-dependent camera-RGB space to CIE 1931 XYZ and then to Smith and Pokorny LMS (L for long, M for middle and S for short wavelength) cone activation space with the consequent transformation errors, is to characterize the camera directly in terms of the later (LMS) space. This can be done if one already knows the camera’s sensor spectral sensitivities by means of finding the best transformation between the two chromatic systems. In this work we have based our analysis in the Smith and Pokorny (1975) cone responses, which are calculated at t