High Dynamic Range from Multiple Images: Which Exposures to Combine?∗

Many computer vision algorithms rely on precise estimates of scene radiances obtained from an image. A simple way to acquire a larger dynamic range of scene radiances is by combining several exposures of the scene. The number of exposures and their values have a dramatic impact on the quality of the combined image. At this point, there exists no principled method to determine these values. Given a camera with known response function and dynamic range, we wish to find the exposures that would result in a set of images that when combined would emulate an effective camera with a desired dynamic range and a desired response function. We first prove that simple summation combines all the information in the individual exposures without loss. We select the exposures by minimizing an objective function that is based on the derivative of the response function. Using our algorithm, we demonstrate the emulation of cameras with a variety of response functions, ranging from linear to logarithmic. We verify our method on several real scenes. Our method makes it possible to construct a table of optimal exposure values. This table can be easily incorporated into a digital camera so that a photographer can emulate a wide variety of high dynamic range cameras by selecting from a menu. 1 Capturing a Flexible Dynamic Range Many computer vision algorithms require accurate estimates of scene radiance such as color constancy [9], inverse rendering [13, 1] and shape recovery [17, 8, 18]. It is difficult to capture both the wide range of radiance values real scenes produce and the subtle variations within them using a low cost digital camera. This is because any camera must assign a limited number of brightness values to the entire range of scene radi∗This work was completed with support from a National Science Foundation ITR Award (IIS-00-85864) and a grant from the Human ID Program: Flexible Imaging Over a Wide Range of Distances Award No. N000-14-00-1-0929 (a) Small and large exposures combine to capture a high dynamic range (b) Similar exposures combine to capture suble variations Figure 1: Illustration showing the impact of the choice of exposure values on which scene radiances are captured. (a) When large and small exposures are combined the resulting image has a high dynamic range, but does not capture some scene variations. (b) When similar exposure values are combined, the result includes subtle variations, but within a limited dynamic range. In both cases, a set of exposures taken with a camera results in an “effective camera.” Which exposures must we use to emulate a desired effective camera? ances. The response function of the camera determines the assignment of brightness to radiance. The response therefore determines both the camera’s sensitivity to changes in scene radiance and its dynamic range. A simple method for extending the dynamic range of a camera is to combine multiple images of a scene taken with different exposures [6, 2, 3, 10, 11, 12, 15, 16]. For example, the left of Fig. 1(a) shows a small and a large exposure, each capturing a different range of scene radiances. The illustration on the right of Fig. 1(a) shows that the result of combining the exposures includes the entire dynamic range of the scene. Note that by using these exposures values we fail to capture subtle variations in the scene, such as the shading of the ball. Once these variations are lost they can not be restored by methods that change the brightness of an image, such as the recent work on tone mapping [4, 5, 14]. In Fig. 1(b), two similar exposures combine to produce an image that captures subtle variations, but within a limited dynamic range. As a result, in both Fig. 1(a) and (b), the images on the right can be considered as the outputs of two different “effective cameras.” The number and choice of exposures determines the dynamic range and the response of each effective camera. This relationship has been ignored in the past. In this paper we explore this relationship to address the general problem of determining which exposure values to use in order to emulate an effective camera with a desired response and a desired dynamic range. Solving this problem requires us to answer the following questions: • How can we create a combined image that preserves the information from all the exposures? Previous work suggested heuristics for combining the exposures [3, 11, 12]. We prove that even without linearizing the camera, simple summation preserves all the information contained in the set of individual exposures. • What are the best exposure values to achieve a desired effective response function for the combined image? It is customary to arbitrarily choose the number of exposures and the ratio (say, 2) between consecutive exposure values [3, 10, 11, 12]. For example, when this is done with a linear real camera, the resulting combined image is relatively insensitive to changes in large radiances. This can bias vision algorithms that use derivatives of radiance. Such biases are eliminated using our algorithm, which selects the exposure values to best achieve a desired response. • How can we best achieve a desired dynamic range and effective response function from a limited number of images? It is common to combine images with consecutive exposure ratios of 2 (see [3, 11, 12]). to create a high dynamic range image. With that choice of exposure ratio, is often necessary to use 5 or more exposures to capture the full dynamic range of a scene. This is impractical when the number of exposures that can be captured is limited by the time to acquire the images, changes in the scene, or resources needed to process the images. Our algorithm determines the exposure values needed to best emulate a desired camera with a fixed number of images. Our method allows us to emulate cameras with a wide variety of response functions. For the class of linear real cameras, we present a table of optimal exposure values for emulating high dynamic range cameras with, for example, linear and logarithmic (constant contrast) responses. Such a table can be easily incorporated into a digital camera so that a photographer can select his/her desired dynamic range and camera response from a menu. In other words, a camera with fixed response and dynamic range can be turned into one that has a “flexible” dynamic range. We show several experimental results using images of real scenes that demonstrate the power of this notion of flexible dynamic range. 2 The Effective Camera When we take multiple exposures of the same scene, each exposure adds new information about the radiance values in the scene. In this section, we create an effective camera by constructing a single image which retains all the information from the individual exposures. By information we mean image brightness values which represent measurements of scene radiance. Scene radiance is proportional to image irradiance E [7]. In a digital camera, the camera response function f jumps from one image brightness value B to the next at a list of positive irradiance values (shown below the graph in Fig. 2) which we call the measured irradiance levels. An image brightness value indicates that the corresponding measured irradiance lies in the interval between two of these levels. Hence, without loss of generality, we define B as the index of the first of these two levels, EB , so that f(EB) = B. Hence, the response function is equivalent to the list of measured irradiance levels. Now, consider the measured irradiance levels using unit exposure e1 = 1 with a real non-linear camera having 4 brightness levels. These levels are shown on the bar at the bottom of Fig. 3(a). The irradiance levels for a second exposure scale by 1/e2, as shown in Fig. 3(b). We combine the measured irradiance levels from the first and the second exposures by taking the union of all the 1The value we call exposure accounts for all the attenuations of light by the optics. One can change the exposure by changing a filter on the lens, the aperture size, the integration time, or the gain. 2Note that the slope of the response function determines the density of the levels, as shown by the short line segment in Fig. 2. 3Note that the number of exposures and brightness levels are for illustration only. Our arguments hold in general.