Realtime Estimation of Illumination Direction for Augmented Reality on Mobile Devices

Augmented reality simulations aims to provide realistic blending between real world and virtual objects. One of the important factors for realistic augmented reality is correct illumination simulation. Mobile augmented reality systems is one of the best options for introducing augmented reality to the mass market due to its low production cost and ubiquitousness. In mobile augmented reality systems, the ability to correctly simulate in realtime the illumination direction that matches the illumination direction of the real world is limited. Developing a mobile augmented reality systems with the ability to estimate illumination direction presents a challenge due to low computation power and dynamically changing environment. In this paper, we described a new method that we have developed for realtime illumination direction estimation for mobile augmented reality systems, using analysis of shadow produced by a reference object that doubles as a 3D augmented reality marker. The implementation of the method could estimate the direction of a single strong light source in a controlled environment with a very good degree of accuracy, with angular error averaging lower than 0.038 radians. The current implementation achieved 2.1 FPS performance in a low-end Android mobile device, produced proper estimation within 15 seconds using a uniform surface, and demonstrated scalability potential. Background An augmented reality system aims to add simulated virtual objects, for example computer-generated 3D objects, to a real world scene, and presents the combined scene to the user. This combination of the real world and virtual objects augments the user’s perception of the real world, and offer a new method of interacting with his or her environment. Augmented reality is used in multiple fields, including medical, manufacturing, entertainment and military [1]. Taken in a larger context, Milgram et al. [2] positioned augmented reality as an intermediate between a real world environment and a virtual reality environment. They also define real world environment and virtual reality environment as polar opposites in what they called a reality-virtuality continuum. Mobile Augmented Reality One important aspect of an augmented reality system is portability [1]. An augmented reality system with high portability would give the user greater degree of liberty during his or her interaction with the augmented environment. Based on the positioning of its display, two major categories of augmented reality system that have relatively high degrees of portability are head-worn augmented reality and hand-held augmented reality [3]. From these two categories, the hand-held augmented reality systems are currently considered to be the best for introducing augmented reality to the mass market due to its low production cost and ease of use [3]. The ubiquitous proliferation of smart mobile devices that could be used as hand-held augmented reality systems also helps the ease of adoption by potential users. Registration and Tracking To properly align the simulated virtual object with the real world scene and produce a consistent simulation, an augmented reality system needs to be able to detect and track its position and orientation with respect to the real world. This is typically performed using markers with known shapes or textures [4]. Augmented reality markers are typically two-dimensional markers, but some libraries provide the ability to use three-dimensional markers. Recent development using feature extraction and recognition provides a new markerless approach for augmented reality tracking, although this approach is still under development [5]. Reproduction Fidelity Tracking and alignment is required to provide overlay information. However, if the objective is to make the virtual object appear integrated into the real world scene, we must embed the virtual objects in the real world [2]. To improve the user experience the system must provide more realistic blending between the virtual objects and the real world. The degree of this blending could be represented as reproduction fidelity [2]. Low reproduction fidelity is exemplified by having a simple wireframe simulation of the virtual objects, while higher reproduction fidelity takes into account more complex factors for the simulation such as shading, texture, transparency, and illuminations, with the ultimate target of producing simulations with photorealistic fidelity. Current state of the reproduction fidelity for mobile augmented reality systems is limited to texture, shading, and transparency simulation of the rendered virtual objects. The ability to properly simulate the illumination condition of the virtual objects to match the illumination condition of the real world is still limited, mainly due to the inability to estimate the illumination conditions of the real world environment. Without this estimate, simulating the illumination condition in an augmented reality systems using abstract parameters would risk producing less realistic simulations. In order to address this issue, several methods are proposed and developed to estimate the illumination conditions and incorporate the estimate in an augmented reality system [6, 7, 8, 9, 10, 11, 12, 13, 14]. However, none of the explored methods is applied to mobile augmented reality systems, and most of them requires off-line processing that precludes real time application 20th Color and Imaging Conference Final Program and Proceedings 111 of the illumination estimation on mobile devices. In this paper, we developed a illumination estimation method that is suitable for simulating real time illumination on mobile augmented reality systems. Illumination in Augmented Reality The first issue in understanding illumination in augmented reality is understanding the types and properties of illumination sources that could be simulated in an augmented reality system. Since the simulated part of an augmented reality system is essentially a virtual reality simulation, types of illumination sources that could be simulated in an augmented reality follow the types normally found in a virtual reality. Strauss [15] defines three basic types of illumination sources in a virtual reality simulation: 1. Point illumination source. The illumination originates from a single point in space (e.g. a light bulb). 2. Directional illumination source. The illumination rays run parallel from a certain direction (e.g. the sun). Could be considered similar to a point illumination source, but positioned at a large or infinite distance. 3. Ambient illumination source. The illumination is spread diffusely and evenly (e.g. illumination from a lamp equipped with lampshades). Strauss [15] states that direction and color of the illumination source are two important properties necessary to simulate proper illumination effect. Moreover, experiment by Slater, Usoh, and Chrysanthou in [16] shows that the existence of shadows could assist the spatial perception of the user. We therefore focused on detecting directional and point illumination sources in order to be able to simulate realistic shadows for a mobile augmented reality system. To achieve this, we require real-time estimation of the illumination condition of the environment surrounding the mobile device. Methods for Estimating Real World Illumination Conditions We explored three different methods that could be used to estimate the real world illumination condition. The first method is using a light probe. A light probe is a spherical object coated with reflective materials that could be used to capture the surrounding illumination of an environment [17, 18]. Since the intensity of illumination sources in an environment normally spans a large dynamic range compared to the environment itself, it is common to use HDR imaging to capture the light probe images. The light probe method has been applied to augmented reality simulations to produce photorealistic virtual objects [6, 7, 8, 9, 10]. In general, the hardware setup of augmented reality simulations developed using this method is using two separate cameras [6, 8], one camera dedicated for viewing the light probe and another camera dedicated to viewing and tracking the AR marker. A one-camera setup is possible [9, 6], but necessitate the light probe to be constantly viewable in the same scene with the AR marker. An additional problem exists since the position of the light probe is dynamically changing with respect to the camera, thus introducing the need to detect and track the position of the light probe. Mounting the probe in a fixed platform is one possible solution for this problem [9]. The second method is using a fish-eye lens to directly detect and estimate the illumination sources surrounding the environment [19]. HDR imaging could be done by varying the shutter speed and aperture size of the imaging device. However, since capturing the environment in several different dynamic range requires the time of several seconds, this means that this method is not suitable for real time applications. The fish-eye lens method has been applied to augmented reality simulations [6, 20]. The third method is using analysis of shadows generated by reference objects with known geometry to estimate the direction of the illumination sources [21, 22] and, in an outdoor environment, the strength of sky irradiance [11]. Panagopoulos et al. use von Mises-Fisher distribution [21] and Markov Random Field [22] to model the direction of the illumination sources from the shadows detected in the input image. Their method and the resulting illumination models was implemented to simulate shadows of virtual objects for augmented reality, but they noted that their algorithm requires three to five minutes of processing time per image, making it unsuitable for real-time augmented reality. Madsen and Nielsen [11] use contrast obtained by comparing the brightness of the shadowed and unshadowed region of an outdo