Real-time color blending of rendered and captured video

Augmented reality involves mixing captured video with rendered elements in real-time. For augmented reality to be effective in training and simulation applications, the computer generated c omponents need to blend in well with the captured video. Straightforward compositing is not sufficient, since the chromatic content of video and rendered data may be very different such that it is immediately obvious which parts of the composited image were rendered and which were captured. We propose a simple and effective method to color-correct the computer generated imagery. The method relies on the computation of simple statistics such as mean and variance, but does so in an appropriately chosen color space - which is key to the effectiveness of our approach. By shifting and scaling the pixel data in the rendered stream to take on the mean and variance of the captured video stream, the rendered elements blend in very well. Our implementation currently reads, color-corrects and composites video and rendered streams at a rate of more than 22 frames per second for a 720x480 pixel format. Without color correction, our implementation generates around 30 frames per second, indicating that our approach comes at a reasonably small computational cost.