Video-Rate Z Keying: A New Method for Merging Images

Abstract : Video-rate z keying is a new image keying method for merging real and synthetic images in real time. In visual media communication and display, it is often necessary to merge video signals from a real camera and a synthetic video produced by computer graphics. A standard technique for such a purpose is chroma keying which is used, for example, in TV weather reports. Chroma keying, however, simply puts real world objects in the foreground of the synthetic image, and cannot deal with situation where real and synthetic objects occlude each other. The z key method we present merges real and virtual world images in a more flexible way. The z key uses pixel-by-pixel depth information in the form of a depth map as a switch. For each pixel, the z key switch compares the pixel depth values of two images, and routes the color value of the foreground image that is nearer to the camera for the merged output image. The result of this pixel-by-pixel switching is that real and virtual objects can occlude each other correctly depending on their geometrical relationships. The critical capability for realizing such video-rate z keying is video-rate pixel-by-pixel depth mapping of a real scene. We have developed a video-rate stereo machine which can produce 200 x 200 depth images at video rate. With this machine, merging a real scene with a synthetic scene by means of z keying in real-time has been demonstrated; a real person walks around in a synthetic room with correct relationships with virtual objects in the room at the rate of 15 frames/sec.