Video segmentation and occlusion detection over multiple frames

Spatial segmentation of image sequences is usually performed based on motion between two frames, and then followed by tracking. Some recent approaches extend this to joint segmentation in space-time; the resulting 3-D segmentation (in x-y-t space) can be interpreted as a volume 'carved out by a moving object in the image sequence domain. We call such volumes 'object tunnels'. In this paper, we propose a new approach to occlusion analysis and characterization that is based on object tunnels. It results from the observation that object-tunnel wall for a fully visible object has different shape than that for an object undergoing occlusion or exposure. Walls of tunnels associated with moving objects have tangent planes that are, in general, non-parallel to the time axis. When an object gets occluded or exposed by a static feature, part of the object tunnel wall stops evolving freely; its spatial coordinates remain fixed (static occlusion boundary) while the temporal coordinate increases linearly (time evolution). This forces part of the wall to be comprised of lines parallel to the time axis, each line defined by a single point on the occlusion boundary. In case this boundary is a straight-line edge, the occluding part of the wall becomes planar. We propose to detect occlusions by searching for such characteristic surfaces of object tunnel walls. We formulate the problem for planar occlusion walls based on a robust distance metric, and we show experimental results for various occlusion types on synthetic and camera-acquired image sequences.