Appearance modeling under geometric context for object tracking and recognition Dissertation Proposal

In computer vision literature, most object recognition algorithms are based on shape models or the appearance inside a rectangular box. However, only recently, appearance modeling inside arbitrary contours has been undertaken. It becomes particularly important when we need to fingerprint the appearance of humans with articulated motion or vehicles under arbitrary poses. Essentially we need a framework that models the appearance under certain shape context or combines appearance and shape information. In this paper, we propose a unifying framework based on a general definition of Geometric Transform (GeT) for modeling appearance under geometric context. Geometric context refers to the geometric prior information. It can be based on a model, inference from the contour, or prior knowledge of the motion etc. The GeT incorporates the geometric context by applying designed functionals over certain geometric sets of an image. We show that linear and non-linear image transformations, Radon Transform, and Trace Transform are special cases of GeT. We also propose some innovative ways of generating the geometric sets, such as from the contour boundary, or from skeletons of the shape, rather than simply from some feature points as in Active Appearance Model (AAM) [3]. In the case when we only use sets of straight lines as in Radon Transform, we propose a multi-resolution representation that combines both shape and appearance information. We test our methods by classifying the appearance of pedestrians according to their appearance. An example of registering a 3D vehicle model for 3D tracking is also illustrated. In the application of fingerprinting vehicles in the video, we often need to extract a 3D model of the vehicle before modeling its appearance across different views. Therefore, we propose a factorization approach for structure from planar motion that can be used to reconstruct 3D model of vehicle in surveillance videos. Compared with [20] for general motion, our work has three major differences: a different measurement matrix specialized for planar motion is formed. The measurement matrix has a rank of at most 3, instead of 4; the measurement matrix needs similar scalings, but estimation of fundamental matrices or epipoles is not needed; we have an Euclidean reconstruction instead of a projective reconstruction. The camera is not required to be calibrated. A simple semi-automatic calibration method using vanishing points and lines is sufficient. Experimental results show that the algorithm is accurate and fairly robust to noise and inaccurate calibration.