Vision-Based Object Registration for Real-Time Image Overlay

This paper presents a computer vision based technique for object registration, real-time tracking and image overlay. The capability can be used to superimpose registered medical images such as those from CT or MRI onto a video image of a patient body. Real-time object registration enables an image to be overlaid consistently onto objects even while the objects and cameras viewing it are moving in three dimension. Reliable real-time object registration is carried out by a sequential process of feature detection in the image, correspondence of those features in the model, and object pose calculation. Image overlay is the projection of models, models to be superimposed, onto image planes with object pose. Feature detection is executed by computing normalized correlation to reference images at every point in the small search area. The search area is updated every cycle and its repetition in sequential frames realizes feature tracking. The change of appearance of feature points due to view change is compensated by skewing reference images using the object pose information computed in every cycle during tracking. In feature correspondence, successfully tracked feature points are chosen not only by normalized correlation values , but also by computing variations of geometric invariants from initial values. In the case where feature points do not have simple textures, geometric relationship and constraints between feature points are effective for check of tracking results. " Five Coplanar Points " , which are one of major projective invariants, is used in this paper. After the feature correspondence, object position and orientation is computed from those feature positions in the image. In the case where we have object-centered coordinates of objects, the object pose can be calculated from a monocular image and the problem is formulated as an inverse problem to solve non-linear relationship between object pose and feature positions in the image. This problem can be solved by recursive methods such as Newton's method. Two experimental results to superimpose registered model data are shown. The first example is the tracking of a PC and image overly of the image of an I/O board. The second one is the tracking of a phantom leg with some marks on it and the overlay of a bone model on the view of the leg. These problems are implemented on multiple digital signal processors system with low latency vision hardware. Real-time tracking and image overlay is carried out at frame rate (30 Hz) …