Fusing online and offline information for stable 3D tracking in real-time

We propose an efficient online real-time solution for single-camera 3D tracking of rigid objects that can handle large camera displacements, drastic aspect changes, and partial occlusions. While the offline camera registration problem can be considered as essentially solved, robust online tracking remains an open issue because many real-time algorithms described in the literature still lack robustness and are prone to drift and jitter. To solve these problems, we have developed a robust approach to 3D feature matching that can handle wide-baseline matching: our method merges the information from preceding frames in traditional recursive tracking fashion with that provided by a very limited number of keyframes created during an offline stage. This combination results in a system that does not suffer from the above difficulties and can deal with drastic aspect changes. We use augmented reality applications to demonstrate its behavior because they are particularly demanding in terms of tracking performance.