Relocalization using virtual keyframes for online environment map construction

The acquisition of surround-view panoramas using a single hand-held or head-worn camera relies on robust real-time camera orientation tracking. In absence of robust tracking recovery methods, the complete acquisition process has to be re-started when tracking fails. This paper presents methodology for camera orientation relocalization, using virtual keyframes for online environment map construction. Instead of relying on real keyframes from incoming video, the proposed approach enables camera orientation relocalization by employing virtual keyframes which are distributed strategically within an environment map. We discuss our insights about a suitable number and distribution of virtual keyframes, as suggested by our experiments on virtual keyframe generation and orientation relocalization. After a shading correction step, we relocalize camera orientation in real-time by comparing the current camera frame to virtual keyframes. While expanding the captured environment map, we continue to simultaneously generate virtual keyframes within the completed portion of the map, as descriptors to estimate camera orientation. We implemented our camera orientation relocalizer with the help of a GPU fragment shader for real-time application, and evaluated the speed and accuracy of the proposed approach.