Joint localization and target tracking with a monocular camera

Abstract Localization capabilities are necessary for autonomous robots that need to keep track of their position with respect to a surrounding environment. A pursuit robot is an autonomous robot that tracks and pursues a moving target, requiring accurate localization relative to the target’s position and obstacles in the local environment. Small unmanned ground vehicles (SUGVs) equipped with a monocular camera and wheel encoders could act as effective pursuit robots, but the noisy 2D target position and size estimates from the monocular camera will in turn lead to overly noisy 3D target pose estimates. One possible approach to relative localization for pursuit robots is, rather than simply tracking and estimating a relative robot–target position in each frame, joint localization, in which the purser and target are both localized with respect to a common reference frame. In this paper, we propose a novel method for joint localization of a pursuit robot and arbitrary target. The proposed method fuses the pursuit robot’s kinematics and the target’s dynamics in a joint state space model. We show that predicting and correcting pursuer and target trajectories simultaneously produces improved results compared to standard filters for estimating relative target trajectories in a 3D coordinate system. For visual tracking, we also introduce an adaptive histogram matching threshold for suspending tracking when the target is lost in a cluttered environment. When tracking is suspended, rather than traversing the entire image to search for a reappearance of the target, we only search the part of the image segmented by histogram backprojection and correctly reinitialize the tracker. The experimental results show that the joint localization method outperforms standard localization methods and that the visual tracker for pursuit robot can deal effectively with target occlusions.