Position-based monocular visual servoing of an unknown target using online self-supervised learning

Visual servoing, i.e. control with visual information, is a valuable capability in many robotic applications. In particular, position based visual servoing (PBVS) estimates position information from the observed image to generate visual servo control. However, the estimation of the position of an unknown target using monocular images is still difficult due to the complexity of the image information. For the target estimation problem, we propose to integrate three complementary techniques for monocular visual servoing. First, to estimate the probability of a target’s existence, the learning model with spatial features from convolution neural network is proposed. Second, the extended Kalman filter based on epipolar geometry estimates the 3D position of the target; moreover, from this 3D position, the perception model is trained online by self-generated virtual ground-truth. Finally, visual servo control is generated, and the resulting movement helps to construct epipolar geometry. Finally. the experimental validation is performed in a challenging setting involving occlusion and target’s shape change.