Learning Kalman Network: A deep monocular visual odometry for on-road driving

Abstract This paper proposes a Learning Kalman Network (LKN) based monocular visual odometry (VO), i.e. LKN-VO, for on-road driving. Most existing learning-based VO focus on ego-motion estimation by comparing the two most recent consecutive frames. By contrast, the LKN-VO incorporates a learning ego-motion estimation through the current measurement, and a discriminative state estimator through a sequence of previous measurements. Superior to the model-based monocular VO, a more accurate absolute scale can be learned by LKN without any geometric constraints. In contrast to the model-based Kalman Filter (KF), the optimal model parameters of LKN can be obtained from dynamic and deterministic outputs of the neural network without elaborate human design. LKN is a hybrid approach where we achieve the non-linearity of the observation model and the transition model though deep neural networks, and update the state following the Kalman probabilistic mechanism. In contrast to the learning-based state estimator, a sparse representation is further proposed to learn the correlations within the states from the car’s movement behaviour, thereby applying better filtering on the 6DOF trajectory for on-road driving. The experimental results show that the proposed LKN-VO outperforms both model-based and learning state-estimator-based monocular VO on the most well-cited on-road driving datasets, i.e. KITTI and Apolloscape. In addition, LKN-VO is integrated with dense 3D mapping, which can be deployed for simultaneous localization and mapping in urban environments.