Dense height map estimation from oblique aerial image sequences

We are concerned with dense height map reconstruction from aerial oblique image sequences. This configuration occurs when estimating a DSM (digital surface model) of areas where flying over is not allowed or for updating an on-board DSM, for instance, in trajectory planning with obstacle avoidance. We present a complete process starting from a partially calibrated sequence and leading to an estimated height map. The calibration step consists in refining the extrinsic parameters given by on-board ego-motion sensors (GPS and inertial measurement unit, IMU) by means of interest points tracking and bundle adjustment. We then propose a dense matching process based on the minimization of a multi-view pixelwise similarity criterion combined with a discretized L1-norm or total variation (TV) regularization term. Minimization is conducted thanks to an optimal graph-cut approach. Occlusions are accounted for without additional computational cost by a modification of the similarity criterion based on a dictionary of visibility patterns. Finally, two ways of refinement of the height map are proposed. The first one uses a local similarity minimization followed by non-linear Gaussian smoothing. The second relies on a novel approach to increase the height map resolution which combines multi-view 3-D reconstruction and image super-resolution. This method is validated on various synthetic and real aerial sequences, on either side-looking or forward-looking configurations.