Constrained self-calibration for augmented reality registration