On Viewpoint Invariance for Non-Planar Scenes