Auto-rectification of user photos

The image auto rectification project at Google aims to create a pleasanter version of user photos by correcting the small, involuntary camera rotations (roll / pitch/ yaw) that often occur in non-professional photographs. Our system takes the image closer to the fronto-parallel view by performing an affine rectification on the image that restores parallelism of lines that are parallel in the fronto-parallel image view. This partially corrects perspective distortions, but falls short of full metric rectification which also restores angles between lines. On the other hand the 2D homography for our rectification can be computed from only two (as opposed to three) estimated vanishing points, allowing us to fire upon many more images. A new RANSAC based approach to vanishing point estimation has been developed. The main strength of our vanishing point detector is that it is line-less, thereby avoiding the hard, binary (line/no-line) upstream decisions that cause traditional algorithm to ignore much supporting evidence and/or admit noisy evidence for vanishing points. A robust RANSAC based technique for detecting horizon lines in an image is also proposed for analyzing correctness of the estimated rectification. We post-multiply our affine rectification homography with a 2D rotation which aligns the closer vanishing point with the image Y axis.

[1]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[2]  Stephen T. Barnard,et al.  Interpreting Perspective Image , 1983, Artif. Intell..

[3]  Andrew Zisserman,et al.  Combining scene and auto-calibration constraints , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[4]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[5]  Carsten Rother,et al.  A New Approach for Vanishing Point Detection in Architectural Environments , 2000, BMVC.

[6]  Seth J. Teller,et al.  Automatic recovery of relative camera rotations for urban scenes , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[7]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[8]  Andrew Zisserman,et al.  Metric rectification for perspective images of planes , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[9]  Seungyong Lee,et al.  Automatic upright adjustment of photographs , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Andrew C. Gallagher.  Using vanishing points to correct camera rotation in images , 2005, The 2nd Canadian Conference on Computer and Robot Vision (CRV'05).