Learning to Photograph: A Compositional Perspective

In this paper, we present an intelligent photography system which can recommend the most user-favored view rectangle for arbitrary camera input, from a photographic compositional perspective. Automating this process is difficult, due to the subjectivity of human's aesthetics judgement and large variations of image contents, where heuristic compositional rules lack generality. Motivated by the recent prevalence of photo-sharing websites, e.g., Flickr.com, we develop a learning-based framework which discovers the underlying aesthetic photographic compositional structures from a large set of user-favored online sharing photographs and utilizes the implicitly shared knowledge among the professional photographers for aesthetically optimal view recommendation. In particular, we propose an Omni-Range Context method which explicitly encodes the spatial and geometric distributions of various visual elements in the photograph as well as cooccurrence characteristics of visual element pairs by using generative mixture models. Searching the optimal view rectangle is then formulated as maximum a posterior by imposing the trained prior distributions along with additional photographic constraints. The proposed system has the potential to operate in near real-time. Comprehensive user studies well demonstrate the effectiveness of the proposed framework for aesthetically optimal view recommendation.

[1]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[2]  Ariel Shamir,et al.  Cropping Scaling Seam carving Warping Multi-operator , 2009 .

[3]  Denis Simakov,et al.  Summarizing visual data using bidirectional similarity , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Bingbing Ni,et al.  Learning to photograph , 2010, ACM Multimedia.

[5]  John F. Hughes,et al.  User-guided composition effects for art-based rendering , 2001, I3D '01.

[6]  Bingbing Ni,et al.  Web image mining towards universal age estimator , 2009, ACM Multimedia.

[7]  William T. Freeman,et al.  The patch transform and its applications to image editing , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Erik Reinhard,et al.  Artistic Composition for Image Creation , 2001, Rendering Techniques.

[9]  Benjamin Martinez,et al.  Visual Forces: An Introduction to Design , 1988 .

[10]  Wei-Ying Ma,et al.  Auto cropping for digital photographs , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[11]  David A. McAllester,et al.  Object Detection with Discriminatively Trained Part Based Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Daniel Cohen-Or,et al.  Non-homogeneous Content-driven Video-retargeting , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[13]  Changhu Wang,et al.  Equip tourists with knowledge mined from travelogues , 2010, WWW '10.

[14]  Rongrong Ji,et al.  Photo assessment based on computational visual attention model , 2009, ACM Multimedia.

[15]  Ariel Shamir,et al.  Seam Carving for Content-Aware Image Resizing , 2007, ACM Trans. Graph..

[16]  Jian Shi,et al.  Image Retargeting Using Mesh Parametrization , 2009, IEEE Transactions on Multimedia.

[17]  Gustavo de Veciana,et al.  An information fidelity criterion for image quality assessment using natural scene statistics , 2005, IEEE Transactions on Image Processing.

[18]  Yoichi Sato,et al.  Sensation-based photo cropping , 2009, ACM Multimedia.

[19]  Masayuki Nakajima,et al.  Example-Based Color Transformation of Image and Video Using Basic Color Categories , 2007, IEEE Transactions on Image Processing.

[20]  David Salesin,et al.  Gaze-based interaction for semi-automatic photo cropping , 2006, CHI.

[21]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[22]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[23]  Shuicheng Yan,et al.  An HOG-LBP human detector with partial occlusion handling , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[24]  Dorin Comaniciu,et al.  Mean Shift: A Robust Approach Toward Feature Space Analysis , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Qi Tian,et al.  Visual Synset: Towards a higher-level visual representation , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Yang Song,et al.  Tour the world: Building a web-scale landmark recognition engine , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Daniel P. Huttenlocher,et al.  Efficient Graph-Based Image Segmentation , 2004, International Journal of Computer Vision.

[28]  William D. Smart,et al.  Say Cheese! Experiences with a Robot Photographer , 2004, AI Mag..

[29]  Wei Luo,et al.  Content-Based Photo Quality Assessment , 2013, IEEE Trans. Multim..

[30]  Pietro Perona,et al.  A Bayesian hierarchical model for learning natural scene categories , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[31]  Bingbing Ni,et al.  Geometric ℓp-norm feature pooling for image classification , 2011, CVPR 2011.

[32]  Stanley T. Birchfield,et al.  Spatiograms versus histograms for region-based tracking , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[33]  Xing Xie,et al.  Mining city landmarks from blogs by graph modeling , 2009, ACM Multimedia.

[34]  Paul A. Viola,et al.  Robust Real-time Object Detection , 2001 .

[35]  Yan Ke,et al.  The Design of High-Level Features for Photo Quality Assessment , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[36]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Eli Shechtman,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, ACM Trans. Graph..

[38]  Long Quan,et al.  Image deblurring with blurred/noisy image pairs , 2007, SIGGRAPH 2007.

[39]  Gabriela Csurka,et al.  Assessing the aesthetic quality of photographs using generic image descriptors , 2011, 2011 International Conference on Computer Vision.

[40]  Ariel Shamir,et al.  Improved seam carving for video retargeting , 2008, SIGGRAPH 2008.

[41]  Daniel Cohen-Or,et al.  Optimizing Photo Composition , 2010, Comput. Graph. Forum.

[42]  O. Sorkine,et al.  Optimized scale-and-stretch for image resizing , 2008, SIGGRAPH 2008.

[43]  Ying He,et al.  On the Transfer of Painting Style to Photographic Images through Attention to Colour Contrast , 2010, 2010 Fourth Pacific-Rim Symposium on Image and Video Technology.

[44]  Gang Hua,et al.  Face Re-Lighting from a Single Image under Harsh Lighting Conditions , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[45]  Bingbing Ni,et al.  Contextualizing histogram , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[46]  Ramesh Raskar,et al.  Automatic image retargeting , 2004, SIGGRAPH '04.

[47]  Xiaoou Tang,et al.  Photo and Video Quality Evaluation: Focusing on the Subject , 2008, ECCV.

[48]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[49]  Robert M. Haralick,et al.  Textural Features for Image Classification , 1973, IEEE Trans. Syst. Man Cybern..

[50]  Steven K. Feiner,et al.  Evaluation of visual balance for automated layout , 2004, IUI '04.

[51]  Tao Wang,et al.  One step beyond histograms: Image representation using Markov stationary features , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[52]  Bingbing Ni,et al.  Web Image and Video Mining Towards Universal and Robust Age Estimator , 2011, IEEE Transactions on Multimedia.

[53]  Liqing Zhang,et al.  Saliency Detection: A Spectral Residual Approach , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.