Photographic composition classification and dominant geometric element detection for outdoor scenes

Abstract Despite the practical importance of photographic composition for improving or assessing the aesthetical quality of photographs, only a few simple composition rules have been considered for its classification. In this work, we propose novel techniques to classify photographic composition rules of outdoor scenes and detect dominant geometric elements, called composition elements, for each composition class. Specifically, we first categorize composition rules of outdoor photographs into nine classes: RoT, center, horizontal, symmetric, diagonal, curved, vertical, triangle, and pattern. Then, we develop a photographic composition classification algorithm using a convolutional neural network (CNN). To train the CNN, we construct a photographic composition database, which is publicly available. Finally, for each composition class, we propose an effective scheme to locate composition elements, i.e., bounding boxes for main subjects, leading lines, axes of symmetry, triangles, and sky regions. Extensive experimental results demonstrate that the proposed algorithm classifies composition classes reliably and detects composition elements accurately.

[1]  Chang-Su Kim,et al.  Optimized contrast enhancement for real-time image and video dehazing , 2013, J. Vis. Commun. Image Represent..

[2]  Michael Freeman,et al.  The Photographer's Eye: Composition and Design for Better Digital Photos , 2007 .

[3]  Tao Mei,et al.  Query-Dependent Aesthetic Model With Deep Learning for Photo Quality Assessment , 2015, IEEE Transactions on Multimedia.

[4]  Nanning Zheng,et al.  Learning to Detect a Salient Object , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Chang-Su Kim,et al.  Spatiotemporal Saliency Detection for Video Sequences Based on Random Walk With Restart , 2015, IEEE Transactions on Image Processing.

[6]  Daniel Cohen-Or,et al.  Optimizing Photo Composition , 2010, Comput. Graph. Forum.

[7]  Yanwen Guo,et al.  Improving Photo Composition Elegantly: Considering Image Similarity During Composition Optimization , 2012, Comput. Graph. Forum.

[8]  Alexei A. Efros,et al.  What Do the Sun and the Sky Tell Us About the Camera? , 2010, International Journal of Computer Vision.

[9]  Chul Lee,et al.  Power-Constrained Contrast Enhancement for Emissive Displays Based on Histogram Equalization , 2012, IEEE Transactions on Image Processing.

[10]  Xiaoou Tang,et al.  Image Aesthetic Assessment: An experimental survey , 2016, IEEE Signal Processing Magazine.

[11]  Zihan Zhou,et al.  Discovering Triangles in Portraits for Supporting Photographic Creation , 2017, IEEE Transactions on Multimedia.

[12]  Chul Lee,et al.  Contrast Enhancement Based on Layered Difference Representation of 2D Histograms , 2013, IEEE Transactions on Image Processing.

[13]  Shao-Yi Chien,et al.  Preference-Aware View Recommendation System for Scenic Photos Based on Bag-of-Aesthetics-Preserving Features , 2012, IEEE Transactions on Multimedia.

[14]  Jian Sun,et al.  SkyFinder: attribute-based sky image search , 2009, SIGGRAPH 2009.

[15]  Yoshua Bengio,et al.  Convolutional networks for images, speech, and time series , 1998 .

[16]  Jitendra Malik,et al.  Region-Based Convolutional Networks for Accurate Object Detection and Segmentation , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Dilip K Prasad,et al.  Illuminant estimation for color constancy: why spatial-domain methods work and the role of the color distribution. , 2014, Journal of the Optical Society of America. A, Optics, image science, and vision.

[18]  Antonio Torralba,et al.  Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope , 2001, International Journal of Computer Vision.

[19]  Chang-Su Kim,et al.  Multiscale Saliency Detection Using Random Walk With Restart , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[20]  Yi Yang,et al.  Weakly Supervised Photo Cropping , 2014, IEEE Transactions on Multimedia.

[21]  Mohan S. Kankanhalli,et al.  Context-Aware Photography Learning for Smart Mobile Devices , 2015, ACM Trans. Multim. Comput. Commun. Appl..

[22]  Bingbing Ni,et al.  Learning to Photograph: A Compositional Perspective , 2013, IEEE Transactions on Multimedia.

[23]  Mohan S. Kankanhalli,et al.  A Spring-Electric Graph Model for Socialized Group Photography , 2018, IEEE Transactions on Multimedia.

[24]  Henry Kang,et al.  Photo Aesthetics Analysis via DCNN Feature Encoding , 2017, IEEE Transactions on Multimedia.

[25]  Chih-Jen Lin,et al.  LIBLINEAR: A Library for Large Linear Classification , 2008, J. Mach. Learn. Res..

[26]  Alexei A. Efros,et al.  Automatic photo pop-up , 2005, SIGGRAPH 2005.

[27]  Rafael Grompone von Gioi,et al.  LSD: A Fast Line Segment Detector with a False Detection Control , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Xiao Liu,et al.  Probabilistic Graphlet Transfer for Photo Cropping , 2013, IEEE Transactions on Image Processing.

[29]  Miguel Oliveira,et al.  A Probabilistic Approach for Color Correction in Image Mosaicking Applications , 2015, IEEE Transactions on Image Processing.

[30]  Jiajun Bu,et al.  Where2Stand: A Human Position Recommendation System for Souvenir Photography , 2015, ACM Trans. Intell. Syst. Technol..

[31]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.