Can Computers Learn from the Aesthetic Wisdom of the Crowd?

The social media revolution has led to an abundance of image and video data on the Internet. Since this data is typically annotated, rated, or commented upon by large communities, it provides new opportunities and challenges for computer vision. Social networking and content sharing sites seem to hold the key to the integration of context and semantics into image analysis. In this paper, we explore the use of social media in this regard. We present empirical results obtained on a set of 127,593 images with 3,741,176 tag assignments that were harvested from Flickr, a photo sharing site. We report on how users tag and rate photos and present an approach towards automatically recognizing the aesthetic appeal of images using confidence-based classifiers to alleviate effects due to ambiguously labeled data. Our results indicate that user generated content allows for learning about aesthetic appeal. In particular, established low-level image features seem to enable the recognition of beauty. A reliable recognition of unseemliness, on the other hand, appears to require more elaborate high-level analysis.

[1]  Krishna P. Gummadi,et al.  Media Landscape in Twitter: A World of New Conventions and Political Diversity , 2011, ICWSM.

[2]  Bernard J. Jansen,et al.  Classifying ecommerce information sharing behaviour by youths on social networking sites , 2011, J. Inf. Sci..

[3]  Christian Bauckhage,et al.  Image retrieval and Web 2.0 — where can we go from here? , 2008, 2008 15th IEEE International Conference on Image Processing.

[4]  W. Chu.  Studying Aesthetics in Photographic Images Using a Computational Approach , 2013 .

[5]  Jacques Jérôme Pierre Maquet.  The Aesthetic Experience: An Anthropologist Looks at the Visual Arts , 1986 .

[6]  Daniel M. Romero,et al.  Influence and passivity in social media , 2010, ECML/PKDD.

[7]  Jure Leskovec,et al.  The dynamics of viral marketing , 2005, EC '06.

[8]  Christian Bauckhage,et al.  Insights into Internet Memes , 2011, ICWSM.

[9]  Christian Wallraven,et al.  Image Statistics for Clustering Paintings According to their Visual Appearance , 2009, CAe.

[10]  Alberto Maria Segre,et al.  The Use of Twitter to Track Levels of Disease Activity and Public Concern in the U.S. during the Influenza A H1N1 Pandemic , 2011, PloS one.

[11]  Johanna D. Moore,et al.  Twitter Sentiment Analysis: The Good the Bad and the OMG! , 2011, ICWSM.

[12]  Tansu Alpcan,et al.  An unsupervised hierarchical approach to document categorization , 2007 .

[13]  Jacob Ratkiewicz,et al.  Political Polarization on Twitter , 2011, ICWSM.

[14]  James A. Hendler,et al.  Web science: an interdisciplinary approach to understanding the web , 2008, CACM.

[15]  Aapo Hyvärinen,et al.  Natural Image Statistics - A Probabilistic Approach to Early Computational Vision , 2009, Computational Imaging and Vision.

[16]  Jacob Ratkiewicz,et al.  Predicting the Political Alignment of Twitter Users , 2011, 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing.

[17]  Lars Backstrom,et al.  Structural diversity in social contagion , 2012, Proceedings of the National Academy of Sciences.

[18]  H. Leder,et al.  A model of aesthetic appreciation and aesthetic judgments. , 2004, British journal of psychology.

[19]  Gabriele Peters,et al.  Aesthetic Primitives of Images for Visualization , 2007, 2007 11th International Conference Information Visualization (IV '07).

[20]  Steffen Staab,et al.  ATT: analyzing temporal dynamics of topics and authors in social media , 2011, WebSci '11.

[21]  Joachim Denzler,et al.  1/f2 Characteristics and Isotropy in the Fourier Power Spectra of Visual Art, Cartoons, Comics, Mangas, and Different Categories of Photographs , 2010, PloS one.

[22]  Alan F. Smeaton,et al.  Using Twitter to Detect and Tag Important Events in Live Sports , 2011 .

[23]  Christian Bauckhage,et al.  I tag, you tag: translating tags for advanced user models , 2010, WSDM '10.

[24]  Allen B. Downey,et al.  Lognormal and Pareto distributions in the Internet , 2005, Comput. Commun..

[25]  Elaine Peterson,et al.  Beneath the Metadata: Some Philosophical Problems with Folksonomy , 2006 .

[26]  Antonio Torralba,et al.  Building the gist of a scene: the role of global image features in recognition. , 2006, Progress in brain research.

[27]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[28]  Christian Bauckhage,et al.  Archetypal Images in Large Photo Collections , 2009, 2009 IEEE International Conference on Semantic Computing.

[29]  Johan Bollen,et al.  Modeling Public Mood and Emotion: Twitter Sentiment and Socio-Economic Phenomena , 2009, ICWSM.

[30]  C. Redies,et al.  Artists portray human faces with the Fourier statistics of complex natural scenes , 2007, Network.

[31]  Jiebo Luo,et al.  Aesthetics and Emotions in Images , 2011, IEEE Signal Processing Magazine.

[32]  Christian Bauckhage,et al.  The slashdot zoo: mining a social network with negative edges , 2009, WWW.

[33]  Tony Hey,et al.  The Fourth Paradigm: Data-Intensive Scientific Discovery , 2009 .

[34]  Mark E. J. Newman,et al.  Power-Law Distributions in Empirical Data , 2007, SIAM Rev..

[35]  Toshikazu Kato,et al.  Database architecture for content-based image retrieval , 1992, Electronic Imaging.

[36]  Jure Leskovec,et al.  Meme-tracking and the dynamics of the news cycle , 2009, KDD.

[37]  Antonio Torralba,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence 1 80 Million Tiny Images: a Large Dataset for Non-parametric Object and Scene Recognition , 2022 .

[38]  Alan F. Smeaton,et al.  Using Twitter to Detect and Tag Important Events in Sports Media , 2011, ICWSM.

[39]  Jingrui He,et al.  Classification of Digital Photos Taken by Photographers or Home Users , 2004, PCM.

[40]  Michael Mitzenmacher,et al.  A Brief History of Generative Models for Power Law and Lognormal Distributions , 2004, Internet Math..

[41]  Kok-Lim Low,et al.  Saliency-enhanced image aesthetics class prediction , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[42]  Mark Dredze,et al.  You Are What You Tweet: Analyzing Twitter for Public Health , 2011, ICWSM.

[43]  Nathan Moroney,et al.  Low level features for image appeal measurement , 2009, Electronic Imaging.

[44]  Charles A. Hill,et al.  Defining visual rhetorics , 2004 .

[45]  Fang Wu,et al.  Novelty and collective attention , 2007, Proceedings of the National Academy of Sciences.

[46]  Christian Bauckhage,et al.  Detecting Trends in Social Bookmarking Systems: A del.icio.us Endeavor , 2010, Int. J. Data Warehous. Min..

[47]  R. L. Solso.  Cognition and the visual arts , 1994 .

[48]  Allan Shields.  The Aesthetic Experience: An Anthropologist Looks at the Visual Arts by Jacques Maquet (review) , 1987 .

[49]  Duncan J. Watts,et al.  Who says what to whom on twitter , 2011, WWW.

[50]  Peter Norvig,et al.  The Unreasonable Effectiveness of Data , 2009, IEEE Intelligent Systems.

[51]  Stefanie Nowak,et al.  Content-based mood classification for photos and music: a generic multi-modal classification framework and evaluation approach , 2008, MIR '08.

[52]  Sethuraman Panchanathan,et al.  Indexing natural images for retrieval based on Kansei factors , 2004, IS&T/SPIE Electronic Imaging.

[53]  W. Niblack,et al.  Image Storage and Retrieval Systems , 1992 .

[54]  Harold J. McWhinnie,et al.  Psychology and the Visual Arts , 1970 .

[55]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[56]  Jure Leskovec,et al.  Discovering value from community activity on focused question answering sites: a case study of stack overflow , 2012, KDD.

[57]  Jacques Jérôme Pierre Maquet.  The Aesthetic Experience: An Anthropologist Looks at the Visual Arts , 1989 .

[58]  Bu-Sung Lee,et al.  Event Detection in Twitter , 2011, ICWSM.

[59]  Jon Kleinberg,et al.  Differences in the mechanics of information diffusion across topics: idioms, political hashtags, and complex contagion on twitter , 2011, WWW.

[60]  Christian Borgs,et al.  We know who you followed last summer: inferring social link creation times in twitter , 2011, WWW.