Emotion related structures in large image databases

We introduce two large databases consisting of 750 000 and 1.2 million thumbnail-sized images, labeled with emotion-related keywords. The smaller database consists of images from Matton Images, an image provider. The larger database consists of web images that were indexed by the crawler of the image search engine Picsearch. The images in the Picsearch database belong to one of 98 emotion related categories and contain meta-data in the form of secondary keywords, the originating website and some view statistics. We use two psycho-physics related feature vectors based on the emotional impact of color combinations, the standard RGB-histogram and two SIFT-related descriptors to characterize the visual properties of the images. These features are then used in two-class classification experiments to explore the discrimination properties of emotion-related categories. The clustering software and the classifiers are available in the public domain, and the same standard configurations are used in all experiments. Our findings show that for emotional categories, descriptors based on global image statistics (global histograms) perform better than local image descriptors (bag-of-words models). This indicates that content-based indexing and retrieval using emotion-based approaches are fundamentally different from the dominant object-recognition based approaches for which SIFT-related features are the standard descriptors.

[1]  Shigenobu Kobayashi,et al.  Color Image Scale , 1992 .

[2]  Alberto Del Bimbo,et al.  Sensations and psychological effects in color image database , 1997, Proceedings of International Conference on Image Processing.

[3]  Alberto Del Bimbo,et al.  Image Retrieval by Color Semantics with Incomplete Knowledge , 1998, J. Am. Soc. Inf. Sci..

[4]  Thorsten Joachims,et al.  Making large-scale support vector machine learning practical , 1999 .

[5]  John Platt,et al.  Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods , 1999 .

[6]  Alberto Del Bimbo,et al.  Image retrieval by color semantics , 1999, Multimedia Systems.

[7]  Sung-Bae Cho,et al.  A human-oriented image retrieval system using interactive genetic algorithm , 2002, IEEE Trans. Syst. Man Cybern. Part A.

[8]  L. Ou,et al.  A study of colour emotion and colour preference. Part III: Colour preference modeling , 2004 .

[9]  L. Ou,et al.  A study of colour emotion and colour preference. Part I: Colour emotions for single colours , 2004 .

[10]  L. Ou,et al.  A study of colour emotion and colour preference. Part II: Colour emotions for two‐colour combinations , 2004 .

[11]  Wei-Ning Wang,et al.  Image emotional semantic query based on color semantic description , 2005, 2005 International Conference on Machine Learning and Cybernetics.

[12]  Hun-Woo Yoo,et al.  Visual-Based Emotional Descriptor and Feedback Mechanism for Image Retrieval , 2006, J. Inf. Sci. Eng..

[13]  Daniel Cohen-Or,et al.  Color harmonization , 2006, ACM Trans. Graph..

[14]  Shengming Jiang,et al.  Image Retrieval by Emotional Semantics: A Study of Emotional Space and Feature Extraction , 2006, SMC.

[15]  James Ze Wang,et al.  Studying Aesthetics in Photographic Images Using a Computational Approach , 2006, ECCV.

[16]  Yu Ying-lin,et al.  Image Retrieval by Emotional Semantics: A Study of Emotional Space and Feature Extraction , 2006, 2006 IEEE International Conference on Systems, Man and Cybernetics.

[17]  Seong-Yong Hong.  COLOR IMAGE SEMANTIC INFORMATION RETRIEVAL SYSTEM USING HUMAN SENSATION AND EMOTION , 2006 .

[18]  Cordelia Schmid,et al.  Coloring Local Feature Extraction , 2006, ECCV.

[19]  J. Xin,et al.  Analysis of cross‐cultural color emotion , 2007 .

[20]  Ying Liu,et al.  A survey of content-based image retrieval with high-level semantics , 2007, Pattern Recognit..

[21]  Joonwhoan Lee,et al.  Emotional Evaluation of Color Patterns Based on Rough Sets , 2007, 2007 International Symposium on Information Technology Convergence (ISITC 2007).

[22]  Hsuan-Tien Lin,et al.  A note on Platt’s probabilistic outputs for support vector machines , 2007, Machine Learning.

[23]  James Ze Wang,et al.  Learning the consensus on visual quality for next-generation image management , 2007, ACM Multimedia.

[24]  James Ze Wang,et al.  Image retrieval: Ideas, influences, and trends of the new age , 2008, CSUR.

[25]  Reiner Lenz,et al.  Color Emotions for Image Classification and Retrieval , 2008, CGIV/MCS.

[26]  James Ze Wang,et al.  Algorithmic inferencing of aesthetics and emotion in natural images: An exposition , 2008, 2008 15th IEEE International Conference on Image Processing.

[27]  Antonio Torralba,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence 1 80 Million Tiny Images: a Large Dataset for Non-parametric Object and Scene Recognition , 2022 .

[28]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[29]  Cordelia Schmid,et al.  Evaluation of GIST descriptors for web-scale image search , 2009, CIVR '09.

[30]  Reiner Lenz,et al.  Color Based Bags-of-Emotions , 2009, CAIP.

[31]  Andrea Esuli,et al.  CoPhIR: a Test Collection for Content-Based Image Retrieval , 2009, ArXiv.

[32]  Reiner Lenz,et al.  Hierarchical S(3)-Coding of RGB Histograms , 2009, VISIGRAPP.

[33]  Koen E. A. van de Sande,et al.  Evaluating Color Descriptors for Object and Scene Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.