Discovering visual features for recognizing user's sentiments in social images

Recently, with the increasing of users and activities in social network service, an image sentiment analysis has been an important keyword for psychological study and commercial marketing. To recognize accurately user's sentiments of the image, it is essential to identify discriminative visual features and then conduct analysis based on observed features. In this paper, we propose two hand-designed features: color composition and SIFT-based shape descriptor. These features are designed based on psychological study and experiments. First, two visual dictionaries are built by Kobayashi's color image scale and Hierarchical clustering. Next, color compositions and SIFT-based descriptors are extracted from image. Then, the set of extracted features are separately transformed into a histogram representation by calculating the occurrences of the respective feature assigned to each visual word in the dictionary. To verify the effectiveness of the proposed features, we apply them to image sentiment analysis for predicting user's polarity and affects. The recognition results were compared with man-labeled ground truth and then showed the performance with an F1-measure results of above 93%.