Visual and Textual Sentiment Analysis of Brand-Related Social Media Pictures Using Deep Convolutional Neural Networks

Social media pictures represent a rich source of knowledge for companies to understand consumers’ opinions, as they are available in real time and at low costs and represent an active feedback which is of importance not only for companies developing products, but also to their rivals and potential consumers. In order to estimate the overall sentiment of a picture, it is essential to not only judge the sentiment of the visual elements but also to understand the meaning of the included text. This paper introduces an approach to estimate the overall sentiment of brand-related pictures from social media based on both visual and textual clues. In contrast to existing papers, we do not consider text accompanying a picture, but text embedded in a picture, which is more challenging since the text has to be detected and recognized first, before its sentiment can be identified. Based on visual and textual features extracted from two trained Deep Convolutional Neural Networks (DCNNs), the sentiment of a picture is identified by a machine learning classifier. The approach was applied and tested on a newly collected dataset, “GfK Verein Dataset” and several machine learning algorithms are compared. The experiments yield high accuracy, demonstrating the effectiveness and suitability of the proposed approach.

[1]  Razvan C. Bunescu,et al.  Sentiment analyzer: extracting sentiments about a given topic using natural language processing techniques , 2003, Third IEEE International Conference on Data Mining.

[2]  Wenyu Liu,et al.  TextBoxes: A Fast Text Detector with a Single Deep Neural Network , 2016, AAAI.

[3]  Jiebo Luo,et al.  Sentribute: image sentiment analysis from a mid-level perspective , 2013, WISDOM '13.

[4]  Roberto Pierdicca,et al.  Automatic Analysis of Eye-Tracking Data for Augmented Reality Applications: A Prospective Outlook , 2016, AVR.

[5]  Anupam Basu,et al.  An Agreement Measure for Determining Inter-Annotator Reliability of Human Judgements on Affective Text , 2008, Proceedings of the Workshop on Human Judgements in Computational Linguistics - HumanJudge '08.

[6]  Andrea Esuli,et al.  SENTIWORDNET: A Publicly Available Lexical Resource for Opinion Mining , 2006, LREC.

[7]  Emanuele Frontoni,et al.  Person Re-identification Dataset with RGB-D Camera in a Top-View Configuration , 2016, VAAM/FFER@ICPR.

[8]  Lillian Lee,et al.  Opinion Mining and Sentiment Analysis , 2008, Found. Trends Inf. Retr..

[9]  Yan Liu,et al.  What is Tumblr: a statistical overview and comparison , 2014, SKDD.

[10]  Erik Cambria,et al.  Affective Computing and Sentiment Analysis , 2016, IEEE Intelligent Systems.

[11]  Xiang Zhang,et al.  Character-level Convolutional Networks for Text Classification , 2015, NIPS.

[12]  Li-Jia Li,et al.  Visual Sentiment Prediction with Deep Convolutional Neural Networks , 2014, ArXiv.

[13]  Andrew Zisserman,et al.  Reading Text in the Wild with Convolutional Neural Networks , 2014, International Journal of Computer Vision.

[14]  Yoon Kim,et al.  Convolutional Neural Networks for Sentence Classification , 2014, EMNLP.

[15]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[16]  Richard P. Lippmann,et al.  An introduction to computing with neural nets , 1987 .

[17]  Marc'Aurelio Ranzato,et al.  Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews , 2014, ICLR.

[18]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[19]  Tao Chen,et al.  DeepSentiBank: Visual Sentiment Concept Classification with Deep Convolutional Neural Networks , 2014, ArXiv.

[20]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[21]  Irina Rish,et al.  An empirical study of the naive Bayes classifier , 2001 .

[22]  Hongfei Lin,et al.  Visual and Textual Sentiment Analysis of a Microblog Using Deep Convolutional Neural Networks , 2016, Algorithms.

[23]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[24]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[25]  Erik Cambria,et al.  The CLSA Model: A Novel Framework for Concept-Level Sentiment Analysis , 2015, CICLing.

[26]  Jiebo Luo,et al.  Joint Visual-Textual Sentiment Analysis with Deep Neural Networks , 2015, ACM Multimedia.

[27]  Roberto Pierdicca,et al.  Automatic Classification for Anti Mixup Events in Advanced Manufacturing System , 2015 .

[28]  Russ B. Altman,et al.  Missing value estimation methods for DNA microarrays , 2001, Bioinform..

[29]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[30]  Mike Thelwall,et al.  Sentiment in short strength detection informal text , 2010 .

[31]  Pushpak Bhattacharyya,et al.  Feature Specific Sentiment Analysis for Product Reviews , 2012, CICLing.

[32]  Erik Cambria,et al.  Jumping NLP Curves: A Review of Natural Language Processing Research [Review Article] , 2014, IEEE Computational Intelligence Magazine.

[33]  Carlo Strapparava,et al.  WordNet Affect: an Affective Extension of WordNet , 2004, LREC.

[34]  T. H. Bø,et al.  LSimpute: accurate estimation of missing values in microarray data with least squares methods. , 2004, Nucleic acids research.