Words and Pictures: Crowdsource Discovery beyond Image Semantics

Large annotated images from the Web and crowdsource, together with powerful machine learning tools, play a crucial role in rapid progress in semantic recognition of image data in recent years. However, as inferred in the saying "A Picture is Worth More Than 1,000 Words," there is much richer information than just semantic labels associated with images from the Web resources and Crowdsource Fora. Such additional information covers the rich unexploited aspects, such as visual aesthetics, emotions, sentiments, user intention, and knowledge structure. Discovering such novel dimensions of image descriptions beyond semantics will have huge impact for exciting emerging applications such as personalized search and content recommendation. But it requires rigorous research in concept definition, task formulation, data crawling, and evaluation mechanisms. In this talk, I will address these issues by sharing our experiences [1-4] in discovering beyond-semantic visual descriptions related to visual sentiment, video upload intent classification, cultural influence on visual sentiment, and finally a wiki-style video event ontology.