Through The Eyes of A Poet: Classical Poetry Recommendation with Visual Input on Social Media

With the increasing popularity of portable devices with cameras (e.g., smartphones and tablets) and ubiquitous Internet connectivity, travelers can share their instant experience during the travel by posting photos they took to social media platforms. In this paper, we present a new image-driven poetry recommender system that takes a traveler's photo as input and recommends classical poems that can enrich the photo with aesthetically pleasing quotes from the poems. Three critical challenges exist to solve this new problem: i) how to extract the implicit artistic conception embedded in both poems and images? ii) How to identify the salient objects in the image without knowing the creator's intent? iii) How to accommodate the diverse user perceptions of the image and make a diversified poetry recommendation? The proposed iPoemRec system jointly addresses the above challenges by developing heterogeneous information network and neural embedding techniques. Evaluation results from real-world datasets and a user study demonstrate that our system can recommend highly relevant classical poems for a given photo and receive significantly higher user ratings compared to the state-of-the-art baselines.

[1]  Tefko Saracevic,et al.  Evaluation of evaluation in information retrieval , 1995, SIGIR '95.

[2]  Xing Xie,et al.  Image Inspired Poetry Generation in XiaoIce , 2018, ArXiv.

[3]  M. de Rijke,et al.  Short Text Similarity with Word Embeddings , 2015, CIKM.

[4]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[5]  Zhen Wang,et al.  Knowledge Graph Embedding by Translating on Hyperplanes , 2014, AAAI.

[6]  Shumeet Baluja,et al.  VisualRank: Applying PageRank to Large-Scale Image Search , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Heng Ji,et al.  The Age of Social Sensing , 2018, Computer.

[8]  Yang Zhang,et al.  CrowdLearn: A Crowd-AI Hybrid System for Deep Learning-based Damage Assessment Applications , 2019, 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS).

[9]  Charu C. Aggarwal,et al.  On Credibility Estimation Tradeoffs in Assured Social Sensing , 2013, IEEE Journal on Selected Areas in Communications.

[10]  Dong Wang,et al.  Mood-Sensitive Truth Discovery For Reliable Recommendation Systems in Social Sensing , 2016, RecSys.

[11]  Yoshua Bengio,et al.  Greedy Layer-Wise Training of Deep Networks , 2006, NIPS.

[12]  Cecile Chu-chin Sun,et al.  Mimesis and 興Xing: Two Modes of Viewing Reality Comparing English and Chinese Poetry , 2006, Comparative Literature Studies.

[13]  Rongrong Ji,et al.  SentiBank: large-scale ontology and classifiers for detecting sentiment and emotions in visual content , 2013, ACM Multimedia.

[14]  Rafael Valencia-García,et al.  Solving the cold-start problem in recommender systems with social tags , 2010, Expert Syst. Appl..

[15]  Chen Rui Realization of Figure-Ground in Tang Poems and Its Effect on Artistic Conception , 2008 .

[16]  Gang Wang,et al.  Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[17]  Qi Li,et al.  Crowdsourcing-Based Copyright Infringement Detection in Live Video Streams , 2018, 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM).

[18]  Michael J. Swain,et al.  WebSeer: An Image Search Engine for the World Wide Web , 1996 .

[19]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Ke Li,et al.  FauxBuster: A Content-free Fauxtography Detector Using Social Media Comments , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[21]  Yixin Cao,et al.  Explainable Reasoning over Knowledge Graphs for Recommendation , 2018, AAAI.

[22]  刘 宗亮 Research on Personalized Recommendation of Ancient Poetry Based on Word2vec Model , 2018 .

[23]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[24]  Dong Wang,et al.  When Social Sensing Meets Edge Computing: Vision and Challenges , 2019, 2019 28th International Conference on Computer Communication and Networks (ICCCN).

[25]  Jonathan Krause,et al.  A Hierarchical Approach for Generating Descriptive Image Paragraphs , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Chuan Qin,et al.  How Images Inspire Poems: Generating Classical Chinese Poetry from Images with Memory Networks , 2018, AAAI.

[27]  Xavier Serra,et al.  Sound and Music Recommendation with Knowledge Graphs , 2016, ACM Trans. Intell. Syst. Technol..

[28]  Malene Charlotte Larsen,et al.  A snap of intimacy: Photo-sharing practices among young people on social media , 2016, First Monday.

[29]  Xi Chen,et al.  Stacked Cross Attention for Image-Text Matching , 2018, ECCV.

[30]  Ronan Sicre,et al.  Particular object retrieval with integral max-pooling of CNN activations , 2015, ICLR.

[31]  Dan Roth,et al.  Provenance-Assisted Classification in Social Networks , 2014, IEEE Journal of Selected Topics in Signal Processing.

[32]  Dong Wang,et al.  Social Sensing: Building Reliable Systems on Unreliable Data , 2015 .

[33]  Qi Li,et al.  Large-scale point-of-interest category prediction using natural language processing models , 2017, 2017 IEEE International Conference on Big Data (Big Data).

[34]  Jure Leskovec,et al.  node2vec: Scalable Feature Learning for Networks , 2016, KDD.

[35]  Jatinderkumar R. Saini,et al.  Emotion Detection and Sentiment Analysis in Text Corpus: A Differential Study with Informal and Formal Writing Styles , 2014 .

[36]  Jiebo Luo,et al.  Image Captioning with Semantic Attention , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Gang Hua,et al.  Hierarchical Multimodal LSTM for Dense Visual-Semantic Embedding , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).