Image Inspired Poetry Generation in XiaoIce

Vision is a common source of inspiration for poetry. The objects and the sentimental imprints that one perceives from an image may lead to various feelings depending on the reader. In this paper, we present a system of poetry generation from images to mimic the process. Given an image, we first extract a few keywords representing objects and sentiments perceived from the image. These keywords are then expanded to related ones based on their associations in human written poems. Finally, verses are generated gradually from the keywords using recurrent neural networks trained on existing poems. Our approach is evaluated by human assessors and compared to other generation baselines. The results show that our method can generate poems that are more artistic than the baseline methods. This is one of the few attempts to generate poetry from images. By deploying our proposed approach, XiaoIce has already generated more than 12 million poems for users since its release in July 2017. A book of its poems has been published by Cheers Publishing, which claimed that the book is the first-ever poetry collection written by an AI in human history.

[1]  Maosong Sun,et al.  Generating Chinese Classical Poems with RNN Encoder-Decoder , 2016, CCL.

[2]  Hendrik P. A. Lensch,et al.  Auto-Illustrating Poems and Songs with Style , 2016, ACCV.

[3]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[4]  Rui Yan,et al.  i, Poet: Automatic Poetry Composition through Recurrent Neural Networks with Iterative Polishing Schema , 2016, IJCAI.

[5]  Dong Wang,et al.  Chinese Song Iambics Generation with Neural Attention-Based Model , 2016, IJCAI.

[6]  Joelle Pineau,et al.  How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation , 2016, EMNLP.

[7]  Enhong Chen,et al.  Chinese Poetry Generation with Planning based Neural Network , 2016, COLING.

[8]  Yoav Goldberg,et al.  Gaiku : Generating Haiku with Word Associations Norms , 2009 .

[9]  Mirella Lapata,et al.  Chinese Poetry Generation with Recurrent Neural Networks , 2014, EMNLP.

[10]  Ruli Manurung,et al.  Using genetic algorithms to create meaningful poetic text , 2012, J. Exp. Theor. Artif. Intell..

[11]  Trevor Darrell,et al.  Long-term recurrent convolutional networks for visual recognition and description , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Chen Xu,et al.  The SUN Attribute Database: Beyond Categories for Deeper Scene Understanding , 2014, International Journal of Computer Vision.

[13]  John B. Shoven,et al.  I , Edinburgh Medical and Surgical Journal.

[14]  Long Jiang,et al.  Generating Chinese Couplets using a Statistical MT Approach , 2008, COLING.

[15]  Tao Mei,et al.  Tagging Personal Photos with Transfer Deep Learning , 2015, WWW.

[16]  Quoc V. Le,et al.  Grounded Compositional Semantics for Finding and Describing Images with Sentences , 2014, TACL.

[17]  Geoffrey Zweig,et al.  Language Models for Image Captioning: The Quirks and What Works , 2015, ACL.

[18]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[19]  Michihiko Minoh,et al.  Hitch Haiku: An Interactive Supporting System for Composing Haiku Poem , 2008, ICEC.

[20]  Ryohei Nakatsu,et al.  New Hitch Haiku: An Interactive Renku Poem Composition Supporting Tool Applied for Sightseeing Navigation System , 2009, ICEC.

[21]  Shou-De Lin,et al.  i, Poet: Automatic Chinese Poetry Composition through a Generative Summarization Framework under Constrained Optimization , 2013, IJCAI.

[22]  Hugo Gonçalo Oliveira.  PoeTryMe : a versatile platform for poetry generation , 2012 .

[23]  Xinlei Chen,et al.  Microsoft COCO Captions: Data Collection and Evaluation Server , 2015, ArXiv.

[24]  Lukás Burget,et al.  Recurrent neural network based language model , 2010, INTERSPEECH.

[25]  H. Manurung.  An evolutionary algorithm approach to poetry generation , 2004 .

[26]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[27]  Harry Shum,et al.  From Eliza to XiaoIce: challenges and opportunities with social chatbots , 2018, Frontiers of Information Technology & Electronic Engineering.

[28]  Fei-Fei Li,et al.  Deep visual-semantic alignments for generating image descriptions , 2015, CVPR.

[29]  Yejin Choi,et al.  Generating Topical Poetry , 2016, EMNLP.

[30]  Nazli Ikizler-Cinbis,et al.  Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures , 2016, J. Artif. Intell. Res..

[31]  Long Jiang,et al.  Generating Chinese Classical Poems with Statistical Machine Translation Models , 2012, AAAI.

[32]  Geoffrey Zweig,et al.  From captions to visual concepts and back , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Tairui Chen,et al.  Going Deeper with Convolutional Neural Network for Intelligent Transportation , 2016 .

[34]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[35]  Vlado Keselj,et al.  Machine learning meets visualization for extracting insights from text data , 2016, SIGAI.

[36]  Rongrong Ji,et al.  Large-scale visual sentiment ontology and detectors using adjective noun pairs , 2013, ACM Multimedia.