A Turing Test to Evaluate a Complex Summarization Task

This paper deals with a new strategy to evaluate a Natural Language Processing NLP complex task using the Turing test. Automatic summarization based on sentence compression requires to asses informativeness and modify inner sentence structures. This is much more intrinsically related with real rephrasing than plain sentence extraction and ranking paradigm so new evaluation methods are needed. We propose a novel imitation game to evaluate Automatic Summarization by Compression ASC. Rationale of this Turing-like evaluation could be applied to many other NLP complex tasks like Machine translation or Text Generation. We show that a state of the art ASC system can pass such a test and simulate a human summary in 60% of the cases.

[1]  Gerardo Sierra,et al.  Discourse Segmentation for Sentence Compression , 2011, MICAI.

[2]  Eric SanJuan,et al.  Erratum to "DiSeg 1.0: The first system for Spanish discourse segmentation" [Expert Systems with Applications 39 (2) (2011) 1671-1678] , 2012, Expert Syst. Appl..

[3]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[4]  Stevan Harnad,et al.  Minds, Machines and Turing , 2000, J. Log. Lang. Inf..

[5]  Eric SanJuan,et al.  DiSeg 1.0: The first system for Spanish discourse segmentation , 2012, Expert Syst. Appl..

[6]  Gerardo Sierra,et al.  On the Development of the RST Spanish Treebank , 2011, Linguistic Annotation Workshop.

[7]  Alexander Gelbukh,et al.  Computational Linguistics and Intelligent Text Processing , 2015, Lecture Notes in Computer Science.

[8]  Gerardo Sierra,et al.  Discursive Sentence Compression , 2013, CICLing.

[9]  Eric SanJuan,et al.  Multilingual Summarization Evaluation without Human Models , 2010, COLING.

[10]  Ani Nenkova,et al.  Automatically Evaluating Content Selection in Summarization without Human Models , 2009, EMNLP.

[11]  Eric SanJuan,et al.  Summary Evaluation with and without References , 2010, Polytech. Open Libr. Int. Bull. Inf. Technol. Sci..

[12]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.