SemEval-2015 Task 3: Answer Selection in Community Question Answering

Community Question Answering (cQA) provides new interesting research directions to the traditional Question Answering (QA) field, e.g., the exploitation of the interaction between users and the structure of related posts. In this context, we organized SemEval2015 Task 3 on Answer Selection in cQA, which included two subtasks: (a) classifying answers as good, bad, or potentially relevant with respect to the question, and (b) answering a YES/NO question with yes, no, or unsure, based on the list of all answers. We set subtask A for Arabic and English on two relatively different cQA domains, i.e., the Qatar Living website for English, and a Quran-related website for Arabic. We used crowdsourcing on Amazon Mechanical Turk to label a large English training dataset, which we released to the research community. Thirteen teams participated in the challenge with a total of 61 submissions: 24 primary and 37 contrastive. The best systems achieved an official score (macro-averaged F1) of 57.19 and 63.7 for the English subtasks A and B, and 78.55 for the Arabic subtask A.

[1]  Xiaolong Wang,et al.  HITSZ-ICRC: Exploiting Classification Approach for Answer Selection in Community Question Answering , 2015, *SEMEVAL.

[2]  Man Lan,et al.  ECNU: Using Multiple Sources of CQA-based Information for Answers Selection and YES/NO Response Inference , 2015, *SEMEVAL.

[3]  Roberto Basili,et al.  Exploiting Syntactic and Shallow Semantic Kernels for Question Answer Classification , 2007, ACL.

[4]  Preslav Nakov,et al.  QCRI: Answer Selection for Community Question Answering - Experiments for Arabic and English , 2015, *SEMEVAL.

[5]  Quan Hung Tran,et al.  JAIST: Combining multiple features for Answer Selection in Community Question Answering , 2015, *SEMEVAL.

[6]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[7]  Chris Callison-Burch,et al.  Answer Extraction as Sequence Tagging with Tree Edit Distance , 2013, NAACL.

[8]  Alessandro Moschitti,et al.  Efficient Convolution Kernels for Dependency and Constituent Syntactic Trees , 2006, ECML.

[9]  Mirella Lapata,et al.  Using Semantic Roles to Improve Question Answering , 2007, EMNLP.

[10]  Eberhard Karls.  CoMiC: Adapting a Short Answer Assessment System for Answer Selection , 2015 .

[11]  Noah A. Smith,et al.  What is the Jeopardy Model? A Quasi-Synchronous Grammar for QA , 2007, EMNLP.

[12]  Nagwa M. El-Makky,et al.  Al-Bayan: A Knowledge-based System for Arabic Answer Selection , 2015, SemEval@NAACL-HLT.

[13]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[14]  Noah A. Smith,et al.  Tree Edit Models for Recognizing Textual Entailments, Paraphrases, and Answers to Questions , 2010, NAACL.

[15]  Saeed Rahmani,et al.  Shiraz: A Proposed List Wise Approach to Answer Validation , 2015, *SEMEVAL.

[16]  Alessandro Moschitti,et al.  Automatic Feature Engineering for Answer Selection and Extraction , 2013, EMNLP.

[17]  W. Bruce Croft,et al.  Finding similar questions in large question and answer archives , 2005, CIKM '05.

[18]  Christopher D. Manning,et al.  Probabilistic Tree-Edit Models with Structured Latent Variables for Textual Entailment and Question Answering , 2010, COLING.

[19]  Alessandro Moschitti,et al.  Structural relationships for large-scale learning of answer re-ranking , 2012, SIGIR '12.

[20]  Yang Xiang,et al.  ICRC-HIT: A Deep Learning based Comment Sequence Labeling System for Answer Selection Challenge , 2015, *SEMEVAL.

[21]  Galia Angelova,et al.  Voltron: A Hybrid System For Answer Validation Based On Lexical And Distance Features , 2015, SemEval@NAACL-HLT.

[22]  Mihai Surdeanu,et al.  Learning to Rank Answers on Large Online QA Collections , 2008, ACL.

[23]  Ngoc Phuoc An Vo,et al.  FBK-HLT: An Application of Semantic Textual Similarity for Answer Selection in Community Question Answering , 2015, SemEval@NAACL-HLT.

[24]  Filip Radlinski,et al.  Query chains: learning to rank from implicit feedback , 2005, KDD '05.

[25]  Yonatan Belinkov,et al.  VectorSLU: A Continuous Word Vector Approach to Answer Selection in Community Question Answering Systems , 2015, *SEMEVAL.