Learning to rank answers to closed-domain questions by using fuzzy logic

Question answering (QA) is a challenging task and has received considerable attention in the last years. Answer selection among candidate answers is one of the main phases for QA and the best answer to be returned is determined in this phase. A common approach consists in considering the selection of the final answer(s) as a ranking problem. So far, different methods have been proposed, mainly oriented to produce a single best ranking model operating in the same way on all the question types. Differently, this paper proposes a fuzzy approach for ranking and selecting the correct answer among a list of candidates in a state-of-the-art QA system operating with factoid and description questions on Italian corpora pertaining a closed domain. Starting from the consideration that this ranking problem can be reduced to a classification one, the proposed approach is based on the Likelihood-Fuzzy Analysis (LFA), applied in this case for mining fuzzy rule-based models able to discern correct (True) from incorrect answers (False). Such fuzzy models are mined as specifically tailored to each question type, and, thus, can be individually applied to produce a more robust and accurate final ranking. An experimental session over a collection of questions pertaining the Cultural Heritage domain, using a manually annotated gold-standard dataset, shows that considering specific fuzzy ranking models for each question type improves the accuracy of the best answer returned back to the user.

[1]  Sanda M. Harabagiu,et al.  LCC Tools for Question Answering , 2002, TREC.

[2]  Massimo Esposito,et al.  A Composite Model for Classifying Parotid Shrinkage in Radiotherapy Patients Using Heterogeneous Data , 2015, AIME.

[3]  Dan Roth,et al.  Learning question classifiers: the role of semantic information , 2005, Natural Language Engineering.

[4]  Jennifer Chu-Carroll,et al.  Building Watson: An Overview of the DeepQA Project , 2010, AI Mag..

[5]  Martin M. Soubbotin,et al.  Use of Patterns for Detection of Likely Answer Strings: A Systematic Approach , 2002, TREC.

[6]  Eyke Hüllermeier,et al.  Preference Learning Using the Choquet Integral: The Case of Multipartite Ranking , 2012, IEEE Transactions on Fuzzy Systems.

[7]  Giovanna Rizzo,et al.  Early classification of parotid glands shrinkage in radiotherapy patients: A comparative study , 2015 .

[8]  Scott Miller,et al.  Answer Selection and Confidence Estimation , 2003, New Directions in Question Answering.

[9]  Aditya Kalyanpur,et al.  A framework for merging and ranking of answers in DeepQA , 2012, IBM J. Res. Dev..

[10]  Giuseppe De Pietro,et al.  Best Fuzzy Partitions to Build Interpretable DSSs for Classification in Medicine , 2013, HAIS.

[11]  Ee-Peng Lim,et al.  Quality-aware collaborative question answering: methods and evaluation , 2009, WSDM '09.

[12]  Dragomir R. Radev,et al.  Question-answering by predictive annotation , 2000, SIGIR '00.

[13]  Marius Pasca.  A Relational and Logic Representation for Open-Domain Textual Question Answering , 2001, ACL.

[14]  Giuseppe De Pietro,et al.  Towards a Framework for Closed-Domain Question Answering in Italian , 2016, 2016 12th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS).

[15]  Bernardo Magnini,et al.  Comparing Statistical and Content-Based Techniques for Answer Validation on the Web , 2002 .

[16]  Jeffrey Pomerantz,et al.  Evaluating and predicting answer quality in community QA , 2010, SIGIR.

[17]  Eduard H. Hovy,et al.  Learning surface text patterns for a Question Answering System , 2002, ACL.

[18]  Jun Suzuki,et al.  SVM Answer Selection for Open-Domain Question Answering , 2002, COLING.

[19]  Yutaka Sasaki.  Question Answering as Question-Biased Term Extraction: A New Approach toward Multilingual QA , 2005, ACL.

[20]  Jennifer Chu-Carroll,et al.  IBM's PIQUANT in TREC2003 , 2003, TREC.

[21]  Eduard H. Hovy,et al.  The Use of External Knowledge of Factoid QA , 2001, TREC.

[22]  Grace Hui Yang,et al.  QUALIFIER: Question Answering by Lexical Fabric and External Resources , 2003, EACL.

[23]  Alexander H. Waibel,et al.  A Pattern Learning Approach to Question Answering Within the Ephyra Framework , 2006, TSD.

[24]  Hwee Tou Ng,et al.  Question Answering Using a Large Text Database: A Machine Learning Approach , 2001, EMNLP.

[25]  Claire Cardie,et al.  Examining the Role of Statistical and Linguistic Knowledge Sources in a General-Knowledge Question-Answering System , 2000, ANLP.

[26]  Eyke Hüllermeier,et al.  Label ranking by learning pairwise preferences , 2008, Artif. Intell..

[27]  Petr Baudis,et al.  Modeling of the Question Answering Task in the YodaQA System , 2015, CLEF.

[28]  M. de Rijke,et al.  Type Checking in Open-Domain Question Answering , 2004, ECAI.

[29]  Pável Calado,et al.  Exploiting user feedback to learn to rank answers in q&a forums: a case study with stack overflow , 2013, SIGIR.

[30]  Yong Yu,et al.  Understanding and Summarizing Answers in Community-Based Question Answering Services , 2008, COLING.

[31]  Sanda M. Harabagiu,et al.  COGEX: A Logic Prover for Question Answering , 2003, NAACL.

[32]  Anonymous Author.  Robust Reductions from Ranking to Classification , 2006 .