A Linguistic Failure Analysis of Classification of Medical Publications: A Study on Stemming vs Lemmatization

English. Technology-Assisted Review (TAR) systems are essential to minimize the effort of the user during the search and retrieval of relevant documents for a specific information need. In this paper, we present a failure analysis based on terminological and linguistic aspects of a TAR system for systematic medical reviews. In particular, we analyze the results of the worst performing topics in terms of recall using the dataset of the CLEF 2017 eHealth task on TAR in Empirical Medicine. Italiano. I sistemi TAR (TechnologyAssisted Review) sono fondamentali per ridurre al minimo lo sforzo dell’utente che intende ricercare e recuperare i documenti rilevanti per uno specifico bisogno informativo. In questo articolo, presentiamo una failure analysis basata su aspetti terminologici e linguistici di un sistema TAR per le revisioni sistematiche in campo medico. In particolare, analizziamo i topic per i quali abbiamo ottenuto dei risultati peggiori in termini di recall utilizzando il dataset di CLEF 2017 eHealth task on TAR in Empirical Medicine.

[1]  Karin M. Verspoor,et al.  BioLemmatizer: a lemmatization tool for morphological processing of biomedical text , 2012, J. Biomed. Semant..

[2]  Gianmaria Silvello,et al.  Statistical Stemmers: A Reproducibility Study , 2018, ECIR.

[3]  Kalervo Järvelin,et al.  To stem or lemmatize a highly inflectional language in a probabilistic IR environment? , 2005, J. Documentation.

[4]  Martin F. Porter,et al.  An algorithm for suffix stripping , 1997, Program.

[5]  Giorgio Maria Di Nunzio,et al.  TriMED: A Multilingual Terminological Database , 2018, International Conference on Language Resources and Evaluation.

[6]  Giorgio Maria Di Nunzio,et al.  An Interactive Two-Dimensional Approach to Query Aspects Rewriting in Systematic Reviews. IMS Unipd At CLEF eHealth Task 2 , 2017, CLEF.

[7]  Giorgio Maria Di Nunzio.  A new decision to take for cost-sensitive Naïve Bayes classifiers , 2014, Inf. Process. Manag..

[8]  Jakub Kanis,et al.  Comparison of Different Lemmatization Approaches through the Means of Information Retrieval Performance , 2010, TSD.

[9]  Helmut Schmid,et al.  Improvements in Part-of-Speech Tagging with an Application to German , 1999 .

[10]  Mihai Surdeanu,et al.  The Stanford CoreNLP Natural Language Processing Toolkit , 2014, ACL.

[11]  Giorgio Maria Di Nunzio.  A Study of an Automatic Stopping Strategy for Technologically Assisted Medical Reviews , 2018, ECIR.

[12]  Leif Azzopardi,et al.  CLEF 2018 Technologically Assisted Reviews in Empirical Medicine Overview , 2018, CLEF.

[13]  Vimala Balakrishnan,et al.  Stemming and lemmatization: A comparison of retrieval performances , 2014 .

[14]  Hugo Zaragoza,et al.  The Probabilistic Relevance Framework: BM25 and Beyond , 2009, Found. Trends Inf. Retr..