Deep or simple models for semantic tagging?

Semantic tagging, which has extensive applications in text mining, predicts whether a given piece of text conveys the meaning of a given semantic tag. The problem of semantic tagging is largely solved with supervised learning and today, deep learning models are widely perceived to be better for semantic tagging. However, there is no comprehensive study supporting the popular belief. Practitioners often have to train different types of models for each semantic tagging task to identify the best model. This process is both expensive and inefficient. We embark on a systematic study to investigate the following question: Are deep models the best performing model for all semantic tagging tasks? To answer this question, we compare deep models against "simple models" over datasets with varying characteristics. Specifically, we select three prevalent deep models (i.e. CNN, LSTM, and BERT) and two simple models (i.e. LR and SVM), and compare their performance on the semantic tagging task over 21 datasets. Results show that the size, the label ratio, and the label cleanliness of a dataset significantly impact the quality of semantic tagging. Simple models achieve similar tagging quality to deep models on large datasets, but the runtime of simple models is much shorter. Moreover, simple models can achieve better tagging quality than deep models when targeting datasets show worse label cleanliness and/or more severe imbalance. Based on these findings, our study can systematically guide practitioners in selecting the right learning model for their semantic tagging task.

[1]  Anna Rumshisky,et al.  SemEval-2017 Task 6: #HashtagWars: Learning a Sense of Humor , 2017, *SEMEVAL.

[2]  Thorsten Joachims,et al.  Text Categorization with Support Vector Machines: Learning with Many Relevant Features , 1998, ECML.

[3]  Xiaojun Wan,et al.  A Neural Approach to Pun Generation , 2018, ACL.

[4]  Chih-Jen Lin,et al.  Training and Testing Low-degree Polynomial Data Mappings via Linear SVM , 2010, J. Mach. Learn. Res..

[5]  Avneesh Saluja,et al.  Using Aspect Extraction Approaches to Generate Review Summaries and User Profiles , 2018, NAACL.

[6]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[7]  Mong-Li Lee,et al.  Author-aware Aspect Topic Sentiment Model to Retrieve Supporting Opinions from Reviews , 2017, EMNLP.

[8]  Michael J. A. Berry,et al.  Data mining techniques - for marketing, sales, and customer support , 1997, Wiley computer publishing.

[9]  Michael J. Franklin,et al.  Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing , 2012, NSDI.

[10]  Wei Lu,et al.  Joint Detection and Location of English Puns , 2019, NAACL.

[11]  Lu Wang,et al.  Argument Mining for Understanding Peer Reviews , 2019, NAACL.

[12]  Mengting Wan,et al.  Fine-Grained Spoiler Detection from Large-Scale Review Corpora , 2019, ACL.

[13]  Christopher D. Manning,et al.  Introduction to Information Retrieval , 2010, J. Assoc. Inf. Sci. Technol..

[14]  Paul Buitelaar,et al.  A Study of Suggestions in Opinionated Texts and their Automatic Detection , 2016, *SEMEVAL.

[15]  Omer Levy,et al.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding , 2018, BlackboxNLP@EMNLP.

[16]  Yifan Sun,et al.  A Sparse Topic Model for Extracting Aspect-Specific Summaries from Online Reviews , 2018, WWW.

[17]  Paul Buitelaar,et al.  SemEval-2019 Task 9: Suggestion Mining from Online Reviews and Forums , 2019, *SEMEVAL.

[18]  Trevor Campbell,et al.  Coresets for Scalable Bayesian Logistic Regression , 2016, NIPS.

[19]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[20]  Aristides Gionis,et al.  Answers, not links: extracting tips from yahoo! answers to address how-to web queries , 2012, WSDM '12.

[21]  Jordan L. Boyd-Graber,et al.  Spoiler alert: Machine learning approaches to detect social media posts with revelatory information , 2013, ASIST.

[22]  Edward A. Fox,et al.  Discovering Product Defects and Solutions from Online User Generated Contents , 2019, WWW.

[23]  Ido Guy,et al.  Generating Product Descriptions from User Reviews , 2019, WWW.

[24]  Pavel Braslavski,et al.  Large Dataset and Language Model Fun-Tuning for Humor Recognition , 2019, ACL.

[25]  Theodoros Lappas,et al.  Unsupervised tip-mining from customer reviews , 2018, Decis. Support Syst..

[26]  Donald E. Brown,et al.  Text Classification Algorithms: A Survey , 2019, Inf..

[27]  Lei Zhang,et al.  Sentiment Analysis and Opinion Mining , 2017, Encyclopedia of Machine Learning and Data Mining.

[28]  Charu C. Aggarwal,et al.  A Survey of Text Classification Algorithms , 2012, Mining Text Data.

[29]  Soo-Min Kim,et al.  Automatic Identification of Pro and Con Reasons in Online Reviews , 2006, ACL.

[30]  Di Wu,et al.  Heterographic Pun Recognition via Pronunciation and Spelling Understanding Gated Attention Network , 2019, WWW.

[31]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[32]  James E. Helmreich Regression Modeling Strategies with Applications to Linear Models, Logistic and Ordinal Regression and Survival Analysis (2nd Edition) , 2016 .

[33]  Diyi Yang,et al.  Humor Recognition and Humor Anchor Extraction , 2015, EMNLP.

[34]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[35]  Xiang Zhang,et al.  Character-level Convolutional Networks for Text Classification , 2015, NIPS.

[36]  Iryna Gurevych,et al.  SemEval-2017 Task 7: Detection and Interpretation of English Puns , 2017, *SEMEVAL.

[37]  Ido Dagan,et al.  Synthesis Lectures on Human Language Technologies , 2009 .

[38]  Jinfeng Li,et al.  Deep or Simple Models for Semantic Tagging? It Depends on your Data , 2020, Proc. VLDB Endow..

[39]  Paul Buitelaar,et al.  Towards the Extraction of Customer-to-Customer Suggestions from Reviews , 2015, EMNLP.

[40]  Shaohua Wang,et al.  Extracting API Tips from Developer Question and Answer Websites , 2019, 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR).

[41]  Ovidiu Ivanciuc,et al.  Applications of Support Vector Machines in Chemistry , 2007 .

[42]  ChengXiang Zhai,et al.  Identifying Humor in Reviews using Background Text Sources , 2017, EMNLP.

[43]  Kevin Gimpel,et al.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations , 2019, ICLR.

[44]  Jiaxiang Liu,et al.  OleNet at SemEval-2019 Task 9: BERT based Multi-Perspective Models for Suggestion Mining , 2019, SemEval@NAACL-HLT.

[45]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[46]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[47]  Ido Guy,et al.  Extracting and Ranking Travel Tips from User-Generated Reviews , 2017, WWW.

[48]  Iryna Gurevych,et al.  Cross-topic Argument Mining from Heterogeneous Sources , 2018, EMNLP.

[49]  Xiaojuan Ma,et al.  Recognizing Humour using Word Associations and Humour Anchor Extraction , 2018, COLING.

[50]  Daniel Jurafsky,et al.  Shallow Semantic Parsing using Support Vector Machines , 2004, NAACL.

[51]  Yoon Kim,et al.  Convolutional Neural Networks for Sentence Classification , 2014, EMNLP.

[52]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[53]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.