Visual Supervision in Bootstrapped Information Extraction

We challenge a common assumption in active learning, that a list-based interface populated by informative samples provides for efficient and effective data annotation. We show how a 2D scatterplot populated with diverse and representative samples can yield improved models given the same time budget. We consider this for bootstrapping-based information extraction, in particular named entity classification, where human and machine jointly label data. To enable effective data annotation in a scatterplot, we have developed an embedding-based bootstrapping model that learns the distributional similarity of entities through the patterns that match them in a large data corpus, while being discriminative with respect to human-labeled and machine-promoted entities. We conducted a user study to assess the effectiveness of these different interfaces, and analyze bootstrapping performance in terms of human labeling accuracy, label quantity, and labeling consensus across multiple users. Our results suggest that supervision acquired from the scatterplot interface, despite being noisier, yields improvements in classification performance compared with the list interface, due to a larger quantity of supervision acquired.

[1]  Roman Yangarber,et al.  Counter-Training in Discovery of Semantic Patterns , 2003, ACL.

[2]  Christopher D. Manning,et al.  Combining Distant and Partial Supervision for Relation Extraction , 2014, EMNLP.

[3]  Thomas Ertl,et al.  Visual Classifier Training for Text Document Retrieval , 2012, IEEE Transactions on Visualization and Computer Graphics.

[4]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[5]  Danqi Chen,et al.  Bootstrapped Self Training for Knowledge Base Population , 2015, TAC.

[6]  Feng Zhou,et al.  Fine-Grained Categorization and Dataset Bootstrapping Using Deep Metric Learning with Humans in the Loop , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Yifan He,et al.  ICE: Rapid Information Extraction Customization for NLP Novices , 2015, HLT-NAACL.

[8]  Christopher D. Manning,et al.  On-the-Job Learning with Bayesian Decision Theory , 2015, NIPS.

[9]  Oren Etzioni,et al.  IKE - An Interactive Tool for Knowledge Extraction , 2016, AKBC@NAACL-HLT.

[10]  Christopher D. Manning,et al.  Distributed Representations of Words to Guide Bootstrapped Entity Classifiers , 2015, NAACL.

[11]  Ramesh Nallapati,et al.  Multi-instance Multi-label Learning for Relation Extraction , 2012, EMNLP.

[12]  Mihai Surdeanu,et al.  Lightly-supervised Representation Learning with Global Interpretability , 2018, SPNLP@NAACL-HLT.

[13]  Niklas Elmqvist,et al.  Evaluating Visual Representations for Topic Understanding and Their Effects on Manually Generated Topic Labels , 2017, TACL.

[14]  Jordan L. Boyd-Graber,et al.  Closing the Loop: User-Centered Design and Evaluation of a Human-in-the-Loop Topic Modeling System , 2018, IUI.

[15]  Daniel Jurafsky,et al.  Distant supervision for relation extraction without labeled data , 2009, ACL.

[16]  Estevam R. Hruschka,et al.  Coupled semi-supervised learning for information extraction , 2010, WSDM '10.

[17]  Christopher D. Manning,et al.  Improved Pattern Learning for Bootstrapped Entity Extraction , 2014, CoNLL.

[18]  Marco Hutter,et al.  Comparing Visual-Interactive Labeling with Active Learning: An Experimental Study , 2018, IEEE Transactions on Visualization and Computer Graphics.

[19]  Paulo E. Rauber,et al.  Visualizing Time-Dependent Data Using Dynamic t-SNE , 2016, EuroVis.

[20]  Yoram Singer,et al.  Unsupervised Models for Named Entity Classification , 1999, EMNLP.

[21]  Ralph Grishman,et al.  An Efficient Active Learning Framework for New Relation Types , 2013, IJCNLP.

[22]  Yi Zhang,et al.  Incorporating Diversity and Density in Active Learning for Relevance Feedback , 2007, ECIR.

[23]  Angli Liu,et al.  Effective Crowd Annotation for Relation Extraction , 2016, NAACL.

[24]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[25]  Gunther Heidemann,et al.  Inter-active learning of ad-hoc classifiers for video visual analytics , 2012, 2012 IEEE Conference on Visual Analytics Science and Technology (VAST).

[26]  Yuji Matsumoto,et al.  Graph-based Analysis of Semantic Drift in Espresso-like Bootstrapping Algorithms , 2008, EMNLP.

[27]  Quentin Pleple,et al.  Interactive Topic Modeling , 2013 .

[28]  David Cohn,et al.  Active Learning , 2010, Encyclopedia of Machine Learning.

[29]  Fei Xia,et al.  CROWD-IN-THE-LOOP: A Hybrid Approach for Annotating Semantic Roles , 2017, EMNLP.

[30]  Kihyuk Sohn,et al.  Improved Deep Metric Learning with Multi-class N-pair Loss Objective , 2016, NIPS.

[31]  Jordan L. Boyd-Graber,et al.  ALTO: Active Learning with Topic Overviews for Speeding Label Induction and Document Labeling , 2016, ACL.

[32]  David Yarowsky,et al.  Unsupervised Word Sense Disambiguation Rivaling Supervised Methods , 1995, ACL.

[33]  Omer Levy,et al.  Neural Word Embedding as Implicit Matrix Factorization , 2014, NIPS.