Error Prediction with Partial Feedback

In this paper, we propose a probabilistic framework for predicting the root causes of errors in data processing pipelines made up of several components when we only have access to partial feedback; that is, we are aware when some error has occurred in one or more of the components, but we do not know which one. The proposed error model enables us to direct the user feedback to the correct components in the pipeline to either automatically correct errors as they occur, retrain the component with assimilated training examples, or take other corrective action. We present the model and describe an Expectation Maximization (EM)-based algorithm to learn the model parameters and predict the error configuration. We demonstrate the accuracy and usefulness of our method first on synthetic data, and then on two distinct tasks: error correction in a 2-component opinion summarization system, and phrase error detection in statistical machine translation.

[1]  Yue Lu,et al.  Rated aspect summarization of short comments , 2009, WWW '09.

[2]  Yorick Wilks,et al.  Software Infrastructure for Natural Language Processing , 1997, ANLP.

[3]  Philipp Koehn,et al.  Moses: Open Source Toolkit for Statistical Machine Translation , 2007, ACL.

[4]  Dan Klein,et al.  Online EM for Unsupervised Models , 2009, NAACL.

[5]  Giorgio Satta,et al.  Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing , 2011 .

[6]  Hua Xu,et al.  Identifying Evaluative Sentences in Online Discussions , 2011, AAAI.

[7]  john maccoll,et al.  ACM / IEEE Joint Conference on Digital Libraries , 2001 .

[8]  Hermann Ney,et al.  Word-Level Confidence Estimation for Machine Translation , 2007, CL.

[9]  Conference on Empirical Methods in Natural Language Processing, EMNLP 1996, Philadelphia, PA, USA, May 17-18, 1996 , 1996, EMNLP.

[10]  Sasha Blair-Goldensohn,et al.  Building a Sentiment Summarizer for Local Service Reviews , 2008 .

[11]  Paul S. Bradley,et al.  Initialization of Iterative Refinement Clustering Algorithms , 1998, KDD.

[12]  Andrew Y. Ng,et al.  Solving the Problem of Cascading Errors: Approximate Bayesian Inference for Linguistic Annotation Pipelines , 2006, EMNLP.

[13]  Radford M. Neal.  Connectionist Learning of Belief Networks , 1992, Artif. Intell..

[14]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[15]  Ken Thompson,et al.  The UNIX time-sharing system , 1974, CACM.

[16]  Christopher D. Manning,et al.  Learning to recognize features of valid textual entailments , 2006, NAACL.

[17]  Oren Etzioni,et al.  Named Entity Recognition in Tweets: An Experimental Study , 2011, EMNLP.

[18]  Haizhou Li,et al.  Error Detection for Statistical Machine Translation Using Linguistic Features , 2010, ACL.

[19]  Michael Strube,et al.  Beyond the Pipeline: Discrete Optimization in NLP , 2005, CoNLL.

[20]  Mirella Lapata,et al.  Discourse Chunking and its Application to Sentence Compression , 2005, HLT.

[21]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[22]  Mark Dredze,et al.  Separating Fact from Fear: Tracking Flu Infections on Twitter , 2013, NAACL.

[23]  Léon Bottou,et al.  Large-Scale Machine Learning with Stochastic Gradient Descent , 2010, COMPSTAT.

[24]  Ani Nenkova,et al.  A compositional context sensitive multi-document summarizer: exploring the factors that influence summarization , 2006, SIGIR.

[25]  Andrew McCallum,et al.  Proceedings of the Ninth Conference on Computational Natural Language Learning, CoNLL 2005, Ann Arbor, Michigan, USA, June 29-30, 2005 , 2005, CoNLL.

[26]  Janyce Wiebe,et al.  Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis , 2005, HLT.