A Maximum Expected Utility Framework for Binary Sequence Labeling

We consider the problem of predictive inference for probabilistic binary sequence labeling models under F-score as utility. For a simple class of models, we show that the number of hypotheses whose expected Fscore needs to be evaluated is linear in the sequence length and present a framework for efficiently evaluating the expectation of many common loss/utility functions, including the F-score. This framework includes both exact and faster inexact calculation methods.

[1]  Chin-Hui Lee,et al.  A maximal figure-of-merit (MFoM)-learning approach to robust classifier design for text categorization , 2006, ACM Trans. Inf. Syst..

[2]  Olga Russakovsky,et al.  Training Conditional Random Fields for Maximum Labelwise Accuracy , 2006, NIPS.

[3]  Kevin Knight,et al.  Decoding Complexity in Word-Replacement Translation Models , 1999, Comput. Linguistics.

[4]  Richard W. Hamming,et al.  Error detecting and error correcting codes , 1950 .

[5]  Martin Jansche,et al.  Maximum Expected F-Measure Training of Logistic Regression Models , 2005, HLT.

[6]  Jun Suzuki,et al.  Training Conditional Random Fields with Multivariate Evaluation Measures , 2006, ACL.

[7]  Mehryar Mohri,et al.  Rational Kernels , 2002, NIPS.

[8]  Francisco Casacuberta,et al.  Submission to ICGI-2000 Computational complexity of problems on probabilistic grammars and transducers , 2007 .

[9]  C. J. van Rijsbergen,et al.  FOUNDATION OF EVALUATION , 1974 .

[10]  John D. Lafferty,et al.  Statistical Models for Text Segmentation , 1999, Machine Learning.

[11]  William M. Shaw,et al.  On the foundation of evaluation , 1986, J. Am. Soc. Inf. Sci..

[12]  Brian Roark,et al.  Generalized Algorithms for Constructing Statistical Language Models , 2003, ACL.

[13]  Vipin Kumar,et al.  Optimizing F-Measure with Support Vector Machines , 2003, FLAIRS Conference.

[14]  Michael C. Mozer,et al.  Prodding the ROC Curve: Constrained Optimization of Classifier Performance , 2001, NIPS.