Quantifying the Uncertainty of Precision Estimates for Rule based Text Classifiers

Rule based classifiers that use the presence and absence of key sub-strings to make classification decisions have a natural mechanism for quantifying the uncertainty of their precision. For a binary classifier, the key insight is to treat partitions of the sub-string set induced by the documents as Bernoulli random variables. The mean value of each random variable is an estimate of the classifier's precision when presented with a document inducing that partition. These means can be compared, using standard statistical tests, to a desired or expected classifier precision. A set of binary classifiers can be combined into a single, multi-label classifier by an application of the Dempster-Shafer theory of evidence. The utility of this approach is demonstrated with a benchmark problem.

[1]  H. Saunders,et al.  Probability, Random Variables and Stochastic Processes (2nd Edition) , 1989 .

[2]  Andrew L. Beam,et al.  Adversarial attacks on medical machine learning , 2019, Science.

[3]  Nhan T. Nguyen,et al.  Stability, Convergence, and Verification and Validation Challenges of Neural Net Adaptive Flight Control , 2010, Applications of Neural Networks in High Assurance Systems.

[4]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[5]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[6]  W. Ashby,et al.  Requisite Variety and Its Implications for the Control of Complex Systems , 1991 .

[7]  Liguo Weng,et al.  Pitch-Depth Control of Submarine Operating in Shallow Water via Neuro-adaptive Approach , 2010, Applications of Neural Networks in High Assurance Systems.

[8]  Athanasios Papoulis,et al.  Probability, Random Variables and Stochastic Processes , 1965 .

[9]  Geoffrey E. Hinton,et al.  Keeping the neural networks simple by minimizing the description length of the weights , 1993, COLT '93.

[10]  J. Denny,et al.  Artificial intelligence, bias and clinical safety , 2019, BMJ Quality & Safety.

[11]  Mikhail Belkin,et al.  To understand deep learning we need to understand kernel learning , 2018, ICML.

[12]  Laurence Hirsch,et al.  Evolving Rules for Document Classification , 2005, EuroGP.

[13]  Alex Graves,et al.  Practical Variational Inference for Neural Networks , 2011, NIPS.

[14]  Wen Yu,et al.  Modelling of crude oil blending via discrete-time neural networks , 2004 .

[15]  A statistician’s perspective on digital epidemiology , 2017, Life sciences, society and policy.

[16]  Sholom M. Weiss,et al.  Automated learning of decision rules for text categorization , 1994, TOIS.

[17]  Jimeng Sun,et al.  Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review , 2018, J. Am. Medical Informatics Assoc..

[18]  David B. Dunson,et al.  Statistics in the big data era: Failures of the machine , 2018 .

[19]  Nida Shahid,et al.  Applications of artificial neural networks in health care organizational decision-making: A scoping review , 2019, PloS one.

[20]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[21]  Haipeng Shen,et al.  Artificial intelligence in healthcare: past, present and future , 2017, Stroke and Vascular Neurology.

[22]  Timothy A. Smith,et al.  Design and Flight Test of an Intelligent Flight Control System , 2010, Applications of Neural Networks in High Assurance Systems.

[23]  Jürg Kohlas,et al.  Theory of evidence — A survey of its mathematical foundations, applications and computational aspects , 1994, Math. Methods Oper. Res..

[24]  Andy Brown,et al.  Recurrent Neural Network Attention Mechanisms for Interpretable System Log Anomaly Detection , 2018, Proceedings of the First Workshop on Machine Learning for Computing Systems.

[25]  Pasquale Rullo,et al.  A Genetic Algorithm for Text Classification Rule Induction , 2008, ECML/PKDD.

[26]  Jianfeng Gao,et al.  DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation , 2020, ACL.

[27]  Sebastian Ruder,et al.  Universal Language Model Fine-tuning for Text Classification , 2018, ACL.