Classifying Emails into Human vs Machine Category

It is an essential product requirement of Yahoo Mail to distinguish between personal and machine-generated emails. The old production classifier in Yahoo Mail was based on a simple logistic regression model. That model was trained by aggregating features at the SMTP address level. We propose building deep learning models at the message level. We built and trained four individual CNN models: (1) a content model with subject and content as input; (2) a sender model with sender email address and name as input; (3) an action model by analyzing email recipients’ action patterns and correspondingly generating target labels based on senders’ opening/deleting behaviors; (4) a salutation model by utilizing senders’ “explicit salutation” signal as positive labels. Next, we built a final full model after exploring different combinations of the above four models. Experimental results on editorial data show that our full model improves the adjusted-recall from 70.5% to 78.8% compared to the old production model, while at the same time lifts the precision from 94.7% to 96.0%. Our full model also significantly beats the state-of-the-art Bert model at this task. This full model has been deployed into the current production system (Yahoo Mail 6).

[1]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[2]  Michael Viderman,et al.  Automated Extractions for Machine Generated Mail , 2018, WWW.

[3]  Quoc V. Le,et al.  Distributed Representations of Sentences and Documents , 2014, ICML.

[4]  Nir Ailon,et al.  Threading machine generated email , 2013, WSDM '13.

[5]  Manoj K. Agarwal,et al.  Template Trees: Extracting Actionable Information from Machine Generated Emails , 2018, DEXA.

[6]  Kevin Gimpel,et al.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations , 2019, ICLR.

[7]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[8]  Sanja Fidler,et al.  Skip-Thought Vectors , 2015, NIPS.

[9]  Yoelle Maarek,et al.  How Many Folders Do You Really Need?: Classifying Email into a Handful of Categories , 2014, CIKM.

[10]  Qi Zhao,et al.  Anatomy of a Privacy-Safe Large-Scale Information Extraction System Over Email , 2018, KDD.

[11]  F. S. Gharehchopogh,et al.  A novel hybrid whale optimization algorithm with flower pollination algorithm for feature selection: Case study Email spam detection , 2020, Comput. Intell..

[12]  Tomas Mikolov,et al.  Bag of Tricks for Efficient Text Classification , 2016, EACL.

[13]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[14]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[15]  Diyi Yang,et al.  Hierarchical Attention Networks for Document Classification , 2016, NAACL.

[16]  R. Balamurugan,et al.  Visual and textual features based email spam classification using S-Cuckoo search and hybrid kernel support vector machine , 2017, Cluster Computing.

[17]  Amin Mantrach,et al.  Deep Character-Level Click-Through Rate Prediction for Sponsored Search , 2017, SIGIR.

[18]  Tomas Mikolov,et al.  Enriching Word Vectors with Subword Information , 2016, TACL.

[19]  Qi Zhao,et al.  Hidden in Plain Sight: Classifying Emails Using Embedded Image Contents , 2018, WWW.

[20]  Sandeep Tata,et al.  Online Template Induction for Machine-Generated Emails , 2019, Proc. VLDB Endow..

[21]  K. Soman,et al.  ScaleNet: Scalable and Hybrid Framework for Cyber Threat Situational Awareness Based on DNS, URL, and Email Data Analysis , 2018, Journal of Cyber Security and Mobility.

[22]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[23]  Dotan Di Castro,et al.  You've got Mail, and Here is What you Could do With It!: Analyzing and Predicting Actions on Email Messages , 2016, WSDM.

[24]  Yu Sun,et al.  Learning Effective Embeddings for Machine Generated Emails with Applications to Email Category Prediction , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[25]  Yiming Yang,et al.  XLNet: Generalized Autoregressive Pretraining for Language Understanding , 2019, NeurIPS.

[26]  Milad Shokouhi,et al.  Learning with Weak Supervision for Email Intent Detection , 2020, SIGIR.

[27]  Qi Zhao,et al.  RiSER: Learning Better Representations for Richly Structured Emails , 2019, WWW.

[28]  Mouad Lemoudden,et al.  Hybrid Email Spam Detection Model Using Artificial Intelligence , 2020 .

[29]  Ryen W. White,et al.  Domain Adaptation for Commitment Detection in Email , 2019, WSDM.