论文信息 - SOREL-20M: A Large Scale Benchmark Dataset for Malicious PE Detection

SOREL-20M: A Large Scale Benchmark Dataset for Malicious PE Detection

In this paper we describe the SOREL-20M (Sophos/ReversingLabs-20 Million) dataset: a large-scale dataset consisting of nearly 20 million files with pre-extracted features and metadata, high-quality labels derived from multiple sources, information about vendor detections of the malware samples at the time of collection, and additional ``tags'' related to each malware sample to serve as additional targets. In addition to features and metadata, we also provide approximately 10 million ``disarmed'' malware samples -- samples with both the optional\_headers.subsystem and file\_header.machine flags set to zero -- that may be used for further exploration of features and detection strategies. We also provide Python code to interact with the data and features, as well as baseline neural network and gradient boosted decision tree models and their results, with full training and evaluation code, to serve as a starting point for further experimentation.

Richard E. Harang | Richard Harang | Ethan M. Rudd

[1] Konstantin Berlin,et al. Improving Zero-Day Malware Testing Methodology Using Statistically Significant Time-Lagged Test Samples , 2016, ArXiv.

[2] Alex Krizhevsky,et al. Learning Multiple Layers of Features from Tiny Images , 2009 .

[3] Hyrum S. Anderson,et al. EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models , 2018, ArXiv.

[4] Richard E. Harang,et al. ALOHA: Auxiliary Loss Optimization for Hypothesis Augmentation , 2019, USENIX Security Symposium.

[5] Christopher Potts,et al. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank , 2013, EMNLP.

[6] Ethan M. Rudd,et al. SMART: Semantic Malware Attribute Relevance Tagging , 2019, ArXiv.

[7] Natalia Gimelshein,et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[8] Tie-Yan Liu,et al. LightGBM: A Highly Efficient Gradient Boosting Decision Tree , 2017, NIPS.

[9] Li Fei-Fei,et al. ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[10] Rich Caruana,et al. A Dozen Tricks with Multitask Learning , 1996, Neural Networks: Tricks of the Trade.