Devise Sparse Compression Schedulers to Enhance FastText Methods

In natural language processing(NLP), the general way to understand the meaning of a word is via word embedding. The word embedding training model can convert words into multidimensional vectors and make the words that do not know “meaning” into vectors with “meaning”. Famous word embedding training models, include models such as FastText, Word2Vec, and GloVe. They can train words into vectors and then they are used for further semantic classifications. In this paper, we work on the efficient support for the FastText. FastText is an open source library created by Facebook(FAIR) lab that allows users to learn word embedding and text classification. We focus on the word representation application in FastText, in which general matrix-Vector multiplication(GEMV) is one of the most computationally intensive operations. In this paper, we adjust the software architecture of FastText, and pre-process the pre-trained model offline. In addition, we introduce a new accelerating method with sparse matrix compression in Halide, which improves performance by compressing the matrix. Our support with Halide sparse compression schedulers include hybrid compression schemes and re-ordering methods to improve the performance.

[1]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[2]  Frédo Durand,et al.  Decoupling algorithms from schedules for easy optimization of image processing pipelines , 2012, ACM Trans. Graph..

[3]  J. W. Walker,et al.  Direct solutions of sparse network equations by optimally ordered triangular factorization , 1967 .

[4]  Cheng-Yen Lin,et al.  Optimized memory access support for data layout conversion on heterogeneous multi-core systems , 2014, 2014 IEEE 12th Symposium on Embedded Systems for Real-time Multimedia (ESTIMedia).

[5]  Chen-Ting Chao,et al.  Sparse-Matrix Compression Primitives with OpenCL Framework to Support Halide , 2019, IWOCL.

[6]  Jenq Kuen Lee,et al.  Support and optimization for parallel sparse programs with array intrinsics of Fortran 90 , 2004, Parallel Comput..

[7]  Christopher Potts,et al.  Learning Word Vectors for Sentiment Analysis , 2011, ACL.

[8]  Chen-Ting Chao,et al.  Accelerate DNN Performance with Sparse Matrix Compression in Halide , 2019, ICPP Workshops.

[9]  Barbara Plank,et al.  Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , 2011 .

[10]  Frédo Durand,et al.  Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines , 2013, PLDI 2013.

[11]  Jimeng Sun,et al.  Efficient and effective sparse tensor reordering , 2019, ICS.

[12]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[13]  Tomas Mikolov,et al.  Enriching Word Vectors with Subword Information , 2016, TACL.

[14]  Srinivasan Parthasarathy,et al.  Efficient sparse-matrix multi-vector product on GPUs , 2018, HPDC.

[15]  Matthijs Douze,et al.  FastText.zip: Compressing text classification models , 2016, ArXiv.

[16]  Jenq Kuen Lee,et al.  Parallel Sparse Supports for Array Intrinsic Functions of Fortran 90 , 2001, The Journal of Supercomputing.

[17]  Tomas Mikolov,et al.  Bag of Tricks for Efficient Text Classification , 2016, EACL.

[18]  Jenq Kuen Lee,et al.  Probabilistic inference schemes for sparsity structures of Fortran 90 array intrinsics , 2001, International Conference on Parallel Processing, 2001..