Deep Model Compression and Inference Speedup of Sum–Product Networks on Tensor Trains

Sum–product networks (SPNs) constitute an emerging class of neural networks with clear probabilistic semantics and superior inference speed over other graphical models. This brief reveals an important connection between SPNs and tensor trains (TTs), leading to a new canonical form which we call tensor SPNs (tSPNs). Specifically, we demonstrate the intimate relationship between a valid SPN and a TT. For the first time, through mapping an SPN onto a tSPN and employing specially customized optimization techniques, we demonstrate improvements up to a factor of 100 on both model compression and inference speedup for various data sets with negligible loss in accuracy.

[1]  Franz Pernkopf,et al.  Modeling speech with sum-product networks: Application to bandwidth extension , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2]  Franz Pernkopf,et al.  On the Latent Variable Interpretation in Sum-Product Networks , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Adnan Darwiche,et al.  A differential approach to inference in Bayesian networks , 2000, JACM.

[4]  Pedro M. Domingos,et al.  Discriminative Learning of Sum-Product Networks , 2012, NIPS.

[5]  Haesun Park,et al.  Algorithms for nonnegative matrix and tensor factorizations: a unified view based on block coordinate descent framework , 2014, J. Glob. Optim..

[6]  Xiaogang Wang,et al.  A Deep Sum-Product Architecture for Robust Facial Attributes Analysis , 2013, 2013 IEEE International Conference on Computer Vision.

[7]  Andrzej Cichocki,et al.  Tensor Decompositions for Signal Processing Applications: From two-way to multiway component analysis , 2014, IEEE Signal Processing Magazine.

[8]  Rajesh P. N. Rao,et al.  Deep Spatial Affordance Hierarchy : Spatial Knowledge Representation for Planning in Large-scale Environments , 2017 .

[9]  Yixin Chen,et al.  Compressing Neural Networks with the Hashing Trick , 2015, ICML.

[10]  Pascal Poupart,et al.  A Unified Approach for Learning the Parameters of Sum-Product Networks , 2016, NIPS.

[11]  Babak Hassibi,et al.  Second Order Derivatives for Network Pruning: Optimal Brain Surgeon , 1992, NIPS.

[12]  Carsten Binnig,et al.  Automatic Mapping of the Sum-Product Network Inference Problem to FPGA-Based Accelerators , 2018, 2018 IEEE 36th International Conference on Computer Design (ICCD).

[13]  Sebastian Tschiatschek,et al.  Sum-Product Networks for Sequence Labeling , 2018, ArXiv.

[14]  Ivan Oseledets,et al.  Expressive power of recurrent neural networks , 2017, ICLR.

[15]  S. Kay.  Fundamentals of statistical signal processing: estimation theory , 1993 .

[16]  Mohamed R. Amer,et al.  Sum Product Networks for Activity Recognition , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Song Han,et al.  Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding , 2015, ICLR.

[18]  Song Han,et al.  Learning both Weights and Connections for Efficient Neural Network , 2015, NIPS.

[19]  Rajesh P. N. Rao,et al.  Learning Graph-Structured Sum-Product Networks for Probabilistic Semantic Maps , 2017, AAAI.

[20]  Kim Batselier,et al.  Fast and Accurate Tensor Completion With Total Variation Regularized Tensor Trains , 2018, IEEE Transactions on Image Processing.

[21]  Mohamed R. Amer,et al.  Sum-product networks for modeling activities with stochastic structure , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Gang Wang,et al.  Hierarchical Spatial Sum–Product Networks for Action Recognition in Still Images , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[23]  Ivan Oseledets,et al.  Tensor-Train Decomposition , 2011, SIAM J. Sci. Comput..

[24]  Pedro M. Domingos,et al.  Sum-product networks: A new deep architecture , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[25]  Ronen Tamari,et al.  Analysis and Design of Convolutional Networks via Hierarchical Tensor Decompositions , 2017, ArXiv.

[26]  Johan A. K. Suykens,et al.  Parallelized Tensor Train Learning of Polynomial Classifiers , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[27]  Hao Wang,et al.  Modeling spatial layout for scene image understanding via a novel multiscale sum-product network , 2016, Expert Syst. Appl..

[28]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[29]  Pedro M. Domingos,et al.  Learning the Structure of Sum-Product Networks , 2013, ICML.

[30]  Vibhav Gogate,et al.  Merging Strategies for Sum-Product Networks: From Trees to Graphs , 2016, UAI.

[31]  Floriana Esposito,et al.  Simplifying, Regularizing and Strengthening Sum-Product Network Structure Learning , 2015, ECML/PKDD.

[32]  Ivan V. Oseledets,et al.  Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition , 2014, ICLR.

[33]  Christoph Schnörr,et al.  Locally Adaptive Probabilistic Models for Global Segmentation of Pathological OCT Scans , 2017, MICCAI.

[34]  Ngai Wong,et al.  Tensor Network alternating linear scheme for MIMO Volterra system identification , 2016, Autom..

[35]  Ali Farhadi,et al.  XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks , 2016, ECCV.

[36]  Rui Peng,et al.  Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures , 2016, ArXiv.

[37]  Alexander Novikov,et al.  Tensorizing Neural Networks , 2015, NIPS.

[38]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[39]  Yoshua Bengio,et al.  BinaryConnect: Training Deep Neural Networks with binary weights during propagations , 2015, NIPS.