Towards Training Probabilistic Topic Models on Neuromorphic Multi-chip Systems

Probabilistic topic models are popular unsupervised learning methods, including probabilistic latent semantic indexing (pLSI) and latent Dirichlet allocation (LDA). By now, their training is implemented on general purpose computers (GPCs), which are flexible in programming but energy-consuming. Towards low-energy implementations, this paper investigates their training on an emerging hardware technology called the neuromorphic multi-chip systems (NMSs). NMSs are very effective for a family of algorithms called spiking neural networks (SNNs). We present three SNNs to train topic models. The first SNN is a batch algorithm combining the conventional collapsed Gibbs sampling (CGS) algorithm and an inference SNN to train LDA. The other two SNNs are online algorithms targeting at both energy- and storage-limited environments. The two online algorithms are equivalent with training LDA by using maximum-a-posterior estimation and maximizing the semi-collapsed likelihood, respectively. They use novel, tailored ordinary differential equations for stochastic optimization. We simulate the new algorithms and show that they are comparable with the GPC algorithms, while being suitable for NMS implementation. We also propose an extension to train pLSI and a method to prune the network to obey the limited fan-in of some NMSs.

[1]  Alexei A. Efros,et al.  Discovering object categories in image collections , 2005 .

[2]  Mark Steyvers,et al.  Finding scientific topics , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[3]  O. Cappé,et al.  On‐line expectation–maximization algorithm for latent data models , 2009 .

[4]  Tie-Yan Liu,et al.  LightLDA: Big Topic Models on Modest Computer Clusters , 2014, WWW.

[5]  Gert Cauwenberghs,et al.  Event-driven contrastive divergence for spiking neuromorphic systems , 2013, Front. Neurosci..

[6]  Aaron Q. Li,et al.  Fast Latent Variable Models for Inference and Visualization on Mobile Devices , 2015, ArXiv.

[7]  Yee Whye Teh,et al.  Stochastic Gradient Riemannian Langevin Dynamics on the Probability Simplex , 2013, NIPS.

[8]  Andrew S. Cassidy,et al.  A million spiking-neuron integrated circuit with a scalable communication network and interface , 2014, Science.

[9]  Yang Gao,et al.  Towards Topic Modeling for Big Data , 2014, ArXiv.

[10]  H. Kushner,et al.  Stochastic Approximation and Recursive Algorithms and Applications , 2003 .

[11]  Chih-Jen Lin,et al.  LIBLINEAR: A Library for Large Linear Classification , 2008, J. Mach. Learn. Res..

[12]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[13]  Tobi Delbruck,et al.  Real-time classification and sensor fusion with a spiking deep belief network , 2013, Front. Neurosci..

[14]  Andre Wibisono,et al.  Streaming Variational Bayes , 2013, NIPS.

[15]  Matthew D. Zeiler.  ADADELTA: An Adaptive Learning Rate Method , 2012, ArXiv.

[16]  Eric P. Xing,et al.  MedLDA: maximum margin supervised topic models , 2012, J. Mach. Learn. Res..

[17]  Wolfgang Maass,et al.  Bayesian Computation Emerges in Generic Cortical Microcircuits through Spike-Timing-Dependent Plasticity , 2013, PLoS Comput. Biol..

[18]  Steve B. Furber,et al.  The SpiNNaker Project , 2014, Proceedings of the IEEE.

[19]  Hui Xiong,et al.  An unsupervised approach to modeling personalized contexts of mobile users , 2010, 2010 IEEE International Conference on Data Mining.

[20]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[21]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[22]  Chong Wang,et al.  Stochastic variational inference , 2012, J. Mach. Learn. Res..

[23]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[24]  John Aitchison,et al.  The Statistical Analysis of Compositional Data , 1986 .

[25]  Yee Whye Teh,et al.  On Smoothing and Inference for Topic Models , 2009, UAI.

[26]  Wenguang Chen,et al.  WarpLDA: a Cache Efficient O(1) Algorithm for Latent Dirichlet Allocation , 2015, Proc. VLDB Endow..

[27]  Misha A. Mahowald,et al.  An Analog VLSI System for Stereoscopic Vision , 1994 .

[28]  Thomas Hofmann,et al.  Probabilistic Latent Semantic Indexing , 1999, SIGIR Forum.