A Unifying Framework of Anytime Sparse Gaussian Process Regression Models with Stochastic Variational Inference for Big Data

This paper presents a novel unifying framework of anytime sparse Gaussian process regression (SGPR) models that can produce good predictive performance fast and improve their predictive performance over time. Our proposed unifying framework reverses the variational inference procedure to theoretically construct a non-trivial, concave functional that is maximized at the predictive distribution of any SGPR model of our choice. As a result, a stochastic natural gradient ascent method can be derived that involves iteratively following the stochastic natural gradient of the functional to improve its estimate of the predictive distribution of the chosen SGPR model and is guaranteed to achieve asymptotic convergence to it. Interestingly, we show that if the predictive distribution of the chosen SGPR model satisfies certain decomposability conditions, then the stochastic natural gradient is an unbiased estimator of the exact natural gradient and can be computed in constant time (i.e., independent of data size) at each iteration. We empirically evaluate the trade-off between the predictive performance vs. time efficiency of the anytime SGPR models on two real-world million-sized datasets.

[1]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[2]  Michalis K. Titsias,et al.  Variational Learning of Inducing Variables in Sparse Gaussian Processes , 2009, AISTATS.

[3]  Carl E. Rasmussen,et al.  A Unifying View of Sparse Approximate Gaussian Process Regression , 2005, J. Mach. Learn. Res..

[4]  Manola Brunet,et al.  Daily Mean Sea Level Pressure Reconstructions for the European–North Atlantic Region for the Period 1850–2003 , 2006 .

[5]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[6]  Mohan S. Kankanhalli,et al.  Active Learning Is Planning: Nonmyopic ε-Bayes-Optimal Active Learning of Gaussian Processes , 2014, ECML/PKDD.

[7]  Chong Wang,et al.  Stochastic variational inference , 2012, J. Mach. Learn. Res..

[8]  Kian Hsiang Low,et al.  Parallel Gaussian Process Regression for Big Data: Low-Rank Representation Meets Markov Approximation , 2014, AAAI.

[9]  Kian Hsiang Low,et al.  Decentralized active robotic exploration and mapping for probabilistic field classification in environmental sensing , 2012, AAMAS.

[10]  Neil D. Lawrence,et al.  Fast Forward Selection to Speed Up Sparse Gaussian Process Regression , 2003, AISTATS.

[11]  Anton Schwaighofer,et al.  Transductive and Inductive Methods for Approximate Gaussian Process Regression , 2002, NIPS.

[12]  Alexander J. Smola,et al.  Sparse Greedy Gaussian Process Regression , 2000, NIPS.

[13]  Kian Hsiang Low,et al.  Multi-robot active sensing of non-stationary gaussian process-based environmental phenomena , 2014, AAMAS.

[14]  Kian Hsiang Low,et al.  Parallel Gaussian Process Regression with Low-Rank Covariance Matrix Approximations , 2013, UAI.

[15]  Kian Hsiang Low,et al.  Multi-robot informative path planning for active sensing of environmental phenomena: a tale of two algorithms , 2013, AAMAS.

[16]  Alberto Elfes,et al.  Cooperative aquatic sensing using the telesupervised adaptive ocean sensor fleet , 2009, Remote Sensing.

[17]  Zoubin Ghahramani,et al.  Sparse Gaussian Processes using Pseudo-inputs , 2005, NIPS.

[18]  Kian Hsiang Low,et al.  Gaussian Process Decentralized Data Fusion and Active Sensing for Spatiotemporal Traffic Modeling and Prediction in Mobility-on-Demand Systems , 2015, IEEE Transactions on Automation Science and Engineering.

[19]  Mohan S. Kankanhalli,et al.  Nonmyopic \(\epsilon\)-Bayes-Optimal Active Learning of Gaussian Processes , 2014, ICML.

[20]  Edward Lloyd Snelson,et al.  Flexible and efficient Gaussian process models for machine learning , 2007 .

[21]  Kian Hsiang Low,et al.  Adaptive multi-robot wide-area exploration and mapping , 2008, AAMAS.

[22]  Kian Hsiang Low,et al.  Telesupervised remote surface water quality sensing , 2010, 2010 IEEE Aerospace Conference.

[23]  Carl E. Rasmussen,et al.  Sparse Spectrum Gaussian Process Regression , 2010, J. Mach. Learn. Res..

[24]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[25]  Kian Hsiang Low,et al.  Gaussian Process-Based Decentralized Data Fusion and Active Sensing for Mobility-on-Demand System , 2013, Robotics: Science and Systems.

[26]  Gaurav S. Sukhatme,et al.  Decentralized Data Fusion and Active Sensing with Mobile Sensors for Modeling and Predicting Spatiotemporal Traffic Phenomena , 2012, UAI.

[27]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[28]  Carl E. Rasmussen,et al.  Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models , 2014, NIPS.

[29]  Kian Hsiang Low,et al.  Active Markov information-theoretic path planning for robotic environmental sensing , 2011, AAMAS.

[30]  Kian Hsiang Low,et al.  GP-Localize: Persistent Mobile Robot Localization using Online Sparse Gaussian Process Observation Model , 2014, AAAI.

[31]  Neil D. Lawrence,et al.  Gaussian Processes for Big Data , 2013, UAI.

[32]  Kian Hsiang Low,et al.  Generalized Online Sparse Gaussian Processes with Application to Persistent Mobile Robot Localization , 2014, ECML/PKDD.

[33]  Zoubin Ghahramani,et al.  Local and global sparse Gaussian process approximations , 2007, AISTATS.

[34]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[35]  Kian Hsiang Low,et al.  Hierarchical Bayesian Nonparametric Approach to Modeling and Learning the Wisdom of Crowds of Urban Traffic Route Planning Agents , 2012, 2012 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology.

[36]  Kian Hsiang Low,et al.  Recent Advances in Scaling Up Gaussian Process Predictive Models for Large Spatiotemporal Data , 2014, DyDESS.

[37]  Kian Hsiang Low,et al.  Information-Theoretic Approach to Efficient Adaptive Path Planning for Mobile Robotic Environmental Sensing , 2009, ICAPS.