Collective Model Fusion for Multiple Black-Box Experts

Model fusion is a fundamental problem in collective machine learning (ML) where independent experts with heterogeneous learning architectures are required to combine expertise to improve predictive performance. This is particularly challenging in information-sensitive domains where experts do not have access to each other’s internal architecture and local data. This paper presents the first collective model fusion framework for multiple experts with heterogeneous black-box architectures. The proposed method will enable this by addressing the key issues of how black-box experts interact to understand the predictive behaviors of one another; how these understandings can be represented and shared efficiently among themselves; and how the shared understandings can be combined to generate high-quality consensus prediction. The performance of the resulting framework is analyzed theoretically and demonstrated empirically on several datasets.

[1]  Gaurav S. Sukhatme,et al.  Decentralized Data Fusion and Active Sensing with Mobile Sensors for Modeling and Predicting Spatiotemporal Traffic Phenomena , 2012, UAI.

[2]  Haitao Liu,et al.  Generalized Robust Bayesian Committee Machine for Large-scale Gaussian Process Regression , 2018, ICML.

[3]  Miguel Lázaro-Gredilla,et al.  Variational Inference for Mahalanobis Distance Metrics in Gaussian Process Regression , 2013, NIPS.

[4]  Yasaman Khazaeni,et al.  Bayesian Nonparametric Federated Learning of Neural Networks , 2019, ICML.

[5]  Marc Peter Deisenroth,et al.  Distributed Gaussian Processes , 2015, ICML.

[6]  Kian Hsiang Low,et al.  Parallel Gaussian Process Regression for Big Data: Low-Rank Representation Meets Markov Approximation , 2014, AAAI.

[7]  Kian Hsiang Low,et al.  Gaussian process decentralized data fusion meets transfer learning in large-scale distributed cooperative perception , 2017, Autonomous Robots.

[8]  Kian Hsiang Low,et al.  A Unifying Framework of Anytime Sparse Gaussian Process Regression Models with Stochastic Variational Inference for Big Data , 2015, ICML.

[9]  Kian Hsiang Low,et al.  Decentralized High-Dimensional Bayesian Optimization with Factor Graphs , 2017, AAAI.

[10]  Christopher M. Gifford.  Collective Machine Learning: Team Learning and Classification in Multi-Agent Systems , 2009 .

[11]  Mohan S. Kankanhalli,et al.  Scalable Decision-Theoretic Coordination and Control for Real-time Active Multi-Camera Surveillance , 2014, ICDSC.

[12]  Kian Hsiang Low,et al.  A Distributed Variational Inference Framework for Unifying Parallel Sparse Gaussian Process Regression Models , 2016, ICML.

[13]  Carl E. Rasmussen,et al.  A Unifying View of Sparse Approximate Gaussian Process Regression , 2005, J. Mach. Learn. Res..

[14]  Kian Hsiang Low,et al.  Parallel Gaussian Process Regression with Low-Rank Covariance Matrix Approximations , 2013, UAI.

[15]  Kian Hsiang Low,et al.  A Generalized Stochastic Variational Bayesian Hyperparameter Learning Framework for Sparse Spectrum Gaussian Process Regression , 2016, AAAI.

[16]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[17]  Kian Hsiang Low,et al.  Gaussian Process-Based Decentralized Data Fusion and Active Sensing for Mobility-on-Demand System , 2013, Robotics: Science and Systems.

[18]  Kian Hsiang Low,et al.  Collective Online Learning of Gaussian Processes in Massive Multi-Agent Systems , 2019, AAAI.

[19]  Neil D. Lawrence,et al.  Gaussian Processes for Big Data , 2013, UAI.

[20]  Blaise Agüera y Arcas,et al.  Communication-Efficient Learning of Deep Networks from Decentralized Data , 2016, AISTATS.

[21]  A. P. Dawid,et al.  Regression and Classification Using Gaussian Process Priors , 2009 .

[22]  Sergey Levine,et al.  Collective robot reinforcement learning with distributed asynchronous guided policy search , 2016, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[23]  Girish Chowdhary,et al.  Communication efficient decentralized Gaussian Process Fusion for multi-UAS path planning , 2017, 2017 American Control Conference (ACC).