Ensemble Learning for Relational Data

We present a theoretical analysis framework for relational ensemble models. We show that ensembles of collective classifiers can improve predictions for graph data by reducing errors due to variance in both learning and inference. In addition, we propose a relational ensemble framework that combines a relational ensemble learning approach with a relational ensemble inference approach for collective classification. The proposed ensemble techniques are applicable for both single and multiple graph settings. Experiments on both synthetic and real-world data demonstrate the effectiveness of the proposed framework. Finally, our experimental results support the theoretical analysis and confirm that ensemble algorithms that explicitly focus on both learning and inference processes and aim at reducing errors associated with both, are the best performers.

[1]  Christos Faloutsos,et al.  Using ghost edges for classification in sparsely labeled networks , 2008, KDD.

[2]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[3]  Kagan Tumer,et al.  Analysis of decision boundaries in linearly combined neural classifiers , 1996, Pattern Recognit..

[4]  Kristian Kersting,et al.  Gradient-based boosting for statistical relational learning: The relational dependency network case , 2011, Machine Learning.

[5]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[6]  Sofus A. Macskassy.  Improving Learning in Networked Data by Combining Explicit and Mined Links , 2007, AAAI.

[7]  David D. Jensen,et al.  Exploiting relational structure to understand publication patterns in high-energy physics , 2003, SKDD.

[8]  Weiqiang Dong.  On Bias , Variance , 0 / 1-Loss , and the Curse of Dimensionality RK April 13 , 2014 .

[9]  Yoav Freund,et al.  Boosting the margin: A new explanation for the effectiveness of voting methods , 1997, ICML.

[10]  Pedro M. Domingos.  A Unified Bias-Variance Decomposition for Zero-One and Squared Loss , 2000, AAAI/IAAI.

[11]  Gareth James,et al.  Variance and Bias for General Loss Functions , 2003, Machine Learning.

[12]  Padraig Cunningham,et al.  Diversity versus Quality in Classification Ensembles Based on Feature Selection , 2000, ECML.

[13]  Bernhard Schölkopf,et al.  Fast protein classification with multiple networks , 2005, ECCB/JBI.

[14]  Lars Schmidt-Thieme,et al.  Ensembles of relational classifiers , 2008, Knowledge and Information Systems.

[15]  Ron Kohavi,et al.  Option Decision Trees with Majority Votes , 1997, ICML.

[16]  William W. Cohen,et al.  Stacked Graphical Models for Efficient Inference in Markov Random Fields , 2007, SDM.

[17]  Geoffrey J. Gordon,et al.  Relational learning via collective matrix factorization , 2008, KDD.

[18]  Leo Breiman,et al.  Bias, Variance , And Arcing Classifiers , 1996 .

[19]  Jennifer Neville,et al.  Leveraging relational autocorrelation with latent group models , 2005, Fifth IEEE International Conference on Data Mining (ICDM'05).

[20]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[21]  Jennifer Neville,et al.  A bias/variance decomposition for models using collective inference , 2008, Machine Learning.

[22]  Yizhou Sun,et al.  Heterogeneous source consensus learning via decision propagation and negotiation , 2009, KDD.

[23]  Jennifer Neville,et al.  Using relational knowledge discovery to prevent securities fraud , 2005, KDD '05.

[24]  David D. Jensen,et al.  Why Stacked Models Perform Effective Collective Classification , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[25]  Roger Th. A. J. Leenders,et al.  Modeling social influence through network autocorrelation: constructing the weight matrix , 2002, Soc. Networks.

[26]  Avrim Blum,et al.  The Bottleneck , 2021, Monopsony Capitalism.

[27]  J. Ross Quinlan,et al.  Bagging, Boosting, and C4.5 , 1996, AAAI/IAAI, Vol. 1.

[28]  Ray L. Somorjai,et al.  Application of Several Methods of Classification Fusion to Magnetic Resonance Spectra , 1996, Connect. Sci..

[29]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[30]  Philip S. Yu,et al.  Learning from Heterogeneous Sources via Gradient Boosting Consensus , 2012, SDM.

[31]  Jennifer Neville,et al.  Relational Dependency Networks , 2007, J. Mach. Learn. Res..

[32]  Nicholas Kushmerick,et al.  Iterative Ensemble Classification for Relational Data: A Case Study of Semantic Web Services , 2004, ECML.

[33]  Ben Taskar,et al.  Multi-View Learning over Structured and Non-Identical Outputs , 2008, UAI.

[34]  Kumar Sricharan,et al.  Multi-source fusion for anomaly detection: using across-domain and across-time peer-group consistency checks , 2014, J. Wirel. Mob. Networks Ubiquitous Comput. Dependable Appl..

[35]  Andrew McCallum,et al.  Introduction to Statistical Relational Learning , 2007 .

[36]  Zenglin Xu,et al.  Web page classification with heterogeneous data fusion , 2007, WWW '07.

[37]  Jennifer Neville,et al.  Modeling relationship strength in online social networks , 2010, WWW '10.

[38]  Eric Bauer,et al.  An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants , 1999, Machine Learning.

[39]  Jennifer Neville,et al.  Understanding Propagation Error and Its Effect on Collective Classification , 2011, 2011 IEEE 11th International Conference on Data Mining.

[40]  Alneu de Andrade Lopes,et al.  Graph-based cross-validated committees ensembles , 2012, 2012 Fourth International Conference on Computational Aspects of Social Networks (CASoN).

[41]  David W. Opitz,et al.  An Empirical Evaluation of Bagging and Boosting , 1997, AAAI/IAAI.

[42]  Yanqing Zhang,et al.  Genetic fuzzy classification fusion of multiple SVMs for biomedical data , 2007, J. Intell. Fuzzy Syst..

[43]  Steven Salzberg,et al.  JIGSAW: integration of multiple sources of evidence for gene prediction , 2005, Bioinform..

[44]  Luc De Raedt,et al.  Logical and relational learning , 2008, Cognitive Technologies.

[45]  Yizhou Sun,et al.  Graph-based Consensus Maximization among Multiple Supervised and Unsupervised Models , 2009, NIPS.

[46]  Nello Cristianini,et al.  A statistical framework for genomic data fusion , 2004, Bioinform..

[47]  Masashi Sugiyama,et al.  Integration of Multiple Networks for Robust Label Propagation , 2008, SDM.

[48]  David W. Aha,et al.  Semi-Supervised Collective Classification via Hybrid Label Regularization , 2012, ICML.

[49]  Lars Schmidt-Thieme,et al.  Relational Ensemble Classification , 2006, Sixth International Conference on Data Mining (ICDM'06).