Transductive Regression for Data With Latent Dependence Structure

Analyzing data with latent spatial and/or temporal structure is a challenge for machine learning. In this paper, we propose a novel nonlinear model for studying data with latent dependence structure. It successfully combines the concepts of Markov random fields, transductive learning, and regression, making heavy use of the notion of joint feature maps. Our transductive conditional random field regression model is able to infer the latent states by combining limited labeled data of high precision with unlabeled data containing measurement uncertainty. In this manner, we can propagate accurate information and greatly reduce uncertainty. We demonstrate the usefulness of our novel framework on generated time series data with the known temporal structure and successfully validate it on synthetic as well as real-world offshore data with the spatial structure from the oil industry to predict rock porosities from acoustic impedance data.

[1]  F. Leisch,et al.  FlexMix Version 2: Finite Mixtures with Concomitant Variables and Varying and Constant Parameters , 2008 .

[2]  Jiliang Tang,et al.  Mobile Location Prediction in Spatio-Temporal Context , 2012 .

[3]  Le Thi Hoai An,et al.  The DC (Difference of Convex Functions) Programming and DCA Revisited with DC Models of Real World Nonconvex Optimization Problems , 2005, Ann. Oper. Res..

[4]  Mikhail Belkin,et al.  Laplacian Support Vector Machines Trained in the Primal , 2009, J. Mach. Learn. Res..

[5]  Gunnar Rätsch,et al.  An introduction to kernel-based learning algorithms , 2001, IEEE Trans. Neural Networks.

[6]  J. MacQueen.  Some methods for classification and analysis of multivariate observations , 1967 .

[7]  Christoph H. Lampert,et al.  Learning to Localize Objects with Structured Output Regression , 2008, ECCV.

[8]  Mecit Cetin,et al.  Short-Term Traffic Flow Prediction with Regime Switching Models , 2006 .

[9]  Philippe Marie Doyen,et al.  Porosity from seismic data: A geostatistical approach , 1988 .

[10]  Liefeng Bo,et al.  Structured output-associative regression , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[12]  Peter Robinson,et al.  Dimensional affect recognition using Continuous Conditional Random Fields , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[13]  Mehryar Mohri,et al.  On Transductive Regression , 2006, NIPS.

[14]  Thorsten Joachims,et al.  Learning structural SVMs with latent variables , 2009, ICML '09.

[15]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[16]  A. E. Hoerl,et al.  Ridge regression: biased estimation for nonorthogonal problems , 2000 .

[17]  F. Leisch.  FlexMix: A general framework for finite mixture models and latent class regression in R , 2004 .

[18]  Geoffrey E. Hinton,et al.  Adaptive Mixtures of Local Experts , 1991, Neural Computation.

[19]  Alan L. Yuille,et al.  The Concave-Convex Procedure , 2003, Neural Computation.

[20]  Nathan Ratliff,et al.  Online) Subgradient Methods for Structured Prediction , 2007 .

[21]  Vladimir Vovk,et al.  Empirical Inference , 2011, Springer Berlin Heidelberg.

[22]  R. M. Srivastava,et al.  Integrating Seismic Data in Reservoir Modeling: The Collocated Cokriging Alternative , 1992 .

[23]  Jason Weston,et al.  Transductive Inference for Estimating Values of Functions , 1999, NIPS.

[24]  Klaus-Robert Müller,et al.  Annealed Competition of Experts for a Segmentation and Classification of Switching Dynamics , 1996, Neural Computation.

[25]  Gunnar Rätsch,et al.  Efficient Training of Graph-Regularized Multitask SVMs , 2012, ECML/PKDD.

[26]  Alexander J. Smola,et al.  Support Vector Regression Machines , 1996, NIPS.

[27]  Peter Robinson,et al.  Continuous Conditional Neural Fields for Structured Regression , 2014, ECCV.

[28]  Alexander Binder,et al.  Learning and Evaluation in Presence of Non-i.i.d. Label Noise , 2014, AISTATS.

[29]  Michael I. Jordan,et al.  Variational inference in graphical models: The view from the marginal polytope , 2008 .

[30]  Thomas Hofmann,et al.  Large Margin Methods for Structured and Interdependent Output Variables , 2005, J. Mach. Learn. Res..

[31]  William T. Freeman,et al.  Correctness of Belief Propagation in Gaussian Graphical Models of Arbitrary Topology , 1999, Neural Computation.

[32]  Klaus-Robert Müller,et al.  Efficient Algorithms for Exact Inference in Sequence Labeling SVMs , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[33]  J. Besag.  Spatial Interaction and the Statistical Analysis of Lattice Systems , 1974 .

[34]  Alexander Binder,et al.  Extracting latent brain states — Towards true labels in cognitive neuroscience experiments , 2015, NeuroImage.

[35]  Vincent S. Tseng,et al.  A pre-processing method to deal with missing values by integrating clustering and regression techniques , 2003, Appl. Artif. Intell..

[36]  Mikhail Belkin,et al.  Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples , 2006, J. Mach. Learn. Res..

[37]  Clayton V. Deutsch,et al.  Geostatistical Reservoir Modeling , 2002 .

[38]  Hiroo Kanamori,et al.  Earthquake Prediction: An Overview , 2003 .

[39]  Mikhail Belkin,et al.  Beyond the point cloud: from transductive to semi-supervised learning , 2005, ICML.

[40]  Santosh S. Venkatesh,et al.  Learning from a mixture of labeled and unlabeled examples with parametric side information , 1995, COLT '95.

[41]  Jian Peng,et al.  Conditional Neural Fields , 2009, NIPS.

[42]  Daniel M. Tartakovsky,et al.  Probabilistic reconstruction of geologic facies , 2004 .

[43]  Minyoung Kim,et al.  Semi-supervised learning of hidden conditional random fields for time-series classification , 2013, Neurocomputing.