Over the past decade there has been considerable interest in spectral algorithms for learning Predictive State Representations (PSRs). Spectral algorithms have appealing theoretical guarantees; however, the resulting models do not always perform well on inference tasks in practice. One reason for this behavior is the mismatch between the intended task (accurate filtering or prediction) and the loss function being optimized by the algorithm (estimation error in model parameters).
A natural idea is to improve performance by refining PSRs using an algorithm such as EM. Unfortunately it is not obvious how to apply apply an EM style algorithm in the context of PSRs as the Log Likelihood is not well defined for all PSRs. We show that it is possible to overcome this problem using ideas from Predictive State Inference Machines.
We combine spectral algorithms for PSRs as a consistent and efficient initialization with PSIM-style updates to refine the resulting model parameters. By combining these two ideas we develop Inference Gradients, a simple, fast, and robust method for practical learning of PSRs. Inference Gradients performs gradient descent in the PSR parameter space to optimize an inference-based loss function like PSIM. Because Inference Gradients uses a spectral initialization we get the same consistency benefits as PSRs. We show that Inference Gradients outperforms both PSRs and PSIMs on real and synthetic data sets.
[1]
Byron Boots,et al.
Predictive State Temporal Difference Learning
,
2010,
NIPS.
[2]
Geoffrey J. Gordon,et al.
Supervised Learning for Dynamical System Learning
,
2015,
NIPS.
[3]
New York Dover,et al.
ON THE CONVERGENCE PROPERTIES OF THE EM ALGORITHM
,
1983
.
[4]
Byron Boots,et al.
Learning to Filter with Predictive State Inference Machines
,
2015,
ICML.
[5]
Martial Hebert,et al.
Learning message-passing inference machines for structured prediction
,
2011,
CVPR 2011.
[6]
John Langford,et al.
Learning nonlinear dynamic models
,
2009,
ICML '09.
[7]
Byron Boots,et al.
Online Instrumental Variable Regression with Applications to Online Linear System Identification
,
2016,
AAAI.
[8]
Richard S. Sutton,et al.
Predictive Representations of State
,
2001,
NIPS.
[9]
Michael R. James,et al.
Predictive State Representations: A New Theory for Modeling Dynamical Systems
,
2004,
UAI.
[10]
Nan Jiang,et al.
Improving Predictive State Representations via Gradient Descent
,
2016,
AAAI.
[11]
Byron Boots,et al.
Closing the learning-planning loop with predictive state representations
,
2009,
Int. J. Robotics Res..
[12]
Le Song,et al.
Learning Latent Variable Models by Improving Spectral Solutions with Exterior Point Method
,
2015,
UAI.
[13]
Herbert Jaeger,et al.
Observable Operator Models for Discrete Stochastic Time Series
,
2000,
Neural Computation.