Discovering Hidden Features with Gaussian Processes Regression

In Gaussian process regression the covariance between the outputs at input locations x and x′ is usually assumed to depend on the distance (x− x′) W (x− x′), where W is a positive definite matrix. W is often taken to be diagonal, but if we allow W to be a general positive definite matrix which can be tuned on the basis of training data, then an eigen-analysis of W shows that we are effectively creating hidden features, where the dimensionality of the hidden-feature space is determined by the data. We demonstrate the superiority of predictions using the general matrix over those based on a diagonal matrix on two test problems.