Multi-View Dimensionality Reduction via Canonical Correlation Analysis

We analyze the multi-view regression problem where we have two views X = (X, X) of the input data and a target variableY of interest. We provide sufficient conditions under which we can reduce the dimensionality of X (via a projection) without loosing predictive power of Y . Crucially, this projection can be computed via a Canonical Correlation Analysis only on the unlabeled data. The algorithmic template is as f ollows: with unlabeled data, perform CCA and construct a certain projection; with the labeled data, do least squa res regression in this lower dimensional space. We show how, under certain natural assumptions, the number o f labeled samples could be significantly reduced (in comparison to the single view setting) — in particular, we show how this dimen sionality reduction does not loose predictive power ofY (thus it only introduces little bias but could drastically reduce the variance). We explore two separate assumptions under which this is possible and show how, under either assumption alone, dimensionality reduction could reduce the labeled sample complexity. The two assumptions we consider are a conditional independenceassumption and a redundancyassumption. The typical conditional independence assumption is that conditioned onY the viewsX andX are independent — we relax this assumption to: conditioned on some hidden state H the viewsX andX are independent. Under the redundancy assumption, we show that the best predictor from each view is roughly as good as the best predictor u sing both views.