Supervised Learning of Unsupervised Learning Rules

Supervised learning has proven extremely effective for many problems where large amounts of labeled training data are available. There is a common hope that unsupervised learning will prove similarly powerful in situations where labels are expensive or impractical to collect, or where the prediction target is unknown during training. However, unsupervised learning has yet to fulfill this promise. One explanation for this failure is that unsupervised training rules are typically mismatched to the target task. Ideally, learned representations should linearly expose high level attributes of data (e.g. object identity) and perform well in semi-supervised settings. However, many current unsupervised objectives optimize for objectives such as log-likelihood of a generative model or reconstruction error, and produce useful representations only as a side effect.