When crowds hold privileges: Bayesian unsupervised representation learning with oracle constraints

Representation learning systems typically rely on massive amounts of labeled data in order to be trained effectively. Recently, high-dimensional parametric models like convolutional neural networks have succeeded in building rich representations using either compressive, reconstructive or supervised criteria. However, the semantic structure inherent in observations is oftentimes lost in the process. Human perception excels at understanding semantics but cannot always be expressed in terms of labels. Human-in-the-loop systems like crowdsourcing are often employed to generate similarity constraints using an implicit similarity function encoded in human perception. We propose to combine generative unsupervised feature learning with learning from similarity orderings in order to learn models which take advantage of privileged information coming from the crowd. We use a fast variational algorithm to learn the model on standard datasets and demonstrate applicability to two image datasets, where classification is drastically improved. We show how triplet-samples of the crowd can supplement labels as a source of information to shape latent spaces with rich semantic information.