Learning Sparse Multiple Cause Models

Multiple cause models (MCM) are a way to describe patterns as a superposition of a selection of cause patterns. In contrast to clustering methods and dimensionality reduction, multiple cause models are capable of turning local features on and off and this makes them a more realistic model for many types of data. However, inference and learning in general multiple cause models takes an amount of time that is exponential in the number of causes. We present an approximate inference algorithm that examines only sparse cause patterns, i.e., those configurations of causes where only a small number of causes are active at a time. This leads to an approximate EM algorithm that maximizes a lower bound on the likelihood of a data set. We show that this sparse multiple cause model can model different types of human facial expression patterns. Performance comparison of the MCM classifier with the SNoW (sparse network of winnows) architecture and the nearest neighbor classifier reveals significant improvement in classification accuracy using the MCM classifier.