Motion Segmentation using EM - a short tutorial

The expectation-maximization algorithm (EM) is used for many estimation problems in statistics. Here we give a short tutorial on how to program a segmentation algorithm using EM. Those interested in the theory or in more advanced versions of the algorithm should consult the references at the end. Suppose we are given a set of datapoints that were generated by multiple processes, for example two lines. We need to estimate two things: (1) the parameters (slope and intercept) of the two lines and (2) the assignment of each datapoint to the process that generated it. The intuition behind EM is that each of these steps is easy assuming the other one is solved. That is, assuming we know the assignment of each datapoint, then we can estimate the parameters of each line by taking into consideration only those points assigned to it. Likewise, if we know the parameters of the lines we can assign each point to the line that ts it best. This gives the basic structure of an EM algorithm: