Event-coupled hidden Markov models

Inferences from time series data can be greatly enhanced by taking into account multiple modalities. In some cases, such as audio of speech and the corresponding video of lip gestures, the different time series are tightly coupled. We are interested in loosely-coupled time series where only the onset of events are coupled in time. We present an extension of the forward-backward algorithm that can be used for inference and learning in event-coupled hidden Markov models and give results on a simplified multimedia indexing task where the objective is to detect an event whose onset is loosely coupled in audio and video.