Detection of falls is very important from a health and safety perspective. However, falls occur rarely and infrequently, which leads to either limited or no training data and thus can severely impair the performance of supervised activity recognition algorithms. In this paper, we address the problem of identification of falls in the absence of training data for falls, but with abundant training data for normal activities. We propose two ‘X-Factor’ Hidden Markov Model (XHMMs) approaches that are like normal HMMs, but have “inflated” output covariances (observation models), which can be estimated using cross-validation on the set of ‘outliers’ in the normal data that serve as proxies for the (unseen) fall data. This allows the XHMMs to be learned from only normal activity data. We tested the proposed XHMM approaches on two real activity recognition datasets that show high detection rates for falls in the absence of training data.