Predictive models on fire have been increasingly popular in computer image analysis. Due to late strides of deep learning techniques, we are now unprecedently benefited from its flexible applicability. In most cases, however, the conventional algorithms are limited to only single-framed images unlike sequence data that inevitably entails heavy computational time and memory. In this paper, we propose an effective algorithm exploiting the combination of CNNs (convolution neural networks) and RNNs (recurrent neural networks) in a consecutive way so that sequence data can be allowed for the model. The LSTM (long short-term memory) is well-known to be superior to other RNNtype algorithms in accuracy, especially when applying to sequence data. In our extensive experiments, where fire videos (e.g. indoor fire, forest fire) and non-fire videos collected from a range of scenarios are taken into accounts, it is confirmed that our propose methods are found outstanding in predictive power.