ResFlow: Multi-tasking of Sequentially Pooling Spatiotemporal Features for Action Recognition and Optical Flow Estimation

Since deep-learning-based method has been widely-used and is capable of generating generic model, most existing methods about action recognition use either two-stream structure, considering spatial and temporal features separately, or C3D, costing lots of prices in memory and time. We aim to design a robust system to extract spatiotemporal features with aggregation mechanism to integrate local features in temporal order. In light of this, we propose ResFlow to estimate optical flow and predict action recognition simultaneously. Leveraging the characteristic of optical flow estimation, we extract spatiotemporal feature via an autoencoder. Via a novel Sequentially Pooling Mechanism which literally pool global spatiotemporal feature sequentially, we extract spatiotemporal feature at each time and aggregate these local features into global feature. This design use only RGB images as input with temporal information encoded, pre-trained by optical flow, and sequentially aggregate spatiotemporal features in high efficiency. We evaluate our ability of estimating optical flow on FlyingChairs dataset and show the promising results of action recognition on UCF-101 dataset through a series of experiments.