Tracking the 3D position and orientation of flying swarms with learned kinematic pattern using LSTM network

Accurately and reliably tracking the 3D position and orientation of individuals in large flying swarms is valuable not only for scientific researches but also practical applications. However, large quantity, frequent occlusions, similar appearance, tiny body size and abrupt motion make it remain an open problem. The 3D flying swarm tracking method proposed in this paper tracks both position and orientation of each individual in the swarm using the particle filter framework. Particles are scattered more pertinently by the dynamic model based on the learned kinematic pattern of a single target with a Long Short-Term Memory (LSTM) network. In addition, the observation model combines the Weighted Occupancy Ratio (WOR) and Temporal Appearance Coherency (TAC) cues in each view to improve the accuracy and robustness of the reconstructed body orientation. Experiments on both simulation and real-world data sets demonstrate the effectiveness and superiority of the proposed method.