Perceptual Learned Video Compression with Recurrent Conditional GAN

This paper proposes a Perceptual Learned Video Compression (PLVC) approach with recurrent conditional GAN. In our approach, we employ the recurrent auto-encoder-based compression network as the generator, and the most importantly, we propose a recurrent conditional discriminator, which judges raw and compressed video conditioned on both spatial and temporal features, including the latent representation, temporal motion and hidden states in recurrent cells. This way, in the adversarial training, it pushes the generated video to be not only spatially photo-realistic but also temporally consistent with groundtruth and coherent among video frames. The experimental results show that the proposed PLVC model learns to compress video towards good perceptual quality at low bit-rate, and outperforms the official HEVC test model (HM 16.20) and the previous learned approaches on several perceptual quality metrics and user studies. The codes will be released at the project page: https://github.com/RenYang-home/PLVC.