Spatiotemporal visual saliency guided perceptual high efficiency video coding with neural network

Abstract The perceptual video coding systems for optimization have been developed on the basis of different attributes of the human visual system. The attention-based coding system is considered as an important part of it. The saliency map method representing the region-of-interest (ROI) from the video signal has become a reliable method due to advances in the computer performance and the visual algorithms. In the present study, we propose a hybrid compression algorithm that uses the deep convolutional neural network to compute the spatial saliency followed by extraction of the temporal saliency from the compressed-domain motion information. The level of uncertainty is calculated to combine to form the video's saliency map. Afterwards, the QP search range is dynamically adjusted in HEVC, and a rate distortion calculation method is proposed to choose the pattern and guide the allocation of bits during the video compression process. Empirical reporting results proved the superiority of the proposed method over the state-of-the-art perceptual coding algorithms in terms of saliency detection and perceptual compression quality.