Overview of Screen Content Video Coding: Technologies, Standards, and Beyond

This paper presents recent advances in screen content video coding, with an emphasis on two state-of-the-art standards: HEVC/H.265 Screen Content Coding Extensions (HEVC-SCC) by ISO/IEC Moving Picture Experts Group and ITU-T Video Coding Experts Group, and Display Stream Compression (DSC) by Video Electronics Standards Association. The HEVC-SCC enhances the capabilities of HEVC in coding screen content, while DSC provides lightweight compression for display links. Although targeting different application domains, they share some design principles and are expected to become the leading formats in the marketplace in the coming years. This paper provides a brief account of their background, key elements, performance, and complexity characteristics, according to their final specifications. As we survey these standards, we also summarize prior arts in the last decade and explore future research opportunities and standards developments in order to give a comprehensive overview of this field.

[1]  Sanghoon Lee,et al.  Structural similarity based image compression for LCD overdrive , 2012, IEEE Transactions on Consumer Electronics.

[2]  Satoshi Goto,et al.  Reducing power consumption of HEVC codec with lossless reference frame recompression , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[3]  Jintao Li,et al.  Classification-Based Adaptive Compression Method for Computer Screen Image , 2012, 2012 IEEE International Conference on Multimedia and Expo Workshops.

[4]  Wei Wang,et al.  Low-complexity dictionary based lossless screen content coding , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[5]  Shipeng Li,et al.  Virtualized Screen: A Third Element for Cloud-Mobile Convergence , 2011, IEEE Multim..

[6]  Vivienne Sze,et al.  Rotate intra block copy for still image coding , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[7]  Shan Liu,et al.  Screen content coding using non-square intra block copy for HEVC , 2014, 2014 IEEE International Conference on Multimedia and Expo (ICME).

[8]  C. Rubinstein,et al.  On the Design of Quantizers for DPCM Coders: A Functional Relationship Between Visibility, Probability and Masking , 1978, IEEE Trans. Commun..

[9]  Zhou Wang,et al.  Subjective quality assessment of Screen Content Images , 2014, 2014 Sixth International Workshop on Quality of Multimedia Experience (QoMEX).

[10]  Wei Wang,et al.  Advanced Screen Content Coding Using Color Table and Index Map , 2014, IEEE Transactions on Image Processing.

[11]  Detlev Marpe,et al.  Adaptive cross-component prediction for 4:4:4 high efficiency video coding , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[12]  Baocai Yin,et al.  2-D Dictionary Based Video Coding for Screen Contents , 2014, 2014 Data Compression Conference.

[13]  Hoon Yoo,et al.  A low complexity and lossless frame memory compression for display devices , 2008, IEEE Transactions on Consumer Electronics.

[14]  Jong-Wha Chong,et al.  A Hybrid Image Coding in Overdriving for Motion Blur Reduction in LCD , 2007, ICEC.

[15]  Feng Wu,et al.  Enable Efficient Compound Image Compression in H.264/AVC Intra Coding , 2007, 2007 IEEE International Conference on Image Processing.

[16]  Do-Kyoung Kwon,et al.  Fast intra block copy (IntraBC) search for HEVC screen content coding , 2014, 2014 IEEE International Symposium on Circuits and Systems (ISCAS).

[17]  S. A. Martucci,et al.  Reversible compression of HDTV images using median adaptive prediction and arithmetic coding , 1990, IEEE International Symposium on Circuits and Systems.

[18]  Benoit M. Macq,et al.  Quality and Error Robustness Assessment of Low-Latency Lightweight Intra-Frame Codecs , 2016, 2016 Data Compression Conference (DCC).

[19]  Tao Lin,et al.  Cloudlet-screen computing: a client-server architecture with top graphics performance , 2013, Int. J. Ad Hoc Ubiquitous Comput..

[20]  Shuhui Wang,et al.  A Unified LZ and Hybrid Coding for Compound Image Partial-Lossless Compression , 2009, 2009 2nd International Congress on Image and Signal Processing.

[21]  Shan Liu,et al.  Overview of HEVC extensions on screen content coding , 2015, APSIPA Transactions on Signal and Information Processing.

[22]  Tao Lin,et al.  United coding for compound image compression , 2010, 2010 3rd International Congress on Image and Signal Processing.

[23]  Youn-Long Lin,et al.  A Hybrid Algorithm for Effective Lossless Compression of Video Display Frames , 2012, IEEE Transactions on Multimedia.

[24]  Yifu Li,et al.  Hybrid Pixel Encoding: An Effective Display Frame Compression Algorithm for HD Video Decoder , 2012, 2012 IEEE 15th International Conference on Computational Science and Engineering.

[25]  Xuecheng Zou,et al.  Research of an image map encoding algorithm on frame buffer , 2007, 2007 7th International Conference on ASIC.

[26]  Jianle Chen,et al.  Adaptive Color-Space Transform for HEVC Screen Content Coding , 2015, 2015 Data Compression Conference.

[27]  Wen-Hsiao Peng,et al.  Intra Line Copy for HEVC Screen Content Intra-Picture Prediction , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[28]  Debin Zhao,et al.  Arbitrary-sized motion detection in screen video coding , 2013, 2013 IEEE International Conference on Image Processing.

[29]  David M. Hoffman,et al.  A new standard method of subjective assessment of barely visible image artifacts and a new public database , 2014 .

[30]  Marta Mrak,et al.  Improving screen content coding in HEVC by transform skipping , 2012, 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO).

[31]  Harald Nautsch,et al.  Transform coding of compound images using matching pursuit , 2012, 2012 Picture Coding Symposium.

[32]  Ying Chen,et al.  Standardized Extensions of High Efficiency Video Coding (HEVC) , 2013, IEEE Journal of Selected Topics in Signal Processing.

[33]  Do-Kyoung Kwon,et al.  Intra motion compensation and entropy coding improvements for HEVC screen content coding , 2013, 2013 Picture Coding Symposium (PCS).

[34]  Alexander MacInnis,et al.  27.4L: Late-News Paper: VESA Display Stream Compression: An Overview , 2014 .

[35]  Pengwei Hao,et al.  Compound image compression for real-time computer screen image transmission , 2005, IEEE Transactions on Image Processing.

[36]  Tao Lin,et al.  Compound image compression based on unified LZ and hybrid coding , 2013, IET Image Process..

[37]  Lai-Man Po,et al.  Block-Matching Translational and Rotational Motion Compensated Prediction Using Interpolated Reference Frame , 2010, EURASIP J. Adv. Signal Process..

[38]  Rik Van de Walle,et al.  Guided Chroma Reconstruction for Screen Content Coding , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[39]  Gary J. Sullivan,et al.  Lifting-based reversible color transformations for image compression , 2008, Optical Engineering + Applications.

[40]  Satoshi Goto,et al.  A New Reference Frame Recompression Algorithm and Its VLSI Architecture for UHDTV Video Codec , 2014, IEEE Transactions on Multimedia.

[41]  John Platt,et al.  20.4: Displaced Filtering for Patterned Displays , 2000 .

[42]  Guangming Shi,et al.  Compress Compound Images in H.264/MPGE-4 AVC by Exploiting Spatial Correlation , 2010, IEEE Transactions on Image Processing.

[43]  Marta Mrak,et al.  Improving inter prediction in HEVC with residual DPCM for lossless screen content coding , 2013, 2013 Picture Coding Symposium (PCS).

[44]  Shiqi Wang,et al.  Study on subjective quality assessment of Screen Content Images , 2015, 2015 Picture Coding Symposium (PCS).

[45]  C.-C. Jay Kuo,et al.  Screen content coding for HEVC using edge modes , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[46]  Massoud Pedram,et al.  A compressed frame buffer to reduce display power consumption in mobile systems , 2004 .

[47]  Jong-Wha Chong,et al.  High performance overdrive using improved motion adaptive codec in LCD , 2009, IEEE Transactions on Consumer Electronics.

[48]  Shipeng Li,et al.  Browser-friendly hybrid codec for compound image compression , 2011, 2011 IEEE International Symposium of Circuits and Systems (ISCAS).

[49]  Jong-Wha Chong,et al.  Adaptive multi-level block truncation coding for frame memory reduction in LCD overdrive , 2010, IEEE Transactions on Consumer Electronics.

[50]  Doaa Mohammed.  Image Compression Using Block Truncation Coding , 2011 .

[51]  Weisi Lin,et al.  Learning based screen image compression , 2012, 2012 IEEE 14th International Workshop on Multimedia Signal Processing (MMSP).

[52]  Oscar C. Au,et al.  HEVC-based adaptive quantization for screen content by detecting low contrast edge regions , 2013, 2013 IEEE International Symposium on Circuits and Systems (ISCAS2013).

[53]  Rico Malvar,et al.  YCoCg-R: A Color Space with RGB Reversibility and Low Dynamic Range , 2003 .

[54]  Gregory K. Wallace,et al.  The JPEG still picture compression standard , 1992 .

[55]  Gary J. Sullivan,et al.  Overview of the High Efficiency Video Coding (HEVC) Standard , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[56]  Feng Wu,et al.  Compression of compound images by combining several strategies , 2011, 2011 IEEE 13th International Workshop on Multimedia Signal Processing.

[57]  Baocai Yin,et al.  Screen Content Coding Based on HEVC Framework , 2014, IEEE Transactions on Multimedia.

[58]  Eduardo A. B. da Silva,et al.  Scanned Compound Document Encoding Using Multiscale Recurrent Patterns , 2010, IEEE Transactions on Image Processing.

[59]  Wei Wang,et al.  2-D Index Map Coding for HEVC Screen Content Compression , 2015, 2015 Data Compression Conference.

[60]  Jizheng Xu,et al.  Overview of the Emerging HEVC Screen Content Coding Extension , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[61]  Alan C. Bovik,et al.  Image information and visual quality , 2006, IEEE Trans. Image Process..

[62]  Sung-Jea Ko,et al.  Vector quantizer based block truncation coding for color image compression in LCD overdrive , 2008, IEEE Transactions on Consumer Electronics.

[63]  Yifu Li,et al.  Frame Buffer Compression without Color Information Loss , 2012, 2012 IEEE 12th International Conference on Computer and Information Technology.

[64]  David Flynn,et al.  Spatial transform skip in the emerging High Efficiency Video Coding standard , 2012, 2012 19th IEEE International Conference on Image Processing.

[65]  Ajay Luthra,et al.  Overview of the H.264/AVC video coding standard , 2003, IEEE Trans. Circuits Syst. Video Technol..

[66]  Salih Dikbas,et al.  Lossless image compression using adjustable fractional line-buffer , 2010, Signal Process. Image Commun..

[67]  Tao Lin,et al.  Mixed Chroma Sampling-Rate High Efficiency Video Coding for Full-Chroma Screen Content , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[68]  Tao Lin,et al.  Pseudo 2D String Matching Technique for High Efficiency Screen Content Coding , 2016, IEEE Transactions on Multimedia.

[69]  Baocai Yin,et al.  Hash-Based Block Matching for Screen Content Coding , 2015, IEEE Transactions on Multimedia.

[70]  Wen-Hsiao Peng,et al.  Intra line copy for HEVC screen content coding , 2014, Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2014 Asia-Pacific.

[71]  Youn-Long Lin,et al.  An effective dictionary-based display frame compressor , 2009, 2009 IEEE/ACM/IFIP 7th Workshop on Embedded Systems for Real-Time Multimedia.

[72]  Baining Guo,et al.  A novel cloud gaming framework using joint video and graphics streaming , 2014, 2014 IEEE International Conference on Multimedia and Expo (ICME).

[73]  Yifu Li,et al.  Adaptive Pixel Encoding: An Effective Algorithm for Frame Buffer Compression , 2012, 2012 IEEE 12th International Conference on Computer and Information Technology.

[74]  Tao Lin,et al.  Arbitrary shape matching for screen content coding , 2013, 2013 Picture Coding Symposium (PCS).

[75]  Nenghai Yu,et al.  A Low-Complexity Screen Compression Scheme for Interactive Screen Sharing , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[76]  Bin Li,et al.  Rate control for screen content coding in HEVC , 2015, 2015 IEEE International Symposium on Circuits and Systems (ISCAS).

[77]  Hyuk-Jae Lee,et al.  A Block Truncation Coding Algorithm and Hardware Implementation Targeting 1/12 Compression for LCD Overdrive , 2016, Journal of Display Technology.

[78]  Baocai Yin,et al.  Adaptive LZMA-based coding for screen content , 2013, 2013 Picture Coding Symposium (PCS).

[79]  Zhenyu Liu,et al.  Lossless Frame Memory Compression Using Pixel-Grain Prediction and Dynamic Order Entropy Coding , 2016, IEEE Transactions on Circuits and Systems for Video Technology.

[80]  Jewon Kang,et al.  Explicit residual DPCM for screen contents coding , 2014, The 18th IEEE International Symposium on Consumer Electronics (ISCE 2014).