HFR Projector Camera Based Visible Light Communication System for Real-Time Video Streaming

This study develops a projector–camera-based visible light communication (VLC) system for real-time broadband video streaming, in which a high frame rate (HFR) projector can encode and project a color input video sequence into binary image patterns modulated at thousands of frames per second and an HFR vision system can capture and decode these binary patterns into the input color video sequence with real-time video processing. For maximum utilization of the high-throughput transmission ability of the HFR projector, we introduce a projector–camera VLC protocol, wherein a multi-level color video sequence is binary-modulated with a gray code for encoding and decoding instead of pure-code-based binary modulation. Gray code encoding is introduced to address the ambiguity with mismatched pixel alignments along the gradients between the projector and vision system. Our proposed VLC system consists of an HFR projector, which can project 590 × 1060 binary images at 1041 fps via HDMI streaming and a monochrome HFR camera system, which can capture and process 12-bit 512 × 512 images in real time at 3125 fps; it can simultaneously decode and reconstruct 24-bit RGB video sequences at 31 fps, including an error correction process. The effectiveness of the proposed VLC system was verified via several experiments by streaming offline and live video sequences.

[1]  Jingwen Dai,et al.  Embedding imperceptible codes into video projection and applications in robotics , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[2]  S. Kiravittaya,et al.  A Comprehensive Lighting Configuration for Efficient Indoor Visible Light Communication Networks , 2016 .

[3]  Anthony Rowe,et al.  Visual light landmarks for mobile devices , 2014, IPSN-14 Proceedings of the 13th International Symposium on Information Processing in Sensor Networks.

[4]  Fumihito Arai,et al.  Red blood cell fatigue evaluation based on the close-encountering point between extensibility and recoverability. , 2014, Lab on a chip.

[5]  Stefan Schmid,et al.  Connecting networks of toys and smartphones with visible light communication , 2014, IEEE Communications Magazine.

[6]  Nianyu Zou,et al.  Research on indoor visible light communication system employing white LED lightings , 2011 .

[7]  Prabal Dutta,et al.  Luxapose: indoor positioning with mobile phones and visible light , 2014, MobiCom.

[8]  Takeshi Takaki,et al.  Color-histogram-based tracking at 2000 fps , 2012, J. Electronic Imaging.

[9]  Gordon Wetzstein,et al.  The Visual Computing of Projector‐Camera Systems , 2008, SIGGRAPH '08.

[10]  Min Wang,et al.  Efficient coding modulation and seamless rate adaptation for visible light communications , 2015, IEEE Wireless Communications.

[11]  Shoji Kawahito,et al.  Image-sensor-based visible light communication for automotive applications , 2014, IEEE Communications Magazine.

[12]  Takeshi Takaki,et al.  High-Frame-Rate Optical Flow System , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[13]  L. J. Hornbeck,et al.  Digital Light Processing and MEMS: an overview , 1996, Digest IEEE/Leos 1996 Summer Topical Meeting. Advanced Applications of Lasers in Materials and Processing.

[14]  Myungsik Yoo,et al.  Performance Analysis of Visible Light Communication Using CMOS Sensors , 2016, Sensors.

[15]  Jack M. Younse.  Projection display systems based on the Digital Micromirror Device (DMD) , 1995, Photonics West - Micro and Nano Fabricated Electromechanical and Optical Components.

[16]  Junyi Li,et al.  Visible light communication: opportunities, challenges and the path to market , 2013, IEEE Communications Magazine.

[17]  Toshiaki Fujii,et al.  Interpixel interference cancellation method for road-to-vehicle visible light communication , 2013, 2013 IEEE 5th International Symposium on Wireless Vehicular Communications (WiVeC).

[18]  Tadayoshi Aoyama,et al.  Real-Time Vibration Source Tracking Using High-Speed Vision , 2017, IEEE Sensors Journal.

[19]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[20]  Idaku Ishii,et al.  Development of high-speed and real-time vision platform, H3 vision , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[21]  Reinhard Koch,et al.  Fast projector-camera calibration for interactive projection mapping , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[22]  Tadayoshi Aoyama,et al.  Motion-blur-free microscopic video shooting based on frame-by-frame intermittent tracking , 2015, 2015 IEEE International Conference on Automation Science and Engineering (CASE).

[23]  Hua Yang,et al.  Dynamics-Based Stereo Visual Inspection Using Multidimensional Modal Analysis , 2013, IEEE Sensors Journal.

[24]  Mohsen Kavehrad,et al.  VLC-based indoor positioning system with tracking capability using Kalman and particle filters , 2014, 2014 IEEE International Conference on Consumer Electronics (ICCE).

[25]  Naoki Nakamura,et al.  A full-pixel optical flow system using a GPU-based high-frame-rate vision , 2015, AIR '15.

[26]  Nan Chi,et al.  Reversed Three-Dimensional Visible Light Indoor Positioning Utilizing Annular Receivers with Multi-Photodiodes , 2016, Sensors.

[27]  Hiromasa Oku,et al.  Tracking a protozoon using high-speed visual feedback , 2000, 1st Annual International IEEE-EMBS Special Topic Conference on Microtechnologies in Medicine and Biology. Proceedings (Cat. No.00EX451).

[28]  Cong Wang,et al.  SBVLC: Secure barcode-based visible light communication for smartphones , 2014, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[29]  Kevin Jerome,et al.  Indoor Navigation Using Visible Light Communication , 2014, 2014 Texas Instruments India Educators' Conference (TIIEC).

[30]  Shoji Kawahito,et al.  Optical Vehicle-to-Vehicle Communication System Using LED Transmitter and Camera Receiver , 2014, IEEE Photonics Journal.

[31]  Hao Gao,et al.  High-Frame-Rate Structured Light 3-D Vision for Fast Moving Objects , 2014, J. Robotics Mechatronics.

[32]  Xia Zhou,et al.  Human Sensing Using Visible Light Communication , 2015, MobiCom.

[33]  Hua Yang,et al.  Vibration Sensing of a Bridge Model Using a Multithread Active Vision System , 2017, IEEE/ASME Transactions on Mechatronics.

[34]  Masatoshi Ishikawa,et al.  955-fps Real-time Shape Measurement of a Moving/Deforming Object using High-speed Vision for Numerous-point Analysis , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[35]  K. Sindhubala,et al.  Ecofriendly data transmission in visible light communication , 2015, Proceedings of the 2015 Third International Conference on Computer, Communication, Control and Information Technology (C3IT).

[36]  Sridhar Rajagopal,et al.  IEEE 802.15.7 visible light communication: modulation schemes and dimming support , 2012, IEEE Communications Magazine.

[37]  Dina Katabi,et al.  PixNet: LCD-camera pairs as communication links , 2010, SIGCOMM '10.

[38]  Dr. Robert J. Gove.  DMD Display Systems : The Impact of an All-Digital Display , 2002 .

[39]  Xiongkuo Min,et al.  DLP based anti-piracy display system , 2014, 2014 IEEE Visual Communications and Image Processing Conference.

[40]  Jun Chen,et al.  Simultaneous projection mapping using high-frame-rate depth vision , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[41]  Rajendran Parthiban,et al.  Dimming schemes for visible light communication: the state of research , 2015, IEEE Wireless Communications.

[42]  Jing Xu,et al.  High-accuracy, high-speed 3D structured light imaging techniques and potential applications to intelligent robotics , 2017, International Journal of Intelligent Robotics and Applications.

[43]  Myungsik Yoo,et al.  An in-Depth Survey of Visible Light Communication Based Positioning Systems , 2016, Sensors.

[44]  Serdar Yilmaz,et al.  An overview of visible light communication systems , 2015, ArXiv.

[45]  Shingo Kagami,et al.  High-speed vision systems and projectors for real-time perception of the world , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[46]  Trang Nguyen,et al.  High-speed asynchronous Optical Camera Communication using LED and rolling shutter camera , 2015, 2015 Seventh International Conference on Ubiquitous and Future Networks.

[47]  Harald Haas,et al.  Performance Comparison of MIMO Techniques for Optical Wireless Communications in Indoor Environments , 2013, IEEE Transactions on Communications.

[48]  Toshiaki Fujii,et al.  Detection of LED traffic light by image processing for visible light communication system , 2009, 2009 IEEE Intelligent Vehicles Symposium.

[49]  C. Chow,et al.  Enhancement of Signal Performance in LED Visible Light Communications Using Mobile Phone Camera , 2015, IEEE Photonics Journal.

[50]  Takeshi Takaki,et al.  2000 fps real-time vision system with high-frame-rate video recording , 2010, 2010 IEEE International Conference on Robotics and Automation.

[51]  Shoji Kawahito,et al.  LED and CMOS Image Sensor Based Optical Wireless Communication System for Automotive Applications , 2013, IEEE Photonics Journal.

[52]  Takeshi Takaki,et al.  Motion-Blur-Free High-Speed Video Shooting Using a Resonant Mirror , 2017, Sensors.

[53]  Hung-Yu Chen,et al.  Visible Light Communication Using Receivers of Camera Image Sensor and Solar Cell , 2016, IEEE Photonics Journal.

[54]  Tadayoshi Aoyama,et al.  Simultaneous Vision-Based Shape and Motion Analysis of Cells Fast-Flowing in a Microchannel , 2015, IEEE Transactions on Automation Science and Engineering.

[55]  Tadayoshi Aoyama,et al.  Pixel-Level and Robust Vibration Source Sensing in High-Frame-Rate Video Analysis , 2016, Sensors.

[56]  Masao Nakagawa,et al.  Fundamental analysis for visible-light communication system using LED lights , 2004, IEEE Transactions on Consumer Electronics.

[57]  Thai-Chien Bui,et al.  Demonstration of using camera communication based infrared LED for uplink in indoor visible light communication , 2016, 2016 IEEE Sixth International Conference on Communications and Electronics (ICCE).

[58]  Gabriel Taubin,et al.  Unsynchronized structured light , 2015, ACM Trans. Graph..

[59]  Navin Kumar,et al.  Led-Based Visible Light Communication System: A Brief Survey and Investigation , 2010 .

[60]  Guoliang Xing,et al.  COBRA: color barcode streaming for smartphone systems , 2012, MobiSys '12.

[61]  Masatoshi Ishikawa,et al.  Dynamic Projection Mapping onto Deforming Non-Rigid Surface Using Deformable Dot Cluster Marker , 2017, IEEE Transactions on Visualization and Computer Graphics.

[62]  Zabih Ghassemlooy,et al.  Data detection for Smartphone visible light communications , 2014, 2014 9th International Symposium on Communication Systems, Networks & Digital Sign (CSNDSP).

[63]  Tadayoshi Aoyama,et al.  Real-Time Image Mosaicing System Using a High-Frame-Rate Video Sequence , 2015, J. Robotics Mechatronics.

[64]  Shida Sayaka,et al.  A 1ms High-Speed Vision Chip with 3D-Stacked 140GOPS Column-Parallel PEs for Spatio-Temporal Image Processing , 2017 .

[65]  Larry J. Hornbeck,et al.  Digital Light Processing and MEMS: Timely Convergence for a Bright Future , 2002 .

[66]  Idaku Ishii,et al.  A Self-Projected Light-Section Method for Fast Three-Dimensional Shape Inspection , 2012 .

[67]  Guobin Shen,et al.  InFrame: Multiflexing Full-Frame Visible Communication Channel for Humans and Devices , 2014, HotNets.

[68]  Tadayoshi Aoyama,et al.  LOC-Based High-Throughput Cell Morphology Analysis System , 2015, IEEE Transactions on Automation Science and Engineering.

[69]  Harald Haas,et al.  Using a CMOS camera sensor for visible light communication , 2012, 2012 IEEE Globecom Workshops.

[70]  Masatoshi Ishikawa,et al.  Real-time high-speed motion blur compensation system based on back-and-forth motion control of galvanometer mirror. , 2015, Optics express.

[71]  Wenjun Hu,et al.  LightSync: unsynchronized visual communication over screen-camera links , 2013, MobiCom.

[72]  Guobin Shen,et al.  InFrame++: Achieve Simultaneous Screen-Human Viewing and Hidden Screen-Camera Communication , 2015, MobiSys.

[73]  Masatoshi Ishikawa,et al.  Development of Motion-Blur-Compensated High-Speed Moving Visual Inspection Vehicle for Tunnels , 2016 .

[74]  Koichi Hashimoto,et al.  3,000-fps 3-D shape measurement using a high-speed camera-projector system , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[75]  Xiang Zhao,et al.  Maximum likelihood estimation of vehicle position for outdoor image sensor-based visible light positioning system , 2016 .