A Hardware/Software Co-design Approach for Real-Time Binocular Stereo Vision Based on ZYNQ (Short Paper)

Based on the ZYNQ platform, this paper proposes a hardware/software co-design approach, and implements a binocular stereo vision system with high real-time performance and good human-computer interaction, which can be used to assist advanced driver assistance systems to improve driving safety. Combining the application characteristics of binocular stereo vision, the approach firstly modularizes the system’s functions to perform hardware/software partitioning, accelerates the data processing on FPGA, and performs the data control on ARM cores; then uses the ARM instruction set to configure the registers within FPGA to design relevant interfaces to complete the data interaction between hardware and software; finally, combines the implementation of specific algorithms and logical control to complete the binocular stereo vision system. The test results show that the frame rate with an image resolution of 640 * 480 can reach 121.43 frames per second when the FPGA frequency is 100M, and the frame rate is also high for large resolution images. At the same time, the system can achieve real-time display and human-computer interaction with the control of the graphical user interface.

[1]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[2]  Zhengyou Zhang,et al.  Flexible camera calibration by viewing a plane from unknown orientations , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[3]  Yi Yang,et al.  Lane Detection and Classification for Forward Collision Warning System Based on Stereo Vision , 2018, IEEE Sensors Journal.

[4]  Chen Yao-yu.  Binocular omni-directional vision sensor and epipolar rectification in its omni-directional images , 2011 .

[5]  Azra Fetic,et al.  The procedure of a camera calibration using Camera Calibration Toolbox for MATLAB , 2012, 2012 Proceedings of the 35th International Convention MIPRO.

[6]  Keshav Bimbraw,et al.  Autonomous cars: Past, present and future a review of the developments in the last century, the present scenario and the expected future of autonomous vehicle technology , 2015, 2015 12th International Conference on Informatics in Control, Automation and Robotics (ICINCO).

[7]  Rajesh Gupta,et al.  Hardware/software co-design , 1996, Proc. IEEE.

[8]  Stephen A. Edwards,et al.  Design of embedded systems: formal models, validation, and synthesis , 1997, Proc. IEEE.

[9]  Xing Mei,et al.  On building an accurate stereo matching system on graphics hardware , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[10]  Donald E. Thomas,et al.  The design of mixed hardware/software systems , 1996, DAC '96.

[11]  Jae Wook Jeon,et al.  FPGA Design and Implementation of a Real-Time Stereo Vision System , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[12]  Wayne Wolf,et al.  Hardware-software co-design of embedded systems , 1994, Proc. IEEE.

[13]  Andreas Geiger,et al.  Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art , 2017, Found. Trends Comput. Graph. Vis..

[14]  John Iselin Woodfill,et al.  Tyzx DeepSea High Speed Stereo Vision System , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[15]  Qidan Zhu,et al.  Camera Calibration in Binocular Stereo Vision of Moving Robot , 2006, 2006 6th World Congress on Intelligent Control and Automation.

[16]  Stefania Perri,et al.  Design of Real-Time FPGA-based Embedded System for Stereo Vision , 2018, 2018 IEEE International Symposium on Circuits and Systems (ISCAS).

[17]  Madaín Pérez Patricio,et al.  FPGA implementation of an efficient similarity-based adaptive window algorithm for real-time stereo matching , 2019, Journal of Real-Time Image Processing.