Camera Tracking on Focal-Plane Sensor-Processor Arrays

Focal-Plane Sensor-Processor (FPSP) Arrays are new sensors with parallel image capturing and processing capabilities built into one device. This design eliminates the need for image transfer and thus increases frame rate significantly. Additionally, recent FPSP which are based on analogue technology, consume much less power compared to conventional digital cameras. However, implementing well-known pose estimation and camera tracking algorithms on FPSP is a challenging problem because the processing elements on FPSP have very limited resources. FPSP also require a different programming paradigm. ∗ This paper presents several contributions to camera tracking on FPSP, using limited instruction sets and memory available for each pixel. We demonstrate that FPSP are able to estimate camera pose at very high-frame rates with low power consumption and high accuracy. Simulated and real-world experiments demonstrate the effectiveness of the proposed methods.

[1]  Davide Scaramuzza,et al.  Real-time Visual-Inertial Odometry for Event Cameras using Keyframe-based Nonlinear Optimization , 2017, BMVC.

[2]  Piotr Dudek.  Implementation of SIMD vision chip with 128/spl times/128 array of analogue processing elements , 2005, 2005 IEEE International Symposium on Circuits and Systems.

[3]  Lin-Bao Yang,et al.  Cellular neural networks: theory , 1988 .

[4]  Kostas Daniilidis,et al.  Monocular visual odometry in urban environments using an omnidirectional camera , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[5]  Davide Scaramuzza,et al.  Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High-Speed Scenarios , 2017, IEEE Robotics and Automation Letters.

[6]  Chiara Bartolozzi,et al.  Fast Event-based Corner Detection , 2017, BMVC.

[7]  R. Goecke,et al.  Visual Vehicle Egomotion Estimation using the Fourier-Mellin Transform , 2007, 2007 IEEE Intelligent Vehicles Symposium.

[8]  S. Espejo,et al.  Architectural and Basic Circuit Considerations for a Flexible 128 × 128 Mixed-Signal SIMD Vision Chip , 2002 .

[9]  Davide Scaramuzza,et al.  Event-Based, 6-DOF Camera Tracking from Photometric Depth Maps , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Ángel Rodríguez-Vázquez,et al.  A 0.8-μm CMOS two-dimensional programmable mixed-signal focal-plane array processor with on-chip binary imaging and instructions storage , 1997, IEEE J. Solid State Circuits.

[11]  Jörg Conradt,et al.  Simultaneous Localization and Mapping for Event-Based Vision Systems , 2013, ICVS.

[12]  Piotr Dudek,et al.  An analogue SIMD focal-plane processor array , 2001, ISCAS 2001. The 2001 IEEE International Symposium on Circuits and Systems (Cat. No.01CH37196).

[13]  Peter I. Corke,et al.  Omnidirectional visual odometry for a planetary rover , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[14]  P. Dudek,et al.  A cellular processor array simulation and hardware prototyping tool , 2008, 2008 11th International Workshop on Cellular Neural Networks and Their Applications.

[15]  Bin Wang,et al.  A 100,000 fps vision sensor with embedded 535GOPS/W 256×256 SIMD processor array , 2013, 2013 Symposium on VLSI Circuits.

[16]  Daniel Cremers,et al.  Real-time visual odometry from dense RGB-D images , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[17]  Piotr Dudek,et al.  A CMOS general-purpose sampled-data analog processing element , 2000 .

[18]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[19]  Larry H. Matthies,et al.  Visual odometry on the Mars Exploration Rovers , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[20]  James R. Bergen,et al.  Visual odometry , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[21]  Gordon Wyeth,et al.  Single camera vision-only SLAM on a suburban road network , 2008, 2008 IEEE International Conference on Robotics and Automation.

[22]  Piotr Dudek,et al.  Visual Odometry for Pixel Processor Arrays , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[23]  Ari Paasio,et al.  MIPA4k: A 64×64 cell mixed-mode image processor array , 2009, 2009 IEEE International Symposium on Circuits and Systems.

[24]  Piotr Dudek,et al.  A general-purpose processor-per-pixel analog SIMD vision chip , 2005, IEEE Transactions on Circuits and Systems I: Regular Papers.

[25]  Jörg Conradt,et al.  On-board real-time optic-flow for miniature event-based vision sensors , 2015, 2015 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[26]  Davide Scaramuzza,et al.  Low-latency event-based visual odometry , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[27]  Davide Scaramuzza,et al.  EMVS: Event-Based Multi-View Stereo—3D Reconstruction with an Event Camera in Real-Time , 2017, International Journal of Computer Vision.

[28]  Stefan Leutenegger,et al.  Simultaneous Optical Flow and Intensity Estimation from an Event Camera , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Ryad Benosman,et al.  Simultaneous Mosaicing and Tracking with an Event Camera , 2014, BMVC.

[30]  Bin Wang,et al.  Locating high speed multiple objects using a SCAMP-5 vision-chip , 2012, 2012 13th International Workshop on Cellular Nanoscale Networks and their Applications.

[31]  Davide Scaramuzza,et al.  Event-based, 6-DOF pose tracking for high-speed maneuvers , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[32]  Servando Espejo,et al.  Ultra-high frame rate focal plane image sensor and processor , 2002 .