Distance Surface for Event-Based Optical Flow

We propose DistSurf-OF, a novel optical flow method for neuromorphic cameras. Neuromorphic cameras (or event detection cameras) are an emerging sensor modality that makes use of dynamic vision sensors (DVS) to report asynchronously the log-intensity changes (called “events”) exceeding a predefined threshold at each pixel. In absence of the intensity value at each pixel location, we introduce a notion of “distance surface”—the distance transform computed from the detected events—as a proxy for object texture. The distance surface is then used as an input to the intensity-based optical flow methods to recover the two dimensional pixel motion. Real sensor experiments verify that the proposed DistSurf-OF accurately estimates the angle and speed of each events.

[1]  Chiara Bartolozzi,et al.  Asynchronous frameless event-based optical flow , 2012, Neural Networks.

[2]  Tobias Delbrück,et al.  Frame-free dynamic digital vision , 2008 .

[3]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[4]  Vijayan K. Asari,et al.  Inceptive Event Time-Surfaces for Object Classification Using Neuromorphic Cameras , 2019, ICIAR.

[5]  Kostas Daniilidis,et al.  Unsupervised Event-Based Learning of Optical Flow, Depth, and Egomotion , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  D. Cremers,et al.  Duality TV-L1 flow with fundamental matrix prior , 2008, 2008 23rd International Conference Image and Vision Computing New Zealand.

[7]  Tom Drummond,et al.  Event-Based Motion Segmentation by Motion Compensation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[8]  Tobi Delbrück,et al.  Touchless hand gesture UI with instantaneous responses , 2012, 2012 19th IEEE International Conference on Image Processing.

[9]  Kostas Daniilidis,et al.  EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras , 2018, Robotics: Science and Systems.

[10]  Tobi Delbruck,et al.  A 240 × 180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision Sensor , 2014, IEEE Journal of Solid-State Circuits.

[11]  Davide Scaramuzza,et al.  Asynchronous, Photometric Feature Tracking using Events and Frames , 2018, ECCV.

[12]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[13]  Michael J. Black,et al.  Secrets of optical flow estimation and their principles , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[14]  Tobi Delbruck,et al.  Evaluation of Event-Based Algorithms for Optical Flow with Ground-Truth from Inertial Measurement Sensor , 2016, Front. Neurosci..

[15]  Yiannis Aloimonos,et al.  Unsupervised Learning of Dense Optical Flow and Depth from Sparse Event Data , 2018, ArXiv.

[16]  Garrick Orchard,et al.  A Noise Filtering Algorithm for Event-Based Asynchronous Change Detection Image Sensors on TrueNorth and Its Implementation on TrueNorth , 2018, Front. Neurosci..

[17]  Yiannis Aloimonos,et al.  Contour Motion Estimation for Asynchronous Event-Driven Cameras , 2014, Proceedings of the IEEE.

[18]  Davide Scaramuzza,et al.  A Unifying Contrast Maximization Framework for Event Cameras, with Applications to Motion, Depth, and Optical Flow Estimation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[19]  David J. Fleet,et al.  Performance of optical flow techniques , 1994, International Journal of Computer Vision.

[20]  Michael J. Black,et al.  The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields , 1996, Comput. Vis. Image Underst..

[21]  Min Liu,et al.  Block-matching optical flow for dynamic vision sensors: Algorithm and FPGA implementation , 2017, 2017 IEEE International Symposium on Circuits and Systems (ISCAS).

[22]  Stefan Roth,et al.  UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss , 2017, AAAI.

[23]  Davide Scaramuzza,et al.  Continuous-Time Visual-Inertial Odometry for Event Cameras , 2017, IEEE Transactions on Robotics.

[24]  Sébastien Barré,et al.  A Motion-Based Feature for Event-Based Pattern Recognition , 2017, Front. Neurosci..

[25]  Patrick Pérez,et al.  A multigrid approach for hierarchical motion estimation , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[26]  Joachim Weickert,et al.  Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods , 2005, International Journal of Computer Vision.

[27]  Margarita Chli,et al.  Asynchronous Corner Detection and Tracking for Event Cameras in Real Time , 2018, IEEE Robotics and Automation Letters.

[28]  Richard Szeliski,et al.  A Database and Evaluation Methodology for Optical Flow , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[29]  Davide Scaramuzza,et al.  Focus Is All You Need: Loss Functions for Event-Based Vision , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Patrick Pérez,et al.  Hierarchical Estimation and Segmentation of Dense Motion Fields , 2002, International Journal of Computer Vision.

[31]  Kostas Daniilidis,et al.  Event-Based Visual Inertial Odometry , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  David J. Fleet,et al.  Stability of Phase Information , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  Lindsay Kleeman,et al.  Simultaneous Optical Flow and Segmentation (SOFAS) using Dynamic Vision Sensor , 2018, ICRA 2018.

[34]  Min Liu,et al.  ABMOF: A Novel Optical Flow Algorithm for Dynamic Vision Sensors , 2018, ArXiv.

[35]  Garrick Orchard,et al.  HOTS: A Hierarchy of Event-Based Time-Surfaces for Pattern Recognition , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Konstantinos G. Derpanis,et al.  Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness , 2016, ECCV Workshops.

[37]  Michael J. Black,et al.  A Quantitative Analysis of Current Practices in Optical Flow Estimation and the Principles Behind Them , 2013, International Journal of Computer Vision.

[38]  Ashok Veeraraghavan,et al.  Fast Retinomorphic Event-Driven Representations for Video Gameplay and Action Recognition , 2020, IEEE Transactions on Computational Imaging.

[39]  Ryad Benosman,et al.  HATS: Histograms of Averaged Time Surfaces for Robust Event-Based Object Classification , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[40]  Marc M. Van Hulle,et al.  A phase-based approach to the estimation of the optical flow field using spatial filtering , 2002, IEEE Trans. Neural Networks.

[41]  Stefan Leutenegger,et al.  Simultaneous Optical Flow and Intensity Estimation from an Event Camera , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Luc Van Gool,et al.  Fast Optical Flow Using Dense Inverse Search , 2016, ECCV.

[43]  Hailin Jin,et al.  Fast Edge-Preserving PatchMatch for Large Displacement Optical Flow , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[44]  Chiara Bartolozzi,et al.  Event-Based Visual Flow , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[45]  Kirk Y. W. Scheper,et al.  Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[47]  Tobias Brosch,et al.  On event-based optical flow detection , 2015, Front. Neurosci..

[48]  Keigo Hirakawa,et al.  DAViS Camera Optical Flow , 2020, IEEE Transactions on Computational Imaging.

[49]  Yiannis Aloimonos,et al.  Unsupervised Learning of Dense Optical Flow, Depth and Egomotion from Sparse Event Data , 2018 .

[50]  Keigo Hirakawa,et al.  Fast Convolutional Distance Transform , 2019, IEEE Signal Processing Letters.

[51]  Deqing Sun,et al.  Local Layering for Joint Motion Estimation and Occlusion Detection , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[52]  Ryad Benosman,et al.  Simultaneous Mosaicing and Tracking with an Event Camera , 2014, BMVC.

[53]  Simon Baker,et al.  Lucas-Kanade 20 Years On: A Unifying Framework , 2004, International Journal of Computer Vision.

[54]  David J. Fleet,et al.  Computation of component image velocity from local phase information , 1990, International Journal of Computer Vision.

[55]  Daniel Cremers,et al.  An Improved Algorithm for TV-L 1 Optical Flow , 2009, Statistical and Geometrical Approaches to Visual Motion Analysis.

[56]  Thomas Brox,et al.  FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).