Robust Motion Compensation for Event Cameras With Smooth Constraint

Event cameras capture brightness changes at an ultra-high speed and output a series of asynchronous events. In order to form image frames without motion blur from event stream, motion compensation needs to be applied. Unlike other algorithms which either estimate ego-motion or estimate optical flow with strict assumptions, we directly align the curved event trajectories with time-varying motion parameters. By maximizing the image energy with smooth constraint on motion parameters, our algorithm can perform accurately and robustly. We conduct extensive experiments on several public datasets and the recently released event camera, i.e. CeleX-5, the experimental results verify the superiority of the proposed algorithm.

[1]  Davide Scaramuzza,et al.  Accurate Angular Velocity Estimation With an Event Camera , 2017, IEEE Robotics and Automation Letters.

[2]  L. Brown.  Dirac ’ s The Principles of Quantum Mechanics * , 2006 .

[3]  Nikos G. Tsagarakis,et al.  Real-Time 6DOF Pose Relocalization for Event Cameras With Stacked Spatial LSTM Networks , 2017, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[4]  Yiannis Aloimonos,et al.  Unsupervised Learning of Dense Optical Flow and Depth from Sparse Event Data , 2018, ArXiv.

[5]  Tobi Delbrück,et al.  A 128$\times$ 128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor , 2008, IEEE Journal of Solid-State Circuits.

[6]  T. Delbruck,et al.  > Replace This Line with Your Paper Identification Number (double-click Here to Edit) < 1 , 2022 .

[7]  Leo Jan Velthoven,et al.  Motion blur reduction for liquid crystal displays: motion-compensated inverse filtering , 2004, IS&T/SPIE Electronic Imaging.

[8]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[9]  Davide Scaramuzza,et al.  Real-time Visual-Inertial Odometry for Event Cameras using Keyframe-based Nonlinear Optimization , 2017, BMVC.

[10]  Jiaya Jia,et al.  Image partial blur detection and classification , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Tiejun Huang,et al.  Spike Coding for Dynamic Vision Sensor in Intelligent Driving , 2019, IEEE Internet of Things Journal.

[12]  Christoph Sulzbachner,et al.  Address-Event Based Stereo Vision with Bio-Inspired Silicon Retina Imagers , 2011 .

[13]  Nick Barnes,et al.  Continuous-time Intensity Estimation Using Event Cameras , 2018, ACCV.

[14]  Thomas Pock,et al.  Real-Time Intensity-Image Reconstruction for Event Cameras Using Manifold Regularisation , 2016, International Journal of Computer Vision.

[15]  Davide Scaramuzza,et al.  Low-latency visual odometry using event-based feature tracks , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[16]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[17]  Christoph Sulzbachner,et al.  Bio-inspired Stereo Vision System with Silicon Retina Imagers , 2009, ICVS.

[18]  Tobi Delbrück,et al.  The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and SLAM , 2016, Int. J. Robotics Res..

[19]  Tobi Delbrück,et al.  EV-IMO: Motion Segmentation Dataset and Learning Pipeline for Event Cameras , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[20]  Wei Li,et al.  DET: A High-Resolution DVS Dataset for Lane Extraction , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[21]  Davide Scaramuzza,et al.  A Unifying Contrast Maximization Framework for Event Cameras, with Applications to Motion, Depth, and Optical Flow Estimation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[22]  Davide Scaramuzza,et al.  Focus Is All You Need: Loss Functions for Event-Based Vision , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[24]  Wotao Yin,et al.  Global Convergence of ADMM in Nonconvex Nonsmooth Optimization , 2015, Journal of Scientific Computing.

[25]  Narciso García,et al.  Event-Based Vision Meets Deep Learning on Steering Prediction for Self-Driving Cars , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Tobi Delbruck,et al.  A 240 × 180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision Sensor , 2014, IEEE Journal of Solid-State Circuits.

[27]  Lindsay Kleeman,et al.  Simultaneous Optical Flow and Segmentation (SOFAS) using Dynamic Vision Sensor , 2018, ICRA 2018.

[28]  Shaojie Shen,et al.  VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator , 2017, IEEE Transactions on Robotics.

[29]  P. Dirac.  Principles of Quantum Mechanics , 1982 .

[30]  Stefan Leutenegger,et al.  Simultaneous Optical Flow and Intensity Estimation from an Event Camera , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Vladlen Koltun,et al.  Events-To-Video: Bringing Modern Computer Vision to Event Cameras , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).