High‐speed autonomous obstacle avoidance with pushbroom stereo

We present the design and implementation of a small autonomous unmanned aerial vehicle capable of high‐speed flight through complex natural environments. Using only onboard GPS‐denied sensing and computation, we perform obstacle detection, planning, and feedback control in real time. We present a novel integrated approach to perception and control using pushbroom stereo, which exploits forward motion to enable efficient obstacle detection and avoidance using lightweight processors on an unmanned aerial vehicle. Our use of model‐based planning and control techniques allows us to track precise trajectories that avoid obstacles identified by the vision system. We demonstrate a complete working system detecting obstacles at 120 Hz and avoiding trees at up to 14 m/s (31 MPH). To the best of our knowledge, this is the fastest lightweight aerial vehicle to perform collision avoidance using three‐dimensional geometric information.

[1]  H. Bock,et al.  A Multiple Shooting Algorithm for Direct Solution of Optimal Control Problems , 1984 .

[2]  Lennart Ljung,et al.  System identification toolbox for use with MATLAB , 1988 .

[3]  Christopher G. Harris,et al.  3D positional integration from image sequences , 1988, Image Vis. Comput..

[4]  K. W. Iliff,et al.  Parameter Estimation for Flight Vehicles , 1989 .

[5]  Jeff S. Shamma,et al.  Gain-Scheduled Missile Autopilot Design Using Linear Parameter Varying Transformations , 1993 .

[6]  O. V. Stryk,et al.  Numerical Solution of Optimal Control Problems by Direct Collocation , 1993 .

[7]  Christopher G. Atkeson,et al.  Using Local Trajectory Optimizers to Speed Up Global Optimization in Dynamic Programming , 1993, NIPS.

[8]  Rajiv Gupta,et al.  Linear Pushbroom Cameras , 1994, ECCV.

[9]  B. Faverjon,et al.  Probabilistic Roadmaps for Path Planning in High-Dimensional Con(cid:12)guration Spaces , 1996 .

[10]  Robert T. Collins,et al.  A space-sweep approach to true multi-image matching , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[11]  Mark H. Overmars,et al.  Motion Planning for Carlike Robots Using a Probabilistic Learning Approach , 1997, Int. J. Robotics Res..

[12]  Alberto Bemporad,et al.  Robust model predictive control: A survey , 1998, Robustness in Identification and Control.

[13]  S. LaValle Rapidly-exploring random trees : a new tool for path planning , 1998 .

[14]  E. Feron,et al.  Robust hybrid control for autonomous vehicle motion planning , 2000, Proceedings of the 39th IEEE Conference on Decision and Control (Cat. No.00CH37187).

[15]  L. Singh,et al.  Trajectory generation for a UAV in urban terrain, using nonlinear MPC , 2001, Proceedings of the 2001 American Control Conference. (Cat. No.01CH37148).

[16]  Jun Morimoto,et al.  Nonparametric Representation of Policies and Value Functions: A Trajectory-Based Approach , 2002, NIPS.

[17]  Eric Feron,et al.  Control Logic for Automated Aerobatic Flight of a Miniature Helicopter , 2002 .

[18]  Gaurav S. Sukhatme,et al.  Vision-based autonomous landing of an unmanned aerial vehicle , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[19]  Thomas Netter,et al.  A robotic aircraft that follows terrain using a neuromorphic eye , 2002, IEEE/RSJ International Conference on Intelligent Robots and Systems.

[20]  Ruigang Yang,et al.  Multi-resolution real-time stereo on commodity graphics hardware , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[21]  Peter I. Corke,et al.  Low-cost flight control system for a small autonomous helicopter , 2003, 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422).

[22]  Salah Sukkarieh,et al.  Airborne simultaneous localisation and map building , 2003, 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422).

[23]  Matthew Garratt,et al.  An overview of insect-inspired guidance for application in ground and airborne platforms , 2004 .

[24]  James R. Bergen,et al.  Visual odometry , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[25]  T. Kanade,et al.  Real-time and 3D vision for autonomous small and micro air vehicles , 2004, 2004 43rd IEEE Conference on Decision and Control (CDC) (IEEE Cat. No.04CH37601).

[26]  Sunil K. Agrawal,et al.  Differentially Flat Systems , 2004 .

[27]  Ashutosh Saxena,et al.  High speed obstacle avoidance using monocular vision and reinforcement learning , 2005, ICML.

[28]  James J. Little,et al.  Vision-based SLAM using the Rao-Blackwellised Particle Filter , 2005 .

[29]  H. Hirschmuller Accurate and efficient stereo processing by semi-global matching and mutual information , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[30]  J. Langelaan,et al.  Towards Autonomous UAV Flight in Forests , 2005 .

[31]  Pierre-Brice Wieber,et al.  Fast Direct Multiple Shooting Algorithms for Optimal Robot Control , 2005 .

[32]  Gaurav S. Sukhatme,et al.  Combined optic-flow and stereo-based navigation of urban canyons for a UAV , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[33]  Larry H. Matthies,et al.  Vision Guided Landing of an Autonomous Helicopter in Hazardous Terrain , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[34]  Pieter Abbeel,et al.  An Application of Reinforcement Learning to Aerobatic Helicopter Flight , 2006, NIPS.

[35]  Christopher G. Atkeson,et al.  Policies based on trajectory libraries , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[36]  Jacob Willem Langelaan State estimation for autonomous flight in cluttered environments , 2006 .

[37]  Steven M. LaValle,et al.  Planning algorithms , 2006 .

[38]  Jeffrey Byrne,et al.  Stereo based obstacle detection for an unmanned air vehicle , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[39]  Adrian L. R. Thomas,et al.  Automatic aeroelastic devices in the wings of a steppe eagle Aquila nipalensis , 2007, Journal of Experimental Biology.

[40]  Olivier Stasse,et al.  MonoSLAM: Real-Time Single Camera SLAM , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  G. Klein,et al.  Parallel Tracking and Mapping for Small AR Workspaces , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[42]  Timothy W. McLain,et al.  Autonomous Landing of Miniature Aerial Vehicles , 2005, J. Aerosp. Comput. Inf. Commun..

[43]  Marko Bacic,et al.  Flight Control Mechanisms in Birds of Prey , 2007 .

[44]  Jan-Michael Frahm,et al.  Real-Time Plane-Sweeping Stereo with Multiple Sweeping Directions , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[45]  Sebastian Scherer,et al.  Flying Fast and Low Among Obstacles , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[46]  Graham K. Taylor,et al.  Use and Function of a Leading Edge Flap on the Wings of Eagles , 2007 .

[47]  Dario Floreano,et al.  Near-obstacle flight with small UAVs , 2008 .

[48]  Russ Tedrake,et al.  Experiments in Fixed-Wing UAV Perching , 2008 .

[49]  Dario Floreano,et al.  optiPilot: control of take-off and landing using optic flow , 2009 .

[50]  Russ Tedrake,et al.  LQR-trees: Feedback motion planning on sparse randomized trees , 2009, Robotics: Science and Systems.

[51]  Frantisek M. Sobolic,et al.  Agile flight control techniques for a fixed-wing aircraft , 2009 .

[52]  Dario Floreano,et al.  Vision-based control of near-obstacle flight , 2009, Auton. Robots.

[53]  Rick E. Cory,et al.  Supermaneuverable perching , 2010 .

[54]  Edwin Olson,et al.  LCM: Lightweight Communications and Marshalling , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[55]  Vijay Kumar,et al.  Cooperative Grasping and Transport Using Multiple Quadrotors , 2010, DARS.

[56]  Raffaello D'Andrea,et al.  A simple learning strategy for high-speed quadrocopter multi-flips , 2010, 2010 IEEE International Conference on Robotics and Automation.

[57]  Stephen P. Boyd,et al.  Fast Model Predictive Control Using Online Optimization , 2010, IEEE Transactions on Control Systems Technology.

[58]  Russ Tedrake,et al.  Magnetic localization for perching UAVs on powerlines , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[59]  Markus Hehn,et al.  A flying inverted pendulum , 2011, 2011 IEEE International Conference on Robotics and Automation.

[60]  Jonathan P. How,et al.  Comparison of Fixed and Variable Pitch Actuators for Agile Quadrotors , 2011 .

[61]  Friedrich Fraundorfer,et al.  Visual Odometry Part I: The First 30 Years and Fundamentals , 2022 .

[62]  Dario Floreano,et al.  Aerial Locomotion in Cluttered Environments , 2011, ISRR.

[63]  Larry H. Matthies,et al.  Stereo and IMU assisted visual odometry on an OMAP3530 for small robots , 2011, CVPR 2011 WORKSHOPS.

[64]  Marc Pollefeys,et al.  MAV visual SLAM with plane constraint , 2011, 2011 IEEE International Conference on Robotics and Automation.

[65]  Raffaello D'Andrea,et al.  Quadrocopter ball juggling , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[66]  Vijay Kumar,et al.  Minimum snap trajectory generation and control for quadrotors , 2011, 2011 IEEE International Conference on Robotics and Automation.

[67]  Radu Bogdan Rusu,et al.  3D is here: Point Cloud Library (PCL) , 2011, 2011 IEEE International Conference on Robotics and Automation.

[68]  Vijay Kumar,et al.  Construction with quadrotor teams , 2012, Auton. Robots.

[69]  Andrew J. Davison,et al.  Real-Time Camera Tracking: When is High Frame-Rate Best? , 2012, ECCV.

[70]  Marc Pollefeys,et al.  PIXHAWK: A micro aerial vehicle design for autonomous flight using onboard computer vision , 2012, Auton. Robots.

[71]  Russ Tedrake,et al.  Control synthesis and verification for a perching UAV using LQR-Trees , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).

[72]  Russ Tedrake,et al.  Robust Online Motion Planning with Regions of Finite Time Invariance , 2012, WAFR.

[73]  Daniel Cremers,et al.  Camera-based navigation of a low-cost quadrocopter , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[74]  Russ Tedrake,et al.  Safety verification of reactive controllers for UAV flight in cluttered environments using barrier certificates , 2012, 2012 IEEE International Conference on Robotics and Automation.

[75]  Vijay Kumar,et al.  Trajectory generation and control for precise aggressive maneuvers with quadrotors , 2012, Int. J. Robotics Res..

[76]  Roland Siegwart,et al.  Real-time onboard visual-inertial state estimation and self-calibration of MAVs in unknown environments , 2012, 2012 IEEE International Conference on Robotics and Automation.

[77]  Stanley T. Birchfield,et al.  Autonomous exploration using rapid perception of low-resolution image information , 2012, Auton. Robots.

[78]  Raffaello D'Andrea,et al.  Cooperative quadrocopter ball throwing and catching , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[79]  Nicholas Roy,et al.  State estimation for aggressive flight in GPS-denied environments using onboard sensing , 2012, 2012 IEEE International Conference on Robotics and Automation.

[80]  Claire J. Tomlin,et al.  Learning-based model predictive control on a quadrotor: Onboard implementation and experimental results , 2012, 2012 IEEE International Conference on Robotics and Automation.

[81]  Frank Dellaert,et al.  Saliency detection and model-based tracking: a two part vision system for small robot navigation in forested environment , 2012, Defense, Security, and Sensing.

[82]  Marc Pollefeys,et al.  Real-time velocity estimation based on optical flow and disparity matching , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[83]  Abraham Bachrach,et al.  Trajectory bundle estimation For perception-driven planning , 2013 .

[84]  Martial Hebert,et al.  Learning monocular reactive UAV control in cluttered natural environments , 2012, 2013 IEEE International Conference on Robotics and Automation.

[85]  Peter I. Corke,et al.  Image-based visual navigation for mobile robots , 2013, 2013 IEEE International Conference on Robotics and Automation.

[86]  Anastasios I. Mourikis,et al.  3-D motion estimation and online temporal calibration for camera-IMU systems , 2013, 2013 IEEE International Conference on Robotics and Automation.

[87]  Peter I. Corke,et al.  Monocular vision based autonomous navigation for a cost-effective MAV in GPS-denied environments , 2013, 2013 IEEE/ASME International Conference on Advanced Intelligent Mechatronics.

[88]  Vijay Kumar,et al.  Vision-based state estimation for autonomous rotorcraft MAVs in complex environments , 2013, 2013 IEEE International Conference on Robotics and Automation.

[89]  Charles Richter,et al.  Polynomial Trajectory Planning for Aggressive Quadrotor Flight in Dense Indoor Environments , 2016, ISRR.

[90]  Girish Chowdhary,et al.  GPS‐denied Indoor and Outdoor Monocular Vision Aided Navigation and Control of Unmanned Aircraft , 2013, J. Field Robotics.

[91]  Vijay Kumar,et al.  Towards a swarm of agile micro quadrotors , 2012, Autonomous Robots.

[92]  Anastasios I. Mourikis,et al.  Real-time motion tracking on a cellphone using inertial sensing and a rolling-shutter camera , 2013, 2013 IEEE International Conference on Robotics and Automation.

[93]  Frank Dellaert,et al.  High Frame Rate Egomotion Estimation , 2013, ICVS.

[94]  Peter I. Corke,et al.  Cross-calibration of push-broom 2D LIDARs and cameras in natural scenes , 2013, 2013 IEEE International Conference on Robotics and Automation.

[95]  Davide Scaramuzza,et al.  SVO: Fast semi-direct monocular visual odometry , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[96]  Amir Ali Ahmadi,et al.  Control and verification of high-dimensional systems with DSOS and SDSOS programming , 2014, 53rd IEEE Conference on Decision and Control.

[97]  Andrew A Biewener,et al.  Through the eyes of a bird: modelling visually guided obstacle flight , 2014, Journal of The Royal Society Interface.

[98]  Joseph L. Moore,et al.  Robust post-stall perching with a fixed-wing UAV , 2014 .

[99]  Guido C. H. E. de Croon,et al.  Autonomous flight of a 20-gram Flapping Wing MAV with a 4-gram onboard stereo vision system , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[100]  Mark M. Tobenkin Robustness analysis for identification and control of nonlinear systems , 2014 .

[101]  Marc Pollefeys,et al.  Real-time and low latency embedded computer vision hardware based on a combination of FPGA and mobile CPU , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[102]  Russ Tedrake,et al.  Flying between obstacles with an autonomous knife-edge maneuver , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[103]  Russ Tedrake,et al.  Robust post-stall perching with a simple fixed-wing glider using LQR-Trees , 2014, Bioinspiration & biomimetics.

[104]  Russ Tedrake,et al.  Pushbroom stereo for high-speed navigation in cluttered environments , 2014, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[105]  Martial Hebert,et al.  Vision and Learning for Deliberative Monocular Cluttered Flight , 2014, FSR.

[106]  Marc Pollefeys,et al.  Reactive avoidance using embedded stereo vision for MAV flight , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[107]  Vijay Kumar,et al.  High speed navigation for quadrotors with limited onboard sensing , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[108]  Russ Tedrake,et al.  Funnel libraries for real-time robust feedback motion planning , 2016, Int. J. Robotics Res..