Resource aware and incremental mosaics of wide areas from small-scale UAVs

Small-scale unmanned aerial vehicles (UAVs) are an emerging research area and have been recently demonstrated in many applications including disaster response management, construction site monitoring and wide area surveillance where multiple UAVs impose various benefits. In this work we present a system composed of multiple networked UAVs for autonomously monitoring a wide area scenario. Each UAV is able to follow waypoints and capture high-resolution images. In order to overcome the strong resource limitations we implement an incremental approach for generating an orthographic mosaic from the individual images. Captured images are pre-processed on-board, annotated with other sensor data and transferred by a prioritized transmission scheme. The ultimate goal of our approach is to generate an overview mosaic as fast as possible and to improve its quality over time. The mosaicking exploits position and orientation data of the UAV to compute rough image projections which are incrementally refined by scene structure analysis when more image data is available. We evaluate our incremental mosaicking in the strongly resource limited UAV network composed of up to three concurrently flying UAVs. Our results are compared to state-of-the-art mosaicking methods and show a unique performance in our dedicated application scenarios.

[1]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[2]  U. Ozguner,et al.  Multi-UAV Sensing Over Urban Areas via Layered Data Fusion , 2007, 2007 IEEE/SP 14th Workshop on Statistical Signal Processing.

[3]  David E. Culler,et al.  IP is dead, long live IP for wireless sensor networks , 2008, SenSys '08.

[4]  Andrew W. Fitzgibbon,et al.  Simultaneous linear estimation of multiple view geometry and lens distortion , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[5]  Allen R. Hanson,et al.  An efficient method for geo-referenced video mosaicing for environmental monitoring , 2005, Machine Vision and Applications.

[6]  Reinhard Koch,et al.  Self-Calibration and Metric Reconstruction Inspite of Varying and Unknown Intrinsic Camera Parameters , 1999, International Journal of Computer Vision.

[7]  Kenneth Turkowski,et al.  Registration, calibration and blending in creating high quality panoramas , 1998, Proceedings Fourth IEEE Workshop on Applications of Computer Vision. WACV'98 (Cat. No.98EX201).

[8]  Jinwhan Kim,et al.  Efficient image mosaicing for multi-robot visual underwater mapping , 2014, Pattern Recognit. Lett..

[9]  Peter F. Sturm,et al.  A Factorization Based Algorithm for Multi-Image Projective Structure and Motion , 1996, ECCV.

[10]  Robin R. Murphy,et al.  CONOPS and autonomy recommendations for VTOL small unmanned aerial system based on Hurricane Katrina operations , 2009 .

[11]  S. Sitharama Iyengar,et al.  Grid Coverage for Surveillance and Target Location in Distributed Sensor Networks , 2002, IEEE Trans. Computers.

[12]  James A. Slater,et al.  Maintenance and enhancement of the World Geodetic System 1984 , 1994 .

[13]  Mubarak Shah,et al.  COCOA: tracking in aerial imagery , 2006, SPIE Defense + Commercial Sensing.

[14]  Anuar Ahmad,et al.  Aerial mapping using high resolution digital camera and unmanned aerial vehicle for Geographical Information System , 2010, 2010 6th International Colloquium on Signal Processing & its Applications.

[15]  Fred Stentiford,et al.  JPEG 2000 and Region of Interest Coding , 2002 .

[16]  Nael B. Abu-Ghazaleh,et al.  Scalable target coverage in smart camera networks , 2010, ICDSC '10.

[17]  Van Jacobson,et al.  TCP Extensions for High Performance , 1992, RFC.

[18]  Bernhard Rinner,et al.  Video Analysis in Pan-Tilt-Zoom Camera Networks , 2010, IEEE Signal Processing Magazine.

[19]  Gian Luca Foresti,et al.  Real-time image processing for active monitoring of wide areas , 2006, J. Vis. Commun. Image Represent..

[20]  Gerhard Friedrich,et al.  Routing for Continuous Monitoring by Multiple Micro AVs in Disaster Scenarios , 2012, ECAI.

[21]  Jon Louis Bentley,et al.  An Algorithm for Finding Best Matches in Logarithmic Expected Time , 1977, TOMS.

[22]  Gérard G. Medioni,et al.  Map-Enhanced UAV Image Sequence Registration and Synchronization of Multiple Image Sequences , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  D.K. Olson,et al.  Converting Earth-centered, Earth-fixed coordinates to geodetic coordinates , 1996, IEEE Transactions on Aerospace and Electronic Systems.

[24]  Christian Bettstetter,et al.  Channel measurements over 802.11a-based UAV-to-ground links , 2011, 2011 IEEE GLOBECOM Workshops (GC Wkshps).

[25]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Bernhard Rinner,et al.  A Structure Based Mosaicking Approach for Aerial Images from Low Altitude of Non-Planar Scenes , 2011 .

[27]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[28]  Andrew W. Fitzgibbon,et al.  Bundle Adjustment - A Modern Synthesis , 1999, Workshop on Vision Algorithms.

[29]  Christian Wietfeld,et al.  AirShield: A system-of-systems MUAV remote sensing architecture for disaster response , 2009, 2009 3rd Annual IEEE Systems Conference.

[30]  Sunil Arya,et al.  Approximate nearest neighbor queries in fixed dimensions , 1993, SODA '93.

[31]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[32]  A. Tsourdos,et al.  Contaminant Cloud Boundary Monitoring Using Network of UAV Sensors , 2008, IEEE Sensors Journal.

[33]  Robin R. Murphy,et al.  CONOPS and autonomy recommendations for VTOL small unmanned aerial system based on Hurricane Katrina operations , 2009, J. Field Robotics.

[34]  Zhihai He,et al.  Resource allocation and performance analysis of wireless video sensors , 2006, IEEE Transactions on Circuits and Systems for Video Technology.

[35]  Jingjing Huang,et al.  An improved mosaic method based on SIFT algorithm for UAV sequence images , 2010, 2010 International Conference On Computer Design and Applications.

[36]  Sunil Arya,et al.  ANN: library for approximate nearest neighbor searching , 1998 .

[37]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[38]  Jörn Ostermann,et al.  Mesh-based global motion compensation for robust mosaicking and detection of moving objects in aerial surveillance , 2011, CVPR 2011 WORKSHOPS.

[39]  David G. Lowe,et al.  Shape indexing using approximate nearest-neighbour search in high-dimensional spaces , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[40]  Mihaela van der Schaar,et al.  Information-Constrained Resource Allocation in Multicamera Wireless Surveillance Networks , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[41]  Wendi B. Heinzelman,et al.  A Survey of Visual Sensor Networks , 2009, Adv. Multim..

[42]  Bernd Kleinjohann,et al.  Combining Autonomous Exploration, Goal-Oriented Coordination and Task Allocation in Multi-UAV Scenarios , 2010, 2010 Sixth International Conference on Autonomic and Autonomous Systems.

[43]  Michael G. Strintzis,et al.  Optimized transmission of JPEG2000 streams over wireless channels , 2006, IEEE Transactions on Image Processing.

[44]  Allen R. Hanson,et al.  Automatic resource allocation in a distributed camera network , 2010, Machine Vision and Applications.

[45]  Malte Rast High-Detail Local Aerial Imaging Using Autonomous Drones , 2009 .

[46]  Bernhard Rinner,et al.  Networked UAVs as aerial sensor network for disaster management applications , 2010, Elektrotech. Informationstechnik.

[47]  Bernhard Rinner,et al.  Incremental Mosaicking of Images from Autonomous, Small-Scale UAVs , 2010, 2010 7th IEEE International Conference on Advanced Video and Signal Based Surveillance.

[48]  J. Demmel,et al.  Sun Microsystems , 1996 .

[49]  Aníbal Ollero,et al.  Multiple eyes in the skies: architecture and perception issues in the COMETS unmanned air vehicles project , 2005, IEEE Robotics & Automation Magazine.

[50]  Lu Wang,et al.  Semiautomatic registration between ground-level panoramas and an orthorectified aerial image for building modeling , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[51]  Christian Wietfeld,et al.  Coverage evaluation of wireless networks for Unmanned Aerial Systems , 2010, 2010 IEEE Globecom Workshops.

[52]  Ian F. Akyildiz,et al.  A survey on wireless multimedia sensor networks , 2007, Comput. Networks.

[53]  Janne Heikkilä,et al.  A four-step camera calibration procedure with implicit image correction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[54]  Gian Luca Foresti,et al.  Active Tuning of Intrinsic Camera Parameters , 2009, IEEE Transactions on Automation Science and Engineering.

[55]  S. Shankar Sastry,et al.  An Invitation to 3-D Vision: From Images to Geometric Models , 2003 .

[56]  I. Colomina,et al.  Unmanned aerial systems for photogrammetry and remote sensing: A review , 2014 .

[57]  P. Clarke GPS Satellite Surveying , 2007 .

[58]  Bernhard Rinner,et al.  Prioritized data transmission in airborne camera networks for wide area surveillance and image mosaicking , 2011, CVPR 2011 WORKSHOPS.

[59]  R. Bajcsy,et al.  A Uni ed Procedure for Calibrating Intrinsic Parameters of Spherical Lenses , 1999 .

[60]  M. Tech,et al.  A Survey on Image Mosaicing Techniques , 2013 .

[61]  Peter F. Sturm,et al.  Multi-view geometry for general camera models , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[62]  Jan Flusser,et al.  Image registration methods: a survey , 2003, Image Vis. Comput..

[63]  Larry S. Davis,et al.  A General Method for Sensor Planning in Multi-Sensor Systems: Extension to Random Occlusion , 2007, International Journal of Computer Vision.

[64]  Jan Kautz,et al.  Exposure Fusion , 2007, 15th Pacific Conference on Computer Graphics and Applications (PG'07).

[65]  Fabrizio Frescura,et al.  JPEG2000 and MJPEG2000 transmission in 802.11 wireless local area networks , 2003, IEEE Trans. Consumer Electron..

[66]  Luis Merino,et al.  Unmanned Aerial Vehicle Localization Based on Monocular Vision and Online Mosaicking , 2009, J. Intell. Robotic Syst..

[67]  Jianfei Cai,et al.  Robust Transmission of JPEG2000 Encoded Images Over Packet Loss Channels , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[68]  Christopher Hunt,et al.  Notes on the OpenSURF Library , 2009 .

[69]  Yan Ke,et al.  PCA-SIFT: a more distinctive representation for local image descriptors , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[70]  Guoqing Zhou,et al.  Geo-Referencing of Video Flow From Small Low-Cost Civilian UAV , 2010, IEEE Transactions on Automation Science and Engineering.

[71]  Jack Dongarra,et al.  LAPACK Users' Guide, 3rd ed. , 1999 .

[72]  Eija Honkavaara,et al.  Point Cloud Generation from Aerial Image Data Acquired by a Quadrocopter Type Micro Unmanned Aerial Vehicle and a Digital Still Camera , 2012, Sensors.

[73]  Jon Howell,et al.  Low-cost orthographic imagery , 2008, GIS '08.

[74]  Salah Sukkarieh,et al.  A comparison of feature and pose-based mapping using vision, inertial and GPS on a UAV , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[75]  Bernhard Rinner,et al.  Lens distortion correction for thermal cameras to improve aerial imaging with small-scale UAVs , 2011, 2011 IEEE International Symposium on Robotic and Sensors Environments (ROSE).

[76]  Bernhard Rinner,et al.  Collaborative microdrones: applications and research challenges , 2008, Autonomics.

[77]  Yiding Han,et al.  AggieAir: An Integrated and Effective Small Multi-UAV Command, Control and Data Collection Architecture , 2009 .

[78]  Quang-Tuan Luong,et al.  Self-Calibration of a Moving Camera from Point Correspondences and Fundamental Matrices , 1997, International Journal of Computer Vision.

[79]  Gian Luca Foresti,et al.  PTZ camera network reconfiguration , 2009, 2009 Third ACM/IEEE International Conference on Distributed Smart Cameras (ICDSC).

[80]  Edward H. Adelson,et al.  A multiresolution spline with application to image mosaics , 1983, TOGS.

[81]  Randall R. Stewart,et al.  Stream Control Transmission Protocol , 2000, RFC.

[82]  Esin Turkbeyler,et al.  Building aerial mosaics for visual MTI , 2008, Security + Defence.

[83]  Manolis I. A. Lourakis,et al.  SBA: A software package for generic sparse bundle adjustment , 2009, TOMS.

[84]  Jon Postel,et al.  User Datagram Protocol , 1980, RFC.

[85]  Gerd Hirzinger,et al.  Energy-efficient Autonomous Four-rotor Flying Robot Controlled at 1 kHz , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[86]  Yanping Li,et al.  A Fast Mosaic Approach for Remote Sensing Images , 2007, 2007 International Conference on Mechatronics and Automation.

[87]  Luo Juan,et al.  A comparison of SIFT, PCA-SIFT and SURF , 2009 .

[88]  Ed Anderson,et al.  LAPACK Users' Guide , 1995 .

[89]  Michael F. Cohen,et al.  Capturing and viewing gigapixel images , 2007, ACM Trans. Graph..

[90]  Mohamed F. Younis,et al.  Strategies and techniques for node placement in wireless sensor networks: A survey , 2008, Ad Hoc Networks.