Camera View Planning for Structure from Motion: Achieving Targeted Inspection Through More Intelligent View Planning Methods

Camera View Planning for Structure from Motion: Achieving Targeted Inspection Through More Intelligent View Planning Methods Trent James Okeson Department of Chemical Engineering, BYU Master of Science Remote sensors and unmanned aerial vehicles (UAVs) have the potential to dramatically improve infrastructure health monitoring in terms of accuracy of the information and frequency of data collection. UAV automation has made significant progress but that automation is also creating vast amounts of data that needs to be processed into actionable information. A key aspect of this work is the optimization (not just automation) of data collection from UAVs for targeted planning of mission objectives. This work investigates the use of camera planning for Structure from Motion for 3D modeling of infrastructure. Included in this thesis is a novel multi-scale view-planning algorithm for autonomous targeted inspection. The method presented reduced the number of photos needed and therefore reduced the processing time while maintaining desired accuracies across the test site. A second focus in this work investigates various set covering problem algorithms to use for selecting the optimal camera set. The trade-offs between solve time and quality of results are explored. The Carousel Greedy algorithm is found to be the best method for solving the problem due to its relatively fast solve speeds and the high quality of the solutions found. Finally, physical flight tests are used to demonstrate the quality of the method for determining coverage. Each of the set covering problem algorithms are used to create a camera set that achieves 95% coverage. The models from the different camera sets are comparable despite having a large amount of variability in the camera sets chosen. While this study focuses on multi-scale view planning for optical sensors, the methods could be extended to other remote sensors, such as aerial LiDAR.

[1]  Ivan Rojas,et al.  Optimized Photogrammetric Network Design with Flight Path Planner for UAV-based Terrain Surveillance , 2014 .

[2]  Joachim Denzler,et al.  Multi-View Planning for Simultaneous Coverage and Accuracy Optimisation , 2010, BMVC.

[3]  Srikanth Saripalli,et al.  Rapid mapping of ultrafine fault zone topography with structure from motion , 2014 .

[4]  Xiang Chen,et al.  Modeling Coverage in Camera Networks: A Survey , 2012, International Journal of Computer Vision.

[5]  Ailsa H. Land,et al.  An Automatic Method of Solving Discrete Programming Problems , 1960 .

[6]  Eric Jones,et al.  SciPy: Open Source Scientific Tools for Python , 2001 .

[7]  Yuri Kochetov,et al.  Behavior of the Ant Colony Algorithm for the Set Covering Problem , 2000 .

[8]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[9]  Ali Khaloo,et al.  Hierarchical Dense Structure-from-Motion Reconstructions for Infrastructure Condition Assessment , 2017, J. Comput. Civ. Eng..

[10]  Robin Lougee,et al.  The Common Optimization INterface for Operations Research: Promoting open-source software in the operations research community , 2003, IBM J. Res. Dev..

[11]  Luca Maria Gambardella,et al.  Ant Algorithms for Discrete Optimization , 1999, Artificial Life.

[12]  David L. Woodruff,et al.  Pyomo: modeling and solving mathematical programs in Python , 2011, Math. Program. Comput..

[13]  Russell C. Eberhart,et al.  A discrete binary version of the particle swarm algorithm , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[14]  Mongi A. Abidi,et al.  Can You See Me Now? Sensor Positioning for Automated and Persistent Surveillance , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[15]  Y.F. Li,et al.  Automatic sensor placement for model-based robot vision , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[16]  John D. Hedengren,et al.  Comparison of SfM computer vision point clouds of a landslide derived from multiple small UAV platforms and sensors to a TLS-based model , 2016 .

[17]  Subhashis Banerjee,et al.  Isolated 3D object recognition through next view planning , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[18]  Joachim Denzler,et al.  Online Next-Best-View Planning for Accuracy Optimization Using an Extended E-Criterion , 2010, 2010 20th International Conference on Pattern Recognition.

[19]  Feng Lin,et al.  A Two-Stage Optimized Next-View Planning Framework for 3-D Unknown Environment Exploration, and Structural Reconstruction , 2017, IEEE Robotics and Automation Letters.

[20]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[21]  Zhi-Gang Ren,et al.  New ideas for applying ant colony optimization to the set covering problem , 2010, Comput. Ind. Eng..

[22]  Markus Bläser,et al.  Computing small partial coverings , 2003, Inf. Process. Lett..

[23]  Raffaele Cerulli,et al.  Carousel greedy: A generalized greedy algorithm with applications in optimization , 2017, Comput. Oper. Res..

[24]  Celso C. Ribeiro,et al.  Greedy Randomized Adaptive Search Procedures: Advances, Hybridizations, and Applications , 2010 .

[25]  Anthony Tzes,et al.  Distributed infrastructure inspection path planning for aerial robotics subject to time constraints , 2016, 2016 International Conference on Unmanned Aircraft Systems (ICUAS).

[26]  Saeid Nahavandi,et al.  Target coverage in camera networks for manufacturing workplaces , 2016, J. Intell. Manuf..

[27]  Sridha Sridharan,et al.  Optimal Camera Planning Under Versatile User Constraints in Multi-Camera Image Processing Systems , 2014, IEEE Transactions on Image Processing.

[28]  Farid Kendoul,et al.  Survey of advances in guidance, navigation, and control of unmanned rotorcraft systems , 2012, J. Field Robotics.

[29]  Mohammad Al Hasan,et al.  Optimal placement of stereo sensors , 2007, Optim. Lett..

[30]  M. Resende,et al.  A probabilistic heuristic for a computationally difficult set covering problem , 1989 .

[31]  Omid Amini,et al.  Implicit branching and parameterized partial cover problems , 2011, J. Comput. Syst. Sci..

[32]  G. Roth,et al.  View planning for automated three-dimensional object reconstruction and inspection , 2003, CSUR.

[33]  Gerd Hirzinger,et al.  View Planning for Multi-View Stereo 3D Reconstruction Using an Autonomous Multicopter , 2012, J. Intell. Robotic Syst..

[34]  Mehdi Dehghan,et al.  Optimal visual sensor placement for coverage based on target location profile , 2011, Ad Hoc Networks.

[35]  Jack P. C. Kleijnen,et al.  EUROPEAN JOURNAL OF OPERATIONAL , 1992 .

[36]  Sanjeev Khanna,et al.  Target tracking with distributed sensors: The focus of attention problem , 2005, Comput. Vis. Image Underst..

[37]  Richard Szeliski,et al.  Building Rome in a day , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[38]  Y. Morsly,et al.  Particle Swarm Optimization Inspired Probability Algorithm for Optimal Camera Network Placement , 2012, IEEE Sensors Journal.

[39]  Horst Bischof,et al.  Photogrammetric Camera Network Design for Micro Aerial Vehicles , 2012 .

[40]  Hanil Jeong,et al.  Camera Placement in Smart Cities for Maximizing Weighted Coverage With Budget Limit , 2017, IEEE Sensors Journal.

[41]  Robert C. Leishman,et al.  Relative navigation approach for vision-based aerial GPS-denied navigation , 2013 .

[42]  Nikolaos Papanikolopoulos,et al.  Optimal Camera Placement for Automated Surveillance Tasks , 2007, J. Intell. Robotic Syst..

[43]  Chengnian Long,et al.  Optimal coverage of camera networks using PSO algorithm , 2011, 2011 4th International Congress on Image and Signal Processing.

[44]  Nicola Conci,et al.  Optimal configuration of PTZ camera networks based on visual quality assessment and coverage maximization , 2013, 2013 Seventh International Conference on Distributed Smart Cameras (ICDSC).

[45]  Joseph K. Kearney,et al.  Optimal Camera Placement for Motion Capture Systems , 2017, IEEE Transactions on Visualization and Computer Graphics.

[46]  Hua Wu,et al.  Flying Robot Based Viewpoint Selection for the Electricity Transmission Equipment Inspection , 2014 .

[47]  Sameh Al-Shihabi.  A hybrid of max-min ant system and linear programming for the k-covering problem , 2016, Comput. Oper. Res..

[48]  John D. Hedengren,et al.  Potential Benefits of Combining Anomaly Detection with View Planning for UAV Infrastructure Modeling , 2017, Remote. Sens..

[49]  David L. Woodruff,et al.  Pyomo — Optimization Modeling in Python , 2012, Springer Optimization and Its Applications.

[50]  Vítor Sequeira,et al.  The view-cube: an efficient method of view planning for 3D modelling from range data , 2000, Proceedings Fifth IEEE Workshop on Applications of Computer Vision.

[51]  Pavlo Skladannyi,et al.  ОТРИМАННЯ ТА ОБРОБКА ГЕОПРОСТОРОВИХ ДАНИХ ЗА ДОПОМОГОЮ MATLAB MAPPING TOOLBOX , 2019 .

[52]  Pradeep K. Atrey,et al.  Bus surveillance: how many and where cameras should be placed , 2012, Multimedia Tools and Applications.

[53]  Geoffrey A. Hollinger,et al.  Uncertainty-driven view planning for underwater inspection , 2012, 2012 IEEE International Conference on Robotics and Automation.

[54]  Yinhai Wang,et al.  Google Earth elevation data extraction and accuracy assessment for transportation applications , 2017, PloS one.

[55]  John D. Hedengren,et al.  Evolutionary View Planning for Optimized UAV Terrain Modeling in a Simulated Environment , 2015, Remote. Sens..

[56]  Michael J. Brusco,et al.  A morphing procedure to supplement a simulated annealing heuristic for cost‐ andcoverage‐correlated set‐covering problems , 1999, Ann. Oper. Res..

[57]  Ali Khaloo,et al.  Unmanned aerial vehicle inspection of the Placer River Trail Bridge through image-based 3D modelling , 2018 .

[58]  Roland Siegwart,et al.  Three-dimensional coverage path planning via viewpoint resampling and tour optimization for aerial robots , 2015, Autonomous Robots.

[59]  Fischetti Caprara,et al.  An Indirect Genetic Algorithm for Set Covering Problems , 2002 .

[60]  Samuel T. Thiele,et al.  Ground-based and UAV-Based photogrammetry: A multi-scale, high-resolution mapping tool for structural geology and paleoseismology , 2014 .

[61]  K. Al-Sultan,et al.  A Genetic Algorithm for the Set Covering Problem , 1996 .

[62]  Sridha Sridharan,et al.  Recent Advances in Camera Planning for Large Area Surveillance , 2016, ACM Comput. Surv..

[63]  S. Khuller,et al.  Approximation algorithms for partial covering problems , 2001, J. Algorithms.