Coverage Planning for Robotic Vision Applications in Complex 3D Environment

Using robots to perform the vision-based coverage tasks such as inspection, shape reconstruction and surveillance has attracted increasing attentions from both academia and industry in past few years. These tasks require the robot to carry vision sensors such as RGB camera, laser scanner, thermal camera to take surface measurement of the desired target objects, with a required surface coverage constraint. Due to the high demands and repetitive natures of these tasks, automatically generating efficient robotic motion plans could significantly reduce cost and improve productivity, which is highly desirable. Several planning approaches have been proposed for the vision-based coverage planning problems with robots in the past. However, these planning methods either only focused on coverage problems in 2D environment, or found less optimal results, or were specific to limited scenarios. In this thesis, we proposed the novel planning algorithms for vision-based coverage planning problems with industrial manipulators and Unmanned Aerial Vehicles (UAVs) in complex 3D environment. Different sampling and optimization methods have been used in the proposed planning algorithms to achieve better planning results. The very first and important step of these coverage planning tasks is to identify a suitable viewpoint set that satisfies the application requirements. This is considered as a view planning problem. The second step is to plan collision-free paths, as well as the visiting sequence of the viewpoints. This step can be formulated as a sequential path planning problem, or path planning problem for short. In this thesis, we developed view planning methods that generate candidate viewpoints using randomized sampling-based and Medial Object-based methods. The view planning methods led to better results with fewer required viewpoints and higher coverage ratios. Moreover, the proposed view planning methods were also applied to practical application in which the detailed 3D building model needs to be reconstructed when only 2D public map data is available. In addition to the proposed view planning algorithms, we also combined the view planning and path planning problems as a single coverage planning problem; and solved the combined problem in a single optimization process to achieve better results. The proposed planning method was applied to industrial shape inspection application with robotic manipulators. Additionally, we also extended the planning method to a industrial robotic inspection system with kinematic redundancy to enlarge the workspace and to reduce the required inspection time. Moreover, a learning-based robotic calibration method was also developed in order to accurately position vision sensors to desired viewpoints in these instances with industrial manipulators. Acknowledgments First, I would like to thank my thesis advisor Professor Kenji Shimada for his guidance and advices during my Ph.D research. Dr. Wei Lin as my co-advisor also extended great support with his patience and immense knowledge. Their coach has helped me in all times of my Ph.D research and accomplishment of this thesis, and I could not have imagined having a better advisor and mentor. My sincere thanks also goes to other thesis committee members Professor Koushil Sreenath and Professor David Bourne, who provided me insightful comments and suggestions which triggered me to widen my research from various perspectives. With a special mention and thanks to Dr. Joseph Polden and Dr. Pey Yuen Tao, for their help on the experiments and the work that we have done jointly. Also to everyone from Computational Engineering and Robotics Lab, my colleagues in Carnegie Mellon, my soccer teammates from Pittsburgh United, and my friends from Ewaybot Technology, it was wonderful experience studying and working with all of you during last four years. A very special gratitude goes out to Agency for Science, Technology and Research, for providing the financial support for my Ph.D research. Most importantly, I would like to express my sincere gratitude to my beloved wife Xiaole Wang, my parents, and other family members, who have provided me thorough moral support. This work will never be accomplished without them. Thank you all wholeheartedly, for those help and supports among the journey!