Using Real-Time Vision to Control a Convoy of Semi-Autonomous Unmanned Vehicles

This paper presents a control system that is capable of guiding a convoy of semi-autonomous or fully autonomous unmanned vehicles. The control system will be able to establish different modes of operation, including following the preceding vehicle, avoiding obstacles as they are detected, and following the route specified by the operators of the system. The control system makes heavy use of real-time target tracking to ensure that each vehicle will be able to follow the vehicle immediately ahead of it in the convoy and at the same time maintains a safe distance and avoids obstacles. The lead vehicle could either be controlled directly by a human operator or could be given the coordinates and map of the route that it is to follow. Field Programmable Gate Array (FPGA) technology is used to implement the vision algorithms to ensure that they will run at the camera frame rate. FPGA technology has advanced sufficiently that a hardcore processor can be integrated directly in to the FPGA chip to use a real-time operating system for communication and control tasks. This paper will describe the control system, emphasizing realtime tracking of a target, and demonstrate its potential using small toy trucks as the platform. Introduction Convoying is moving goods from one place to another. There are many different types of things that are convoyed throughout the world from basic foodstuff to computer hardware to military equipment and even troops and other personnel. Sometimes, these convoys are composed of only one vehicle, but for purposes of this paper, we will only consider convoys which consist of two or more vehicles. At the present time, the convoys that are traveling around are driven by humans. However, with the advent of robotic technology, it is becoming more and more possible to have robotic systems replace the human driver in the driver’s seat. There are many benefits that will be a result of replacing the human driver. First, driving a convoy can become a monotonous routine as sometimes the scenery driven through changes very little. This can result in the driver losing his focus. Additionally, human drivers are only allowed to drive for so long before they are required to take a break to alleviate the fatigue associated with long drives. And, in military applications, there are times that the convoy must travel through dangerous territory where it may come under attack from enemy soldiers. If robots are used in place of human drivers in these situations, each of these problems can be either ignored or mitigated. For instance, robots do not grow tired of the scenery, they do not have a limit to the time they can spend on the road, and if a robot is lost, its cost is known. This is not to say that robots will soon be taking the place of humans in all convoys. There are still many limitations in the control theory that these robots would use which prevents them from seeing much use in the general public. However, enough work has been done that in controlled environments the benefits of robotic convoys outweigh the drawbacks. Belkhouche and Belkhouche have presented a convoying control strategy which makes use of vision, fiduciaries, and the Kalman filter to control each individual robot [1]. The lead robot has a very sophisticated methodology for choosing the path of the convoy, and every other vehicle in the convoy uses the same tracking algorithm to follow the lead robot. The use of fiduciaries in such circumstances is normal, as it is usually easier to track a known object than an arbitrary one. Sudin and Cook have presented a similar algorithm, but instead of relying on the information from only the preceding vehicle in the convoy, they allow for short range communication to send information back two vehicles [2]. This allows the vehicle to better predict the motion of the vehicle ahead of it as it is assumed that it is following the vehicle ahead of it. Khan and Boloni have developed a system that provides feedback to the driver in a convoy [3]. This system keeps track of the vehicles on the road through wireless communication and provides different convoys of varying speeds and lengths. The driver can then choose which convoy to choose, allowing for faster travel on the interstate. This system does require that each vehicle on the road use the wireless communication. At the heart of each convoy algorithm is the task of identifying and then tracking the preceding vehicle. There are many different methodologies that have been used to track targets. The vast majority of tracking algorithms utilize the stochastic processes used in detection and estimation theory [4-11]. These algorithms have been well developed and have a solid underlying mathematical foundation. Their main drawback is their reliance on linear or nearly linear environments and the limitations of the models being tracked. By combining stochastic method with fuzzy logic, it is possible to represent human intuition in the control strategy, allowing for experts to more easily train the robot [12-14]. Motion field estimators have also been used to track targets [15-17]. These methods generally provide algorithms robust to most noise problems, but they are susceptible to the aperture problem that often arises when sensors aren’t able to gather enough information. Trailovic presented the Multi-Sensor Joint Probabilistic Data Association (MSJPDA) algorithm, which utilizes combinations of different sensors to provide the input into the tracking algorithm, in her thesis [4]. Of particular note, she discovered that the sequential version of the algorithm outperformed the parallel version in all simulations that she performed. She also found that when sensors of different reliabilities are used, the least reliable sensor should be tested first. This reduces the amount that noise from the less reliable sensor affects the estimation of the target’s state. Sujit’s Kalman filter performs similarly to the MSJPDA algorithm [5]. His algorithm utilizes two cameras to reconstruct a 3D image of the target which is then tracked using a standard Kalman filter. The Interacting Multiple Model with Probabilistic Data Association Filter (IMMPDAF), presented by Bar-Shalom, et al, predicts multiple tracks which the target is likely to take and updates the tracks as the target moves [6]. By using multiple tracks that the target could be on, this algorithm is less likely to lose track of the target. The Cramer Rao Lower Bound (CRLB) provides a good estimate of how well any stochastic algorithm can be expected to perform. Part of the problem of using the CRLB is its computational complexity – it is not tractable in most cases. Hernandez, et al, have developed a less complex approximation of the CRLB and show how it can be used with best-fitting Gaussian distributions to track an objects [7]. Rawicz, et al, have attempted to work past the problems inherent in the Kalman filter [8]. Particularly, they have shown how the more generalized H2 and H∞ can be used even when the model being propagated consists of non-linear entities which the Kalman filter cannot handle as well. Despite the greater complexity of these algorithms, they are highly reliable. Likewise, Hashirao, et al, have developed the α-β filter which uses the past three estimates of the target in the calculations [9]. The α-β filter does have problems when the acceleration is constant. Leven and Lanterman provide details of the unscented Kalman filter, the extended Kalman filter, and the particle filter, and show how each should be used [10]. They describe how each algorithm has inherent weaknesses, such as not being able to model a sum of products and discuss ways that these weaknesses can be addressed. Cheng & Ansari present a modified particle filter, the kernel particle filter, which is able to handle dynamic environments that contain system and/or observation noise as part of the filter [11]. By being able to handle the greater level of noise inherent in these systems, this algorithm can more easily be implemented in real systems where the particle filter fails to perform. Fuzzy logic integrates human knowledge and intuition along with stochastic control. Most often, this fuzzy logic is fed through a neural network which provides the inputs into the stochastic filters which then track the target. Fun-Bin and Chin-Teng have utilized a fuzzy network to decide when to use one of three different models [12]. Each of these models represents another level of input, from velocity to acceleration to jerk. Luo and Tse Min use a grey-fuzzy network to provide the initial guess of the target’s position and then use simple correlation techniques to find the location of the target from the previous image [13]. They only use a small tracking window, but due to the nature of the correlation, this window is usually sufficient. Li, et al, use a fuzzy controller to track targets using infrared sensors [14]. Each of these algorithms show the value of using fuzzy logic in tracking a target. Finally, motion estimators have also been used to track targets. Generally, motion estimators consist of determining the change of each pixel in a scene from image to image. Bartholomeus, et al, presents a multi-resolution system which estimates several different motions and then performs checks in the images to find the target [15]. Saeed and Afzulpurkar present an optical flow and stereo vision algorithm for surveillance purposes [16]. This particular algorithm runs in real time and, although used in a stationary camera, could be moved to a mobile platform quite easily. Zabih and Woodfill have developed a simple algorithm which can be used in a number of different methodologies [17]. Of particular interest is their rank transform, which can be used to register an object in one image with its match in the next and can be used with some reliability to perform target tracking. This paper presents a robot control strategy for controlling semi auto