Algorithms for behavior recognition in wide-area video of urban environments

The use of wide-area persistent surveillance by the US military— employing airborne video-camera systems to observe large expanses over extended periods of time (see Figure 1)—is gaining ground as a highly valuable tool for intelligence, surveillance, and reconnaissance. Motion imaging from Iraq and Afghanistan has been used in post-event forensics to help with the early detection of thousands of roadside bombs and terrorist ambushes. Unlike static, single-frame images, wide-area persistent video can track and detect activities and interactions over time, identify patterns that are hard to disguise, and provide signatures of threat-related activities in urban environments. However, current efforts to exploit video data are mostly manual and require hours, or even days, of painstaking analysis to produce results. Similar to security cameras, wide-area persistent surveillance captures flows of images, at a rate of one to two frames per second, while covering wide geographic areas of up to 40 square miles and generating huge volumes of video data (up to terabytes in a single mission). To detect suspicious behavior and identify potential threats (such as a building used for fabrication of improvised electronic devices or activities indicating a potential ambush), thousands of human analysts would need to assess such volumes of data. For wide-area airborne persistent surveillance to provide commanders and troops with real-time intelligence and situational awareness, algorithms and automated processes that reduce time-to-intelligence andanalyst workloads are required. Can algorithms be developed to analyze millions of video frames for patterns that indicate potential threats, i.e., detect normal from abnormal activities, such as unique driving behavior that may occur before the detonation of a vehicle used for a Figure 1. Wide-area persistent surveillance. FOV: Field of view. (Photo courtesy of the Defense Advanced Research Projects Agency.) EO: Electro-optical.