Towards robust automatic detection of vulnerable road users: monocular pedestrian tracking from a moving vehicle

We present a first step towards the automatic detection of vulnerable road users in video. Such a system can e.g. be used as an automatic blind spot camera for trucks. When the system detects vulnerable road users in the camera images the driver is warned. Such an application implies severe constraints on the vision system, which introduce several challenges that need to be tackled. Firstly, vulnerable road users are a very diverse object class. Besides pedestrians, we also need to detect e.g. bicyclists, children, wheelchair users and mopeds. Therefore, the detection system has to deal with multi-class, multi-view and multi-pose object tracking. Secondly, the field of view of the camera covers the blind spot area on the side of the truck. This yields a camera image with a highly dynamic background. Other challenges include the real-time character of the application. Only limited time is available to detect the vulnerable road users. This contradicts with the need for a high precision and recall rate. This paper presents a first step towards such an intelligent detection system. We extend existing appearance-based pedestrian detectors with specific motion information and constraints, resulting in a robust pedestrian tracker from a moving vehicle. First an initial detection is done throughout the entire frame. Based on the ego motion of the vehicle, combined with both spatial and temporal information, we can estimate where the pedestrians can be found in consecutive video frames. These constraints tremendously reduce the search area for the person detector, thus enabling real-time performance. Based on this prediction, a verification suffices to confirm the presence of a pedestrian. Using the pedestrian trajectories it is possible to predict whether they will impose a threat to the driver, and generate an appropriate alarm signal. Additionally, we propose a new dataset, recorded with a real blind spot camera, and performed initial experiments with promising results.