milliEye: A Lightweight mmWave Radar and Camera Fusion System for Robust Object Detection

A wide range of advanced deep learning algorithms have recently been proposed for image classification and object detection. However, the effectiveness of these methods can be significantly restricted in many real-world scenarios where the visibility or illumination is poor. Compared to RGB cameras, millimeter-wave (mmWave) radars are immune to the above environmental variability and can assist cameras under adverse conditions. To this end, we propose milliEye, a lightweight mmWave radar and camera fusion system for robust object detection on the edge platforms. milliEye has several key advantages over existing sensor fusion approaches. First, while milliEye fuses two sensing modalities in a learning-based fashion, it requires only a small amount of labeled image/radar data of a new scene as it can fully utilize large public image datasets for extensive training. This salient feature enables milliEye to adapt to highly complex real-world environments. Second, based on a novel architecture that decouples the image-based object detector from other modules, milliEye is compatible with different off-the-shelf image-based object detectors. As a result, it can take advantage of the rapid progress of object detection algorithms. Moreover, thanks to the highly compute-efficient fusion approach, milliEye is lightweight and thus suitable for edge-based real-time applications. To evaluate the performance of milliEye, we collect a new radar and camera fusion dataset for object detection, which contains both ordinary-light and low-light illumination conditions. The results show that milliEye can provide substantial performance boosts over state-of-the-art image-based object detectors, including Tiny YOLOv3 and SSD, especially in low-light scenes, while incurring low compute overhead on edge platforms.