Multimodal Object Detection via Probabilistic Ensembling

. Object detection with multimodal inputs can improve many safety-critical systems such as autonomous vehicles (AVs). Motivated by AVs that operate in both day and night, we study multimodal object detection with RGB and thermal cameras, since the latter provides much stronger object signatures under poor illumination. We explore strategies for fusing information from different modalities. Our key contribution is a probabilistic ensembling technique, ProbEn , a simple non-learned method that fuses together detections from multi-modalities. We derive ProbEn from Bayes’ rule and first principles that assume conditional independence across modalities. Through probabilistic marginalization, ProbEn elegantly handles missing modalities when detectors do not fire on the same object. Importantly, ProbEn also notably improves multimodal detection even when the conditional independence assumption does not hold, e.g., fusing outputs from other fusion methods (both off-the-shelf and trained in-house). We validate ProbEn on two benchmarks containing both aligned (KAIST) and unaligned (FLIR) multimodal images, showing that ProbEn outperforms prior work by more than 13% in relative performance! performs quite well on both Day and Night ; average score fusion performs poorly because it double counts class prior. As for box fusion, using the learned variance / uncertainty by v-avg performs better than the heuristic methods (avg and s-avg). Our ProbEn performs significantly better and ProbEn 3 is the best by fusing three models: RGB, Thermal, and MidFusion. Image pairs are split into train-set (8 , 862 images) and a validation set (1 , 366 images). FLIR evaluates on three classes which have imbalanced examples [8,30,60,41,12]: 28 , 151 persons, 46 , 692 cars, and 4 , 457 bicycles. Following [60], we remove 108 thermal images in the val-set that do not have the RGB counterparts. For breakdown analysis w.r.t day/night scenes, we manually tag the validation images with “day” (768) and “night” (490). We will release our annotations to the public.