Hidden Enemy Visualization using Fast Panoptic Segmentation on Battlefields

Most of enemies in battlefields are not visible due to cover and concealment, and it yields fear and a weakened combat power for allies. In order to overcome it, we present a new approach for visualizing hidden enemies in battlefields to enhance cognition ability and survival for soldiers. Our method is composed of two separate sub-task networks. One is an efficient real-time panoptic segmentation network based on YOLACT [1] to find hidden enemies as well as to understand scenes from the viewpoint of soldiers. The other is an image completion network to reconstruct occluded parts of enemies which is guided by the panoptic segmentation networks. Our experiments on the Cityscapes benchmarks show that the proposed panoptic segmentation network achieves almost realtime speed without significant performance drops. We also demonstrate qualitative results of our segmentation-guided image completion method successfully on a dataset constructed from images of the Battlefield4 game.

[1]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Anton van den Hengel,et al.  Bridging Category-level and Instance-level Semantic Image Segmentation , 2016, ArXiv.

[3]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Min Bai,et al.  UPSNet: A Unified Panoptic Segmentation Network , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Guan Huang,et al.  Attention-Guided Unified Network for Panoptic Segmentation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Kaiming He,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Ting-Chun Wang,et al.  Image Inpainting for Irregular Holes Using Partial Convolutions , 2018, ECCV.

[10]  Yong Jae Lee,et al.  Real-time Instance Segmentation , 2019 .

[11]  Gijs Dubbelman,et al.  Fast Panoptic Segmentation Network , 2019, IEEE Robotics and Automation Letters.

[12]  Yong Jae Lee,et al.  YOLACT: Real-Time Instance Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[13]  Ming Yang,et al.  SSAP: Single-Shot Instance Segmentation With Affinity Pyramid , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[14]  Jianfei Cai,et al.  Pluralistic Image Completion , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  George Papandreou,et al.  DeeperLab: Single-Shot Image Parser , 2019, ArXiv.

[16]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[17]  Carsten Rother,et al.  Panoptic Segmentation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Kaiming He,et al.  Panoptic Feature Pyramid Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).