DuFNet: Dual Flow Network of Real-Time Semantic Segmentation for Unmanned Driving Application of Internet of Things

The application of unmanned driving in the Internet of Things is one of the concrete manifestations of the application of artificial intelligence technology. Image semantic segmentation can help the unmanned driving system by achieving road accessibility analysis. Semantic segmentation is also a challenging technology for image understanding and scene parsing. We focused on the challenging task of real-time semantic segmentation in this paper. In this paper, we proposed a novel fast architecture for real-time semantic segmentation named DuFNet. Starting from the existing work of Bilateral Segmentation Network (BiSeNet), DuFNet proposes a novel Semantic Information Flow (SIF) structure for context information and a novel Fringe Information Flow (FIF) structure for spatial information. We also proposed two kinds of SIF with cascaded and paralleled structures, respectively. The SIF encodes the input stage by stage in the ResNet18 backbone and provides context information for the feature fusion module. Features from previous stages usually contain rich low-level details but high-level semantics for later stages. The multiple convolutions embed in Parallel SIF aggregate the corresponding features among different stages and generate a powerful global context representation with less computational cost. The FIF consists of a pooling layer and an upsampling operator followed by projection convolution layer. The concise component provides more spatial details for the network. Compared with BiSeNet, our work achieved faster speed and comparable performance with 72.34% mIoU accuracy and 78 FPS on Cityscapes Dataset based on the ResNet18 backbone.

[1]  Jian Yang,et al.  FBSNet: A Fast Bilateral Symmetrical Network for Real-Time Semantic Segmentation , 2021, IEEE Transactions on Multimedia.

[2]  Michael Ying Yang,et al.  Real-time Semantic Segmentation with Context Aggregation Network , 2021, ISPRS Journal of Photogrammetry and Remote Sensing.

[3]  Huansheng Ning,et al.  Dataflow Management in the Internet of Things: Sensing, Control, and Security , 2021 .

[4]  Z. Zhong,et al.  Internet of things for high-speed railways , 2021, Intelligent and Converged Networks.

[5]  Cordelia Schmid,et al.  Segmenter: Transformer for Semantic Segmentation , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[6]  Xiaohui Yuan,et al.  A review of deep learning methods for semantic segmentation of remote sensing imagery , 2021, Expert Syst. Appl..

[7]  Zhenhua Chai,et al.  Rethinking BiSeNet For Real-time Semantic Segmentation , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  L. Gool,et al.  Exploring Cross-Image Pixel Contrast for Semantic Segmentation , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[9]  Mourade Azrour,et al.  IoT-based data logger for weather monitoring using arduino-based wireless sensor networks with remote graphical application and alerts , 2021, Big Data Min. Anal..

[10]  Mohammed Essaaidi,et al.  Multivariate deep learning approach for electric vehicle speed forecasting , 2021, Big Data Min. Anal..

[11]  Mourade Azrour,et al.  New enhanced authentication protocol for Internet of Things , 2021, Big Data Min. Anal..

[12]  Mourade Azrour,et al.  Intelligent monitoring system for biogas detection based on the Internet of Things: Mohammedia, Morocco city landfill case , 2021, Big Data Min. Anal..

[13]  Yan Huo,et al.  A cross-layer cooperative jamming scheme for social internet of things , 2021, Tsinghua Science and Technology.

[14]  Qinyu Zhang,et al.  Space-air-ground integrated vehicular network for connected and automated vehicles: Challenges and solutions , 2020, Intelligent and Converged Networks.

[15]  Pengbo Si,et al.  Deep reinforcement learning based worker selection for distributed machine learning enhanced edge intelligence in internet of vehicles , 2020 .

[16]  Xiaoguang Zhao,et al.  TB-Net: A Three-Stream Boundary-Aware Network for Fine-Grained Pavement Disease Segmentation , 2020, 2021 IEEE Winter Conference on Applications of Computer Vision (WACV).

[17]  A. Pentland.  Diversity of Idea Flows and Economic Growth , 2020, J. Soc. Comput..

[18]  Siyu Sun,et al.  A Pattern Recognition Framework for Detecting Changes in Chinese Internet Management System , 2020, J. Soc. Comput..

[19]  Kunhao Yang,et al.  Predicting Tie Strength of Chinese Guanxi by Using Big Data of Social Networks , 2020, J. Soc. Comput..

[20]  Michael E. Papka,et al.  Measuring Cities with Software-Defined Sensors , 2020, J. Soc. Comput..

[21]  Zelin Hu,et al.  Dual Stream Segmentation Network for Real-Time Semantic Segmentation , 2020, 2020 IEEE 5th International Conference on Image, Vision and Computing (ICIVC).

[22]  Linqi Song,et al.  Wireless recommendations for Internet of vehicles: Recent advances, challenges, and opportunities , 2020, Intelligent and Converged Networks.

[23]  Gang Yu,et al.  BiSeNet V2: Bilateral Network with Guided Aggregation for Real-Time Semantic Segmentation , 2020, International Journal of Computer Vision.

[24]  Elahe Arani,et al.  RGPNet: A Real-Time General Purpose Semantic Segmentation , 2019, 2021 IEEE Winter Conference on Applications of Computer Vision (WACV).

[25]  Yonghong Tian,et al.  Multi-Class Part Parsing With Joint Boundary-Semantic Awareness , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[26]  Jian Sun,et al.  DFANet: Deep Feature Aggregation for Real-Time Semantic Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Xi Chen,et al.  Boundary-Aware Network for Fast and High-Accuracy Portrait Segmentation , 2019, ArXiv.

[28]  Juntang Zhuang,et al.  ShelfNet for Fast Semantic Segmentation , 2018, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[29]  Gang Yu,et al.  BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation , 2018, ECCV.

[30]  Davide Mazzini,et al.  Guided Upsampling Network for Real-Time Semantic Segmentation , 2018, BMVC.

[31]  Nima Tajbakhsh,et al.  UNet++: A Nested U-Net Architecture for Medical Image Segmentation , 2018, DLMIA/ML-CDS@MICCAI.

[32]  Christopher Zach,et al.  ContextNet: Exploring Context and Detail for Semantic Segmentation in Real-time , 2018, BMVC.

[33]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[34]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[35]  George Papandreou,et al.  Rethinking Atrous Convolution for Semantic Image Segmentation , 2017, ArXiv.

[36]  Xiaojuan Qi,et al.  ICNet for Real-Time Semantic Segmentation on High-Resolution Images , 2017, ECCV.

[37]  Xiaogang Wang,et al.  Pyramid Scene Parsing Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Ian D. Reid,et al.  RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Sepp Hochreiter,et al.  Speeding up Semantic Segmentation for Autonomous Driving , 2016 .

[40]  Eugenio Culurciello,et al.  ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation , 2016, ArXiv.

[41]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Zoltan-Csaba Marton,et al.  Automatic scene parsing for generic object descriptions using shape primitives , 2016, Robotics Auton. Syst..

[43]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Vladlen Koltun,et al.  Multi-Scale Context Aggregation by Dilated Convolutions , 2015, ICLR.

[45]  Yi Yang,et al.  Attention to Scale: Scale-Aware Semantic Image Segmentation , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Seunghoon Hong,et al.  Learning Deconvolution Network for Semantic Segmentation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[48]  Jitendra Malik,et al.  Hypercolumns for object segmentation and fine-grained localization , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Trevor Darrell,et al.  Fully convolutional networks for semantic segmentation , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[51]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[52]  Sanja Fidler,et al.  The Role of Context for Object Detection and Semantic Segmentation in the Wild , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[53]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[55]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[56]  Junyu Ren,et al.  Task offloading strategy with emergency handling and blockchain security in SDN-empowered and fog-assisted healthcare IoT , 2022, Tsinghua Science and Technology.

[57]  Xiaolong Xu,et al.  Artificial intelligence for edge service optimization in Internet of Vehicles: A survey , 2022, Tsinghua Science and Technology.