Distributed and Context Aware Application of Deep Neural Networks in Mobile 3D-Multi-sensor Systems Based on Cloud-, Edge- and FPGA-Computing

The use of deep neural networks (DNN) for 3Dimage processing significantly enhances visual cognition of mobile systems by considering spatial information. However, training and execution require high computing power. This is crucial in applications with real-time constraints since mobile systems have limited resources. Current approaches do not consider the usage of 3D-sensing. Furthermore, suggested system architectures solely focus on cloud- and edge-computing in combination with load balancing and parallelization for a distributed execution of DNNs. In contrast, we propose a novel system architecture for the distributed and context aware usage of DNNs for image processing tasks in mobile 3D-multi-sensor systems. Thereby, the scalable cloud- and edge-infrastructure is complemented by realtime capable and energy-efficient FPGA-computing. The publishsubscriber pattern facilitates the distributed execution of DNNs as well as their dynamic deployment. Moreover, context information is considered. Thus, a rule-based context model dynamically loads specialized DNNs and selects appropriate devices for execution. Finally, a case-study on a mobile 3D-multi-sensor system for wheeled walkers demonstrates applicability and benefits of the proposed approach.

[1]  Maciej Huk.  Measuring the Effectiveness of Hidden Context Usage by Machine Learning Methods under Conditions of Increased Entropy of Noise , 2017, 2017 3rd IEEE International Conference on Cybernetics (CYBCON).

[2]  Berin Martini,et al.  Embedded Streaming Deep Neural Networks Accelerator With Applications , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[3]  Paul Rad,et al.  Distributed Edge Cloud R-CNN for Real Time Object Detection , 2018, 2018 World Automation Congress (WAC).

[4]  Charles L. Forgy,et al.  Rete: a fast algorithm for the many pattern/many object pattern match problem , 1991 .

[5]  Ling Shao,et al.  Transfer Learning for Visual Categorization: A Survey , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[6]  H. Tenhunen,et al.  Edge-AI in LoRa-based Health Monitoring: Fall Detection System with Fog Computing and LSTM Recurrent Neural Networks , 2019, 2019 42nd International Conference on Telecommunications and Signal Processing (TSP).

[7]  Hong Ping Zhao,et al.  Distributed Deep Neural Networks with System Cost Minimization in Fog Networks , 2018, TENCON 2018 - 2018 IEEE Region 10 Conference.

[8]  H. T. Kung,et al.  Distributed Deep Neural Networks Over the Cloud, the Edge and End Devices , 2017, 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS).

[9]  Jian Yu,et al.  EdgeCNN: A Hybrid Architecture for Agile Learning of Healthcare Data from IoT Devices , 2018, 2018 IEEE 24th International Conference on Parallel and Distributed Systems (ICPADS).

[10]  Ming-Hwa Sheu,et al.  Implementation of FPGA-based Accelerator for Deep Neural Networks , 2019, 2019 IEEE 22nd International Symposium on Design and Diagnostics of Electronic Circuits & Systems (DDECS).

[11]  Raymond Y. K. Lau,et al.  Hyperspectral Image Classification With Deep Learning Models , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[12]  Claudia Linnhoff-Popien,et al.  A Context Modeling Survey , 2004 .

[13]  Jiangchuan Liu,et al.  When deep learning meets edge computing , 2017, 2017 IEEE 25th International Conference on Network Protocols (ICNP).

[14]  Xuegong Zhou,et al.  A high performance FPGA-based accelerator for large-scale convolutional neural networks , 2016, 2016 26th International Conference on Field Programmable Logic and Applications (FPL).

[15]  Yung-Hsiang Lu,et al.  Cloud Computing for Mobile Users: Can Offloading Computation Save Energy? , 2010, Computer.

[16]  Hang-Bong Kang,et al.  Urban Safety Prediction Using Context and Object Information via Double-Column Convolutional Neural Network , 2016, 2016 13th Conference on Computer and Robot Vision (CRV).

[17]  Gregory D. Abowd,et al.  Towards a Better Understanding of Context and Context-Awareness , 1999, HUC.

[18]  Kuruvilla Varghese,et al.  Runtime Programmable and Memory Bandwidth Optimized FPGA-Based Coprocessor for Deep Convolutional Neural Network , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[19]  Irina Perfilieva,et al.  Logical foundations of rule-based systems , 2006, Fuzzy Sets Syst..

[20]  Pete Beckman,et al.  Waggle: An open sensor platform for edge computing , 2016, 2016 IEEE SENSORS.

[21]  Andrew Lumsdaine,et al.  Depth of Field in Plenoptic Cameras , 2009, Eurographics.

[22]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[23]  Weisong Shi,et al.  Edge Computing: Vision and Challenges , 2016, IEEE Internet of Things Journal.

[24]  Mohamed H. Elgazzar.  Perspectives on M2M protocols , 2015, 2015 IEEE Seventh International Conference on Intelligent Computing and Information Systems (ICICIS).

[25]  Ana Belen Lago,et al.  An Infrastructure to Enable Lightweight Context-Awareness for Mobile Users , 2013, Sensors.

[26]  In-So Kweon,et al.  EPINET: A Fully-Convolutional Neural Network Using Epipolar Geometry for Depth from Light Field Images , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[27]  Régis Guinvarc'h,et al.  Distributing Deep Neural Networks for Maximising Computing Capabilities and Power Efficiency in Swarm , 2019, 2019 IEEE International Symposium on Circuits and Systems (ISCAS).

[28]  Arijit Mukherjee,et al.  Implementing Deep Learning and Inferencing on Fog and Edge Computing Systems , 2018, 2018 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops).

[29]  Zongpu Zhang,et al.  Towards Ubiquitous Intelligent Computing: Heterogeneous Distributed Deep Neural Networks , 2018, IEEE Transactions on Big Data.

[30]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Heng Li,et al.  A Novel Hyperspectral Image Clustering Method With Context-Aware Unsupervised Discriminative Extreme Learning Machine , 2018, IEEE Access.

[32]  Ivan Hedi,et al.  IoT network protocols comparison for the purpose of IoT constrained networks , 2017, 2017 40th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO).