An open source framework based on Kafka-ML for Distributed DNN inference over the Cloud-to-Things continuum

The current dependency of Artificial Intelligence (AI) systems on Cloud computing implies higher transmission latency and bandwidth consumption. Moreover, it challenges the real-time monitoring of physical objects, e.g., the Internet of Things (IoT). Edge systems bring computing closer to end devices and support timesensitive applications. However, Edge systems struggle with state-of-the-art Deep Neural Networks (DNN) due to computational resource limitations. This paper proposes a technology framework that combines the Edge-Cloud architecture concept with BranchyNet advantages to support fault-tolerant and low-latency AI predictions. The implementation and evaluation of this framework allow assessing the benefits of running Distributed DNN (DDNN) in the Cloud-to-Things continuum. Compared to a Cloud-only deployment, the results obtained show an improvement of 45.34% in the response time. Furthermore, this proposal presents an extension for Kafka-ML that reduces rigidness over the Cloud-to-Things continuum managing and deploying DDNN.

[1]  H. T. Kung,et al.  Distributed Deep Neural Networks Over the Cloud, the Edge and End Devices , 2017, 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS).

[2]  Tao Zhang,et al.  Fog and IoT: An Overview of Research Opportunities , 2016, IEEE Internet of Things Journal.

[3]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[4]  Luiz Angelo Steffenel,et al.  Assessing the impact of unbalanced resources and communications in edge computing , 2021, Pervasive Mob. Comput..

[5]  G. G. Stokes "J." , 1890, The New Yale Book of Quotations.

[6]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[7]  Spyros Makridakis,et al.  The Forthcoming Artificial Intelligence (AI) Revolution: Its Impact on Society and Firms , 2017 .

[8]  Rajkumar Buyya,et al.  HealthFog: An Ensemble Deep Learning based Smart Healthcare System for Automatic Diagnosis of Heart Diseases in Integrated IoT and Fog Computing Environments , 2019, Future Gener. Comput. Syst..

[9]  Antonio Brogi,et al.  Lightweight self-organising distributed monitoring of Fog infrastructures , 2021, Future Gener. Comput. Syst..

[10]  Markus Voelter,et al.  State of the Art , 1997, Pediatric Research.

[11]  Mianxiong Dong,et al.  AAIoT: Accelerating Artificial Intelligence in IoT Systems , 2019, IEEE Wireless Communications Letters.

[12]  Hui Li,et al.  Convolutional neural network‐based data anomaly detection method using multiple information for structural health monitoring , 2018, Structural Control and Health Monitoring.

[13]  Luis Alonso,et al.  Middleware and communication technologies for structural health monitoring of critical infrastructures: A survey , 2018, Comput. Stand. Interfaces.

[14]  Mario Di Francesco,et al.  Distributed Inference Acceleration with Adaptive DNN Partitioning and Offloading , 2020, IEEE INFOCOM 2020 - IEEE Conference on Computer Communications.

[15]  Dan Wang,et al.  Dynamic Adaptive DNN Surgery for Inference Acceleration on the Edge , 2019, IEEE INFOCOM 2019 - IEEE Conference on Computer Communications.

[16]  Matthias Sax,et al.  Apache Kafka , 2019, Encyclopedia of Big Data Technologies.

[17]  Miss A.O. Penney (b) , 1974, The New Yale Book of Quotations.

[18]  Raja Lavanya,et al.  Fog Computing and Its Role in the Internet of Things , 2019, Advances in Computer and Electrical Engineering.

[19]  Hui Li,et al.  Computer vision and deep learning–based data anomaly detection method for structural health monitoring , 2019 .

[20]  Paulo F. Pires,et al.  Leveraging Edge Intelligence for Video Analytics in Smart City Applications , 2020, Inf..

[21]  Rajkumar Buyya,et al.  EdgeLens: Deep Learning based Object Detection in Integrated IoT, Fog and Cloud Computing Environments , 2019, 2019 4th International Conference on Information Systems and Computer Networks (ISCON).

[22]  Mohamed Elhoseny,et al.  Deep learning model for real-time image compression in Internet of Underwater Things (IoUT) , 2020, Journal of Real-Time Image Processing.

[23]  Jérémy Robert,et al.  IoTEF: A Federated Edge-Cloud Architecture for Fault-Tolerant IoT Applications , 2020, Journal of Grid Computing.

[24]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[25]  Rajkumar Buyya,et al.  Distributed data stream processing and edge computing: A survey on resource elasticity and future directions , 2017, J. Netw. Comput. Appl..

[26]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[27]  H. T. Kung,et al.  BranchyNet: Fast inference via early exiting from deep neural networks , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[28]  Weisong Shi,et al.  Edge Computing: Vision and Challenges , 2016, IEEE Internet of Things Journal.

[29]  X. W. Ye,et al.  A review on deep learning-based structural health monitoring of civil infrastructures , 2019 .