Towards Ubiquitous Intelligent Computing: Heterogeneous Distributed Deep Neural Networks

For the pursuit of ubiquitous computing, distributed computing systems containing the cloud, edge devices, and Internet-of-Things devices are highly demanded. However, existing distributed frameworks do not tailor for the fast development of Deep Neural Network (DNN), which is the key technique behind many intelligent applications nowadays. Based on prior exploration on distributed deep neural networks (DDNN), we propose Heterogeneous Distributed Deep Neural Network (HDDNN) over the distributed hierarchy, targeting at ubiquitous intelligent computing. While being able to support basic functionalities of DNNs, our framework is optimized for various types of heterogeneity, including heterogeneous computing nodes, heterogeneous neural networks, and heterogeneous system tasks. Besides, our framework features parallel computing, privacy protection and robustness, with other consideration for the combination of heterogeneous distributed system and DNN. Extensive experiments demonstrate that our framework is capable of utilizing hierarchical distributed system better for DNN and tailoring DNN for real-world distributed system properly, which is with low response time, high performance, and better user experience.

[1]  Albert G. Greenberg,et al.  The cost of a cloud: research problems in data center networks , 2008, CCRV.

[2]  Jiansheng Chen,et al.  Rethinking Feature Distribution for Loss Functions in Image Classification , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[3]  Arkady B. Zaslavsky,et al.  Context Aware Computing for The Internet of Things: A Survey , 2013, IEEE Communications Surveys & Tutorials.

[4]  Ching-Tang Fan,et al.  Heterogeneous Information Fusion and Visualization for a Large-Scale Intelligent Video Surveillance System , 2017, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[5]  Tao Zhang,et al.  Fog and IoT: An Overview of Research Opportunities , 2016, IEEE Internet of Things Journal.

[6]  A. Salman Avestimehr,et al.  A Scalable Framework for Wireless Distributed Computing , 2016, IEEE/ACM Transactions on Networking.

[7]  Christian Decker,et al.  A survey on organic smart labels for the Internet-of-Things , 2010, 2010 Seventh International Conference on Networked Sensing Systems (INSS).

[8]  Feng Gao,et al.  VSaaS Model on DRAGON-Lab , 2013 .

[9]  Forrest N. Iandola,et al.  How to scale distributed deep learning? , 2016, ArXiv.

[10]  Arkady B. Zaslavsky,et al.  Sensing as a Service and Big Data , 2013, ArXiv.

[11]  Mahadev Satyanarayanan,et al.  Towards wearable cognitive assistance , 2014, MobiSys.

[12]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[13]  Rita Cucchiara,et al.  Intelligent Video Surveillance as a Service , 2013, Intelligent Multimedia Surveillance.

[14]  H. T. Kung,et al.  Embedded Binarized Neural Networks , 2017, EWSN.

[15]  Paramvir Bahl,et al.  The Case for VM-Based Cloudlets in Mobile Computing , 2009, IEEE Pervasive Computing.

[16]  Anurag Agarwal,et al.  The Internet of Things—A survey of topics and trends , 2014, Information Systems Frontiers.

[17]  Gerald Penn,et al.  Convolutional Neural Networks for Speech Recognition , 2014, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[18]  Jonathan Le Roux,et al.  A Purely End-to-End System for Multi-speaker Speech Recognition , 2018, ACL.

[19]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Chin-Feng Lai,et al.  Intelligent home-appliance recognition over IoT cloud network , 2013, 2013 9th International Wireless Communications and Mobile Computing Conference (IWCMC).

[21]  Xiaohu You,et al.  Interference-Aware Wireless Networks for Home Monitoring and Performance Evaluation , 2018, IEEE Transactions on Automation Science and Engineering.

[22]  H. T. Kung,et al.  Distributed Deep Neural Networks Over the Cloud, the Edge and End Devices , 2017, 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS).

[23]  Ankur Bapna,et al.  The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation , 2018, ACL.

[24]  Weisong Shi,et al.  Edge Computing: Vision and Challenges , 2016, IEEE Internet of Things Journal.

[25]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[26]  Yinda Zhang,et al.  Deep Depth Completion of a Single RGB-D Image , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[27]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[28]  Raja Lavanya,et al.  Fog Computing and Its Role in the Internet of Things , 2019, Advances in Computer and Electrical Engineering.

[29]  M. Shamim Hossain,et al.  Cloud-assisted Industrial Internet of Things (IIoT) - Enabled framework for health monitoring , 2016, Comput. Networks.

[30]  Wu He,et al.  Internet of Things in Industries: A Survey , 2014, IEEE Transactions on Industrial Informatics.

[31]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[32]  Awais Ahmad,et al.  Urban planning and building smart cities based on the Internet of Things using Big Data analytics , 2016, Comput. Networks.

[33]  Wenhu Chen,et al.  Triangular Architecture for Rare Language Translation , 2018, ACL.

[34]  Dario Pompili,et al.  Collaborative Mobile Edge Computing in 5G Networks: New Paradigms, Scenarios, and Challenges , 2016, IEEE Communications Magazine.

[35]  Josef Spillner,et al.  Creating optimal cloud storage systems , 2013, Future Gener. Comput. Syst..

[36]  Khaled Ben Letaief,et al.  Mobile Edge Computing: Survey and Research Outlook , 2017, ArXiv.

[37]  Nikos Komodakis,et al.  Wide Residual Networks , 2016, BMVC.

[38]  Rodrigo Roman,et al.  Mobile Edge Computing, Fog et al.: A Survey and Analysis of Security Threats and Challenges , 2016, Future Gener. Comput. Syst..

[39]  Marcelo Bertalmío,et al.  A Decomposition Framework for Image Denoising Algorithms , 2016, IEEE Transactions on Image Processing.

[40]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[41]  Byung-Gon Chun,et al.  CloneCloud: elastic execution between mobile device and cloud , 2011, EuroSys '11.

[42]  Randy H. Katz,et al.  Above the Clouds: A Berkeley View of Cloud Computing , 2009 .

[43]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[44]  A. Kuo.  Opportunities and Challenges of Cloud Computing to Improve Health Care Services , 2011, Journal of medical Internet research.

[45]  Forrest N. Iandola,et al.  SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size , 2016, ArXiv.

[46]  Raouf Boutaba,et al.  Cloud computing: state-of-the-art and research challenges , 2010, Journal of Internet Services and Applications.

[47]  Karolj Skala,et al.  Scalable Distributed Computing Hierarchy: Cloud, Fog and Dew Computing , 2015, Open J. Cloud Comput..

[48]  Antonio Pescapè,et al.  Integration of Cloud computing and Internet of Things: A survey , 2016, Future Gener. Comput. Syst..

[49]  Forrest N. Iandola,et al.  FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Ran El-Yaniv,et al.  Binarized Neural Networks , 2016, ArXiv.

[51]  Marc'Aurelio Ranzato,et al.  Large Scale Distributed Deep Networks , 2012, NIPS.