DeFog: fog computing benchmarks

There are currently no benchmarks that can directly compare the performance of an application across the cloud-only, edge-only and cloud-edge (Fog) deployment platforms to obtain any insight on potential performance improvement. This paper proposes DeFog, a first Fog benchmarking suite to: (i) alleviate the burden of Fog benchmarking by using a standard methodology, and (ii) facilitate the understanding of the target platform by collecting a catalogue of relevant metrics for a set of benchmarks. The current portfolio of DeFog benchmarks comprises six relevant applications conducive to using the edge. Experimental studies are carried out on multiple target platforms to demonstrate the use of DeFog for collecting metrics related to application latencies (communication and computation), for understanding the impact of stress and concurrent users on application latencies, and for understanding the performance of deploying different combination of services of an application across the cloud and edge. DeFog is available for public download (https://github.com/qub-blesson/DeFog).

[1]  Mahadev Satyanarayanan,et al.  An empirical study of latency in an emerging class of edge computing applications for wearable cognitive assistance , 2017, SEC.

[2]  Rajkumar Buyya,et al.  iFogSim: A toolkit for modeling and simulation of resource management techniques in the Internet of Things, Edge and Fog computing environments , 2016, Softw. Pract. Exp..

[3]  Kay Römer,et al.  Moving Beyond Competitions: Extending D-Cube to Seamlessly Benchmark Low-Power Wireless Systems , 2018, 2018 IEEE Workshop on Benchmarking Cyber-Physical Networks and Systems (CPSBench).

[4]  Raja Lavanya,et al.  Fog Computing and Its Role in the Internet of Things , 2019, Advances in Computer and Electrical Engineering.

[5]  Michail Matthaiou,et al.  ENORM: A Framework For Edge NOde Resource Management , 2017, IEEE Transactions on Services Computing.

[6]  Alexander I. Rudnicky,et al.  Pocketsphinx: A Free, Real-Time Continuous Speech Recognition System for Hand-Held Devices , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[7]  Gang Lu,et al.  CloudRank-D: benchmarking and ranking cloud computing systems for data processing applications , 2012, Frontiers of Computer Science.

[8]  Klara Nahrstedt,et al.  Impact of Cloudlets on Interactive Mobile Cloud Applications , 2012, 2012 IEEE 16th International Enterprise Distributed Object Computing Conference.

[9]  Yolande Berbers,et al.  Benchmarking leading-edge mobile devices for data-intensive distributed mobile cloud applications , 2015, 2015 IEEE Symposium on Computers and Communication (ISCC).

[10]  Blesson Varghese,et al.  Container-Based Cloud Virtual Machine Benchmarking , 2016, 2016 IEEE International Conference on Cloud Engineering (IC2E).

[11]  Atay Ozgovde,et al.  EdgeCloudSim: An environment for performance evaluation of Edge Computing systems , 2017, 2017 Second International Conference on Fog and Mobile Edge Computing (FMEC).

[12]  Chunjie Luo,et al.  Characterizing data analysis workloads in data centers , 2013, 2013 IEEE International Symposium on Workload Characterization (IISWC).

[13]  David Bermbach,et al.  BenchFoundry: A Benchmarking Framework for Cloud Storage Services , 2017, ICSOC.

[14]  Blesson Varghese,et al.  Cloud Benchmarking for Maximising Performance of Scientific Applications , 2016, IEEE Transactions on Cloud Computing.

[15]  Peter Kilpatrick,et al.  Challenges and Opportunities in Edge Computing , 2016, 2016 IEEE International Conference on Smart Cloud (SmartCloud).

[16]  Jack J. Dongarra,et al.  The LINPACK Benchmark: past, present and future , 2003, Concurr. Comput. Pract. Exp..

[17]  Stacy Patterson,et al.  EdgeBench: Benchmarking Edge Computing Platforms , 2018, 2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion).

[18]  David Bermbach,et al.  MockFog: Emulating Fog Computing Infrastructure in the Cloud , 2019, 2019 IEEE International Conference on Fog Computing (ICFC).

[19]  Rajkumar Buyya,et al.  Next generation cloud computing: New trends and research directions , 2017, Future Gener. Comput. Syst..

[20]  David H. Bailey,et al.  The NAS parallel benchmarks summary and preliminary results , 1991, Proceedings of the 1991 ACM/IEEE Conference on Supercomputing (Supercomputing '91).

[21]  Mahadev Satyanarayanan,et al.  Demo: Scaling on the Edge – A Benchmarking Suite for Human-in-the-Loop Applicationss , 2018, 2018 IEEE/ACM Symposium on Edge Computing (SEC).

[22]  Liam O'Brien,et al.  On a Catalogue of Metrics for Evaluating Commercial Cloud Services , 2012, 2012 ACM/IEEE 13th International Conference on Grid Computing.

[23]  Weisong Shi,et al.  Edge Computing: Vision and Challenges , 2016, IEEE Internet of Things Journal.

[24]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Adam Silberstein,et al.  Benchmarking cloud serving systems with YCSB , 2010, SoCC '10.

[26]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[27]  Blesson Varghese,et al.  DocLite: A Docker-Based Lightweight Cloud Benchmarking Tool , 2016, 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid).

[28]  Xiaopei Wu,et al.  CAVBench: A Benchmark Suite for Connected and Autonomous Vehicles , 2018, 2018 IEEE/ACM Symposium on Edge Computing (SEC).

[29]  Mahadev Satyanarayanan,et al.  The Emergence of Edge Computing , 2017, Computer.

[30]  Blesson Varghese,et al.  Resource Management in Fog/Edge Computing , 2018, ACM Comput. Surv..

[31]  Gerhard P. Hancke,et al.  Benchmarking Internet of things devices , 2014, 2014 12th IEEE International Conference on Industrial Informatics (INDIN).

[32]  Luiz Fernando Bittencourt,et al.  MyiFogSim: A Simulator for Virtual Machine Migration in Fog Computing , 2017, UCC.

[33]  Daniel Sánchez,et al.  Tailbench: a benchmark suite and evaluation methodology for latency-critical applications , 2016, 2016 IEEE International Symposium on Workload Characterization (IISWC).