DeepRT: A Soft Real Time Scheduler for Computer Vision Applications on the Edge

The ubiquity of smartphone cameras and IoT cameras, together with the recent boom of deep learning and deep neural networks, proliferate various computer vision driven mobile and IoT applications deployed on the edge. This paper focuses on applications which make soft real time requests to perform inference on their data – they desire prompt responses within designated deadlines, but occasional deadline misses are acceptable. Supporting soft real time applications on a multi-tenant edge server is not easy, since the requests sharing the limited GPU computing resources of an edge server interfere with each other. In order to tackle this problem, we comprehensively evaluate how latency and throughput respond to different GPU execution plans. Based on this analysis, we propose a GPU scheduler, DeepRT, which provides latency guarantee to the requests while maintaining high overall system throughput. The key component of DeepRT, DisBatcher, batches data from different requests as much as possible while it is proven to provide latency guarantee for requests admitted by an Admission Control Module. DeepRT also includes an Adaptation Module which tackles overruns. Our evaluation results show that DeepRT outperforms state-of-the-art works in terms of the number of deadline misses and throughput.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[3]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Matti Siekkinen,et al.  Latency and throughput characterization of convolutional neural networks for mobile computer vision , 2018, MMSys.

[5]  Mateo Valero,et al.  Enabling preemptive multiprogramming on GPUs , 2014, 2014 ACM/IEEE 41st International Symposium on Computer Architecture (ISCA).

[6]  Jinwoo Shin,et al.  Neural Adaptive Content-aware Internet Video Delivery , 2018, OSDI.

[7]  Zhou Fang,et al.  Serving deep neural networks at the cloud edge for vision applications on mobile platforms , 2019, MMSys.

[8]  Sanjoy K. Baruah,et al.  Schedulability analysis of non-preemptive recurring real-time tasks , 2006, Proceedings 20th IEEE International Parallel & Distributed Processing Symposium.

[9]  Scott A. Mahlke,et al.  Chimera: Collaborative Preemption for Multitasking on a Shared GPU , 2015, ASPLOS.

[10]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[11]  Mohammad Hossein Samavatian,et al.  Adaptive parallel execution of deep neural networks on heterogeneous edge devices , 2019, SEC.

[12]  Sameh Elnikety,et al.  Swayam: distributed autoscaling to meet SLAs of machine learning inference services with resource efficiency , 2017, Middleware.

[13]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[14]  Paramvir Bahl,et al.  Live Video Analytics at Scale with Approximation and Delay-Tolerance , 2017, NSDI.

[15]  Lothar Thiele,et al.  Schedulability of event-driven code blocks in real-time embedded systems , 2002, DAC '02.

[16]  Luc Van Gool,et al.  A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Rami G. Melhem,et al.  Simultaneous Multikernel GPU: Multi-tasking throughput processors via fine-grained sharing , 2016, 2016 IEEE International Symposium on High Performance Computer Architecture (HPCA).

[18]  Rami G. Melhem,et al.  Quality of service support for fine-grained sharing on GPUs , 2017, 2017 ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA).

[19]  T. Abdelzaher,et al.  On Removing Algorithmic Priority Inversion from Mission-critical Machine Inference Pipelines , 2020, 2020 IEEE Real-Time Systems Symposium (RTSS).

[20]  Ole J. Mengshoel,et al.  QoS-Aware Scheduling of Heterogeneous Servers for Inference in Deep Neural Networks , 2017, CIKM.

[21]  Qiang Liu,et al.  An Edge Network Orchestrator for Mobile Augmented Reality , 2018, IEEE INFOCOM 2018 - IEEE Conference on Computer Communications.

[22]  Christophe Moy,et al.  On Schedulability Analysis of Non-cyclic Generalized Multiframe Tasks , 2010, 2010 22nd Euromicro Conference on Real-Time Systems.

[23]  Klara Nahrstedt,et al.  CPU service classes for multimedia applications , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[24]  Robert B. Miller,et al.  Response time in man-computer conversational transactions , 1899, AFIPS Fall Joint Computing Conference.

[25]  Sanjoy K. Baruah.  The Non-cyclic Recurring Real-Time Task Model , 2010, 2010 31st IEEE Real-Time Systems Symposium.

[26]  Aloysius K. Mok,et al.  A multiframe model for real-time tasks , 1996, 17th IEEE Real-Time Systems Symposium.

[27]  Sanjoy K. Baruah,et al.  Preemptive Uniprocessor Scheduling of Non-cyclic GMF Task Systems , 2010, 2010 IEEE 16th International Conference on Embedded and Real-Time Computing Systems and Applications.

[28]  Klara Nahrstedt,et al.  Janus: a cross-layer soft real-time architecture for virtualization , 2010, HPDC '10.

[29]  Cecilia Ekelin,et al.  Clairvoyant non-preemptive EDF scheduling , 2006, 18th Euromicro Conference on Real-Time Systems (ECRTS'06).

[30]  Ada Gavrilovska,et al.  Couper: DNN model slicing for visual analytics containers at the edge , 2019, SEC.

[31]  Changjun Jiang,et al.  FLEP: Enabling Flexible and Efficient Preemption on GPUs , 2017, ASPLOS.

[32]  Tao Mei,et al.  A Deep Learning-Based Approach to Progressive Vehicle Re-identification for Urban Surveillance , 2016, ECCV.

[33]  Xin Wang,et al.  Clipper: A Low-Latency Online Prediction Serving System , 2016, NSDI.

[34]  Gregory R. Ganger,et al.  Mainstream: Dynamic Stem-Sharing for Multi-Tenant Video Processing , 2018, USENIX Annual Technical Conference.

[35]  Indranil Gupta,et al.  4CeeD: Real-Time Data Acquisition and Analysis Framework for Material-Related Cyber-Physical Environments , 2017, 2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID).

[36]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Paul Muhlethaler,et al.  Optimality and non-preemptive real-time scheduling revisited , 1995 .

[38]  James H. Anderson,et al.  GPUSync: A Framework for Real-Time GPU Management , 2013, 2013 IEEE 34th Real-Time Systems Symposium.

[39]  Klara Nahrstedt,et al.  MIRAS: Model-based Reinforcement Learning for Microservice Resource Allocation over Scientific Workflows , 2019, 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS).

[40]  Christopher Olston,et al.  TensorFlow-Serving: Flexible, High-Performance ML Serving , 2017, ArXiv.

[41]  Wei Wang,et al.  MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving , 2019, USENIX Annual Technical Conference.

[42]  Qianlin Liang,et al.  AI on the Edge: Characterizing AI-based IoT Applications Using Specialized Edge Architectures , 2020, 2020 IEEE International Symposium on Workload Characterization (IISWC).

[43]  Justin Manweiler,et al.  OverLay: Practical Mobile Augmented Reality , 2015, MobiSys.