Structuring host communication software for quality-of-service guarantees

STRUCTURING HOST COMMUNICATION SOFTWARE FOR QUALITY OF SERVICE GUARANTEES by Ashish Mehra Chair: Kang G. Shin This dissertation addresses several issues involved in structuring communication software at end hosts to provide per-connection quality of service (QoS) guarantees. Our primary thrust is on realizing deterministic QoS guarantees in an experimental setting. We design a novel QoS-sensitive communication subsystem architecture that provides components and mechanisms to manage communication resources in a QoS-sensitive fashion. This architecture is based on three key design principles: maintenance of QoS guarantees, overload protection via per-connection tra c enforcement, and fairness to best-e ort tra c. We demonstrate the e cacy of the architecture via experiments using our x-kernel-based prototype implementation. We develop admission control extensions to capture important implementation-related aspects not considered by theoretical resource management policies proposed in the literature. Communication performance is signi cantly a ected by implementation-related aspects such as preemption overheads and constraints, simultaneous management of CPU and link bandwidth, link scheduling paradigms at sending hosts, and packet input mechanisms at receiving hosts. We build a fulledged guaranteed-QoS communication service on a contemporary microkernel operating system using a new service architecture integrating a QoS-aware application programming interface, a reliable signalling and resource reservation protocol, and QoS-sensitive data transmission and reception. We develop architectural and admission control enhancements that capture new overheads and constraints imposed by a microkernel server con guration. The service is implemented and evaluated on Pentium-based PCs running OSF MK 7.2 and communicating across switched Ethernet. To enhance the portability of guaranteed-QoS communication services, we design selfparameterizing protocol stacks that pro le and parameterize themselves appropriately during data transfer. We experimentally evaluate a self-parameterizing guaranteed-QoS communication service to demonstrate the feasibility of our approach. Self-parameterizing protocol stacks are a natural way to design portable QoS-sensitive communication software. To provide \better than best-e ort" Internet connectivity, the performance impact of supporting integrated services in TCP/IP protocol stacks must be assessed. We have designed, implemented, and evaluated an RSVP-based QoS architecture for TCP/IP protocol stacks supporting an integrated services Internet. Using detailed kernel-based pro ling of RS/6000-based servers running AIX 4.2, we explore the performance impact of QoS support in TCP/IP protocol stacks. We conclude by outlining strategies for integrating QoS-sensitive communication subsystems with QoS-sensitive application scheduling policies within host operating systems. ii CHAPTER

[1]  Martina Zitterbart,et al.  A Model for Flexible High-Performance Communication Subsystems , 1993, IEEE J. Sel. Areas Commun..

[2]  David P. Anderson,et al.  The performance of message‐passing using restricted virtual memory remapping , 1991, Softw. Pract. Exp..

[3]  Dilip D. Kandlur,et al.  Design and implementation of an RSVP-based quality of service architecture for an integrated services Internet , 1998, IEEE J. Sel. Areas Commun..

[4]  Sang Lyul Min,et al.  Analysis of cache-related preemption delay in fixed-priority preemptive scheduling , 1998, 17th IEEE Real-Time Systems Symposium.

[5]  Krishan K. Sabnani,et al.  VLSI implementations of communication protocols-a survey , 1989, IEEE J. Sel. Areas Commun..

[6]  Jochen Liedtke,et al.  OS-controlled cache predictability for real-time systems , 1997, Proceedings Third IEEE Real-Time Technology and Applications Symposium.

[7]  Arun N. Netravali,et al.  Design and implementation of a high-speed transport protocol , 1990, IEEE Trans. Commun..

[8]  Ramesh Nagarajan,et al.  HeiRAT: The Heidelberg Resource Administration Technique Design Philosophy and Goals , 1993, Kommunikation in Verteilten Systemen.

[9]  Gordon S. Blair,et al.  A Network Interface Unit to Support Continuous Media , 1993, IEEE J. Sel. Areas Commun..

[10]  Joseph Pasquale,et al.  Exploiting In-Kernel Data Paths to Improve I/O Throughput and CPU Availability , 1993, USENIX Winter.

[11]  Kang G. Shin,et al.  QoS-Sensitive Protocol Processing In Shared-Memory Multiprocessor Multimedia Servers , 1995, Third IEEE Workshop on the Architecture and Implementation of High Performance Communication Subsystems.

[12]  David K. Y. Yau,et al.  Adaptive rate-controlled scheduling for multimedia applications , 1997, MULTIMEDIA '96.

[13]  Larry L. Peterson,et al.  Network Subsystem Design : A Case for an Integrated Data Path , 1993 .

[14]  John K. Ousterhout,et al.  Why Aren't Operating Systems Getting Faster As Fast as Hardware? , 1990, USENIX Summer.

[15]  David P. Anderson,et al.  Metascheduling for continuous media , 1993, TOCS.

[16]  Andrew T. Campbell,et al.  A quality of service architecture , 1994, CCRV.

[17]  Kang G. Shin,et al.  Real-time communication in multi-hop networks , 1991, [1991] Proceedings. 11th International Conference on Distributed Computing Systems.

[18]  Guru M. Parulkar,et al.  A real-time upcall facility for protocol processing with QoS guarantees , 1995, SOSP.

[19]  Gerald W. Neufeld,et al.  A Multiprocessor Approach for Meeting the Processing Requirements for OSI , 1993, IEEE J. Sel. Areas Commun..

[20]  ZhangLixia,et al.  A measurement-based admission control algorithm for integrated services packet networks , 1995 .

[21]  Michael B. Jones,et al.  Support for User-Centric Modular Real-Time Resource Management in the Rialto Operating System , 1995, NOSSDAV.

[22]  Domenico Ferrari,et al.  Client requirements for real-time communication services , 1990, IEEE Communications Magazine.

[23]  Carl A. Waldspurger,et al.  Lottery and stride scheduling: flexible proportional-share resource management , 1995 .

[24]  Guru M. Parulkar,et al.  Bringing real-time scheduling theory and practice closer for multimedia computing , 1996, SIGMETRICS '96.

[25]  S. Pink,et al.  Distributed multimedia applications on gigabit networks , 1992, IEEE Network.

[26]  Jay K. Strosnider,et al.  Engineering and Analysis of Fixed Priority Schedulers , 1993, IEEE Trans. Software Eng..

[27]  Krithi Ramamritham,et al.  Integrated scheduling of multimedia and hard real-time tasks , 1996, 17th IEEE Real-Time Systems Symposium.

[28]  Erich M. Nahum,et al.  Performance issues in parallelized network protocols , 1994, OSDI '94.

[29]  Liviu Iftode,et al.  Software support for virtual memory-mapped communication , 1996, Proceedings of International Conference on Parallel Processing.

[30]  C.E. Catlett.  In search of gigabit applications , 1992, IEEE Communications Magazine.

[31]  Hui Zhang,et al.  Service disciplines for guaranteed performance service in packet-switching networks , 1995, Proc. IEEE.

[32]  Thomas F. La Porta,et al.  The MultiStream Protocol: A Highly Flexible High-Speed Transport Protocol , 1993, IEEE J. Sel. Areas Commun..

[33]  Larry L. Peterson,et al.  Making paths explicit in the Scout operating system , 1996, OSDI '96.

[34]  Kang G. Shin,et al.  SPIDER: flexible and efficient communication support for point-to-point distributed systems , 1994, 14th International Conference on Distributed Computing Systems.

[35]  Jason Nieh,et al.  The design, implementation and evaluation of SMART: a scheduler for multimedia applications , 1997, SOSP.

[36]  Onat Menzilcioglu,et al.  Nectar CAB: a high-speed network processor , 1991, [1991] Proceedings. 11th International Conference on Distributed Computing Systems.

[37]  Ian Wakeman,et al.  Implementing Real Time Packet Forwarding Policies Using Streams , 1995, USENIX.

[38]  Olof Hagsand,et al.  Workstation Support for Real-Time Multimedia Communication , 1994, USENIX Winter.

[39]  Douglas C. Schmidt,et al.  Measuring the performance of parallel message-based process architectures , 1995, Proceedings of INFOCOM'95.

[40]  Dinesh C. Verma,et al.  A Scheme for Real-Time Channel Establishment in Wide-Area Networks , 1990, IEEE J. Sel. Areas Commun..

[41]  K. Shin,et al.  Design Tradeo s in Implementing Real-Time Channels on Bus-Based Multiprocessor Hosts , 1995 .

[42]  K. K. Ramakrishnan,et al.  Eliminating receive livelock in an interrupt-driven kernel , 1996, TOCS.

[43]  Ramesh Govindan,et al.  Scheduling and IPC mechanisms for continuous media , 1991, SOSP '91.

[44]  David L. Black,et al.  FLIPC: A Low Latency Messaging System for Distributed Real Time Environments , 1996, USENIX Annual Technical Conference.

[45]  Evangelos P. Markatos,et al.  First-class user-level threads , 1991, SOSP '91.

[46]  Erich M. Nahum,et al.  Cache behavior of network protocols , 1997, SIGMETRICS '97.

[47]  Ragunathan Rajkumar,et al.  An interactive interface and RT-Mach support for monitoring and controlling resource management , 1995, Proceedings Real-Time Technology and Applications Symposium.

[48]  D.R. Cheriton,et al.  VMTP as the transport layer for high-performance distributed systems , 1989, IEEE Communications Magazine.

[49]  Larry L. Peterson,et al.  Experiences with a high-speed network adaptor: a software perspective , 1994 .

[50]  Brian N. Bershad,et al.  Protocol service decomposition for high-performance networking , 1994, SOSP '93.

[51]  Franco Travostino,et al.  Paths: programming with system resources in support of real-time distributed applications , 1996, Proceedings of WORDS'96. The Second Workshop on Object-Oriented Real-Time Dependable Systems.

[52]  David R. Cheriton,et al.  The VMP network adapter board (NAB): high-performance network communication for multiprocessors , 1988, SIGCOMM '88.

[53]  Robin Fairbairns,et al.  The Design and Implementation of an Operating System to Support Distributed Multimedia Applications , 1996, IEEE J. Sel. Areas Commun..

[54]  José Carlos Brustoloni,et al.  Copy emulation in checksummed, multiple-packet communication , 1997, Proceedings of INFOCOM '97.

[55]  Henry M. Levy,et al.  Limits to low-latency communication on high-speed networks , 1993, TOCS.

[56]  Tatsuo Nakajima.  A dynamic QoS control based on optimistic processor reservation , 1996, Proceedings of the Third IEEE International Conference on Multimedia Computing and Systems.

[57]  D. Estrin,et al.  RSVP: a new resource reservation protocol , 1993, IEEE Communications Magazine.

[58]  David D. Clark,et al.  The structuring of systems using upcalls , 1985, SOSP '85.

[59]  Srinivasan Keshav,et al.  Design, implementation, and performance of a native mode ATM transport layer , 1996, Proceedings of IEEE INFOCOM '96. Conference on Computer Communications.

[60]  Kang G. Shin,et al.  The END: a network adapter design tool , 1998, Proceedings. IEEE INFOCOM '98, the Conference on Computer Communications. Seventeenth Annual Joint Conference of the IEEE Computer and Communications Societies. Gateway to the 21st Century (Cat. No.98.

[61]  Jeffrey C. Mogul,et al.  Network locality at the scale of processes , 1991, SIGCOMM '91.

[62]  Ragunathan Rajkumar,et al.  Predictable communication protocol processing in real-time Mach , 1996, Proceedings Real-Time Technology and Applications.

[63]  Jane W.-S. Liu,et al.  A method for bounding the effect of DMA I/O interference on program execution time , 1996, 17th IEEE Real-Time Systems Symposium.

[64]  Brian N. Bershad,et al.  Efficient Packet Demultiplexing for Multiple Endpoints and Large Messages , 1994, USENIX Winter.

[65]  Isidor Kouvelas,et al.  Overcoming workstation scheduling problems in a real-time audio tool , 1997 .

[66]  Guru M. Parulkar,et al.  Experimental evaluation of SUNOS IPC and TCP/IP protocol implementation , 1993, TNET.

[67]  H. T. Kung,et al.  The design of nectar: a network backplane for heterogeneous multicomputers , 1989, ASPLOS III.

[68]  Larry L. Peterson,et al.  Fbufs: a high-bandwidth cross-domain transfer facility , 1994, SOSP '93.

[69]  Abhay Parekh,et al.  A generalized processor sharing approach to flow control in integrated services networks-the single node case , 1992, [Proceedings] IEEE INFOCOM '92: The Conference on Computer Communications.

[70]  Steven McCanne,et al.  The BSD Packet Filter: A New Architecture for User-level Packet Capture , 1993, USENIX Winter.

[71]  Kai Li,et al.  Protected, user-level DMA for the SHRIMP network interface , 1996, Proceedings. Second International Symposium on High-Performance Computer Architecture.

[72]  David Banks,et al.  User-space protocols deliver high performance to applications on a low-cost Gb/s LAN , 1994, SIGCOMM '94.

[73]  Ramesh Govindan,et al.  Support for continuous media in the DASH system , 1990, Proceedings.,10th International Conference on Distributed Computing Systems.

[74]  Anindo Banerjea,et al.  The Tenet real-time protocol suite: design, implementation, and experiences , 1996, TNET.

[75]  James W. Layland,et al.  Scheduling Algorithms for Multiprogramming in a Hard-Real-Time Environment , 1989, JACM.

[76]  Ragunathan Rajkumar,et al.  Experiences with Processor Reservation and Dynamic QOS in Real-Time Mach , 1996 .

[77]  Michael B. Jones,et al.  An overview of the Rialto real-time architecture , 1996, EW 7.

[78]  David P. Anderson,et al.  Process Structure and Scheduling in Real-Time Protocol Implementations , 1991, Kommunikation in Verteilten Systemen.

[79]  D.E. Tolmie.  Gigabit networking , 1992, IEEE LTS.

[80]  Joseph Pasquale,et al.  Measurement, Analysis, and Improvement of UDP/IP Throughput for the DECstation 5000 , 1993, USENIX Winter.

[81]  Margo I. Seltzer,et al.  Operating system benchmarking in the wake of lmbench: a case study of the performance of NetBSD on the Intel x86 architecture , 1997, SIGMETRICS '97.

[82]  B.S. Davie.  The Architecture and Implementation of a High-Speed Host Interface , 1993, IEEE J. Sel. Areas Commun..

[83]  Klara Nahrstedt,et al.  Design, Implementation, and Experiences of the OMEGA End-Point Architecture , 1996, IEEE J. Sel. Areas Commun..

[84]  Larry L. Peterson,et al.  PathFinder: A Pattern-Based Packet Classifier , 1994, OSDI.

[85]  Stefan Savage,et al.  Processor Capacity Reserves for Multimedia Operating Systems , 1993 .

[86]  Peter Steenkiste.  A systematic approach to host interface design for high-speed networks , 1994, Computer.

[87]  Margo I. Seltzer,et al.  Self-monitoring and self-adapting operating systems , 1997, Proceedings. The Sixth Workshop on Hot Topics in Operating Systems (Cat. No.97TB100133).

[88]  Henning Schulzrinne,et al.  Real-time communication in packet-switched networks , 1994, Proc. IEEE.

[89]  Gordon S. Blair,et al.  Micro-Kernel Support for Continuous Media in Distributed Systems , 1994, Comput. Networks ISDN Syst..

[90]  Joseph Pasquale,et al.  Container shipping: operating system support for I/O-intensive applications , 1994, Computer.

[91]  R. Jain.  Congestion control in computer networks: issues and trends , 1990, IEEE Network.

[92]  Peter B. Danzig,et al.  A measurement-based admission control algorithm for integrated services packet networks , 1995, SIGCOMM '95.

[93]  Thorsten von Eicken,et al.  U-Net: a user-level network interface for parallel and distributed computing , 1995, SOSP.

[94]  Enrico Vicario,et al.  Interprocess Communication Dependency on Network Load , 1991, IEEE Trans. Software Eng..

[95]  Kang G. Shin,et al.  Structuring communication software for quality-of-service guarantees , 1996, 17th IEEE Real-Time Systems Symposium.

[96]  Andrew T. Campbell,et al.  QoS adaptive transports: delivering scalable media to the desktop , 1997 .

[97]  Luca Delgrossi,et al.  Internet Stream Protocol Version 2 (ST2) Protocol Specification - Version ST2+ , 1995, RFC.

[98]  Bryan Ford,et al.  CPU inheritance scheduling , 1996, OSDI '96.

[99]  Jay K. Strosnider,et al.  A modeling methodology for real-time/multimedia operating systems , 1995, Proceedings Real-Time Technology and Applications Symposium.

[100]  S. Lam,et al.  An Architecture towards Eecient Os Support for Distributed Multimedia , 1996 .

[101]  Lakshman Krishnamurthy.  AQUA: an adaptive quality of service architecture for distributed multimedia applications , 1997 .

[102]  Gordon S. Blair,et al.  The Design of a QoS-Controlled ATM-Based Communications System in Chorus , 1995, IEEE J. Sel. Areas Commun..

[103]  Douglas C. Schmidt,et al.  The Performance of Alternative Threading Architectures for Parallel Communication Subsystems , 1998 .

[104]  Mischa Schwartz,et al.  Transport Protocol Processing at GBPS Rates , 1990, SIGCOMM.

[105]  Peter Steenkiste,et al.  Analyzing communication latency using the Nectar communication processor , 1992, SIGCOMM '92.

[106]  Jonathan M. Smith,et al.  Hardware/Software Organization of a High-Performance ATM Host Interface , 1993, IEEE J. Sel. Areas Commun..

[107]  Sherali Zeadally,et al.  An Analysis of Process and Memory Models to Support High-Speed Networking in a UNIX Environment , 1996, USENIX Annual Technical Conference.

[108]  Henning Schulzrinne,et al.  RTP: A Transport Protocol for Real-Time Applications , 1996, RFC.

[109]  Brian N. Bershad,et al.  An Extensible Protocol Architecture for Application-Specific Networking , 1996, USENIX Annual Technical Conference.

[110]  Dawson R. Engler,et al.  DPF: Fast, Flexible Message Demultiplexing Using Dynamic Code Generation , 1996, SIGCOMM.

[111]  Hideyuki Tokuda,et al.  Preemptibility in real-time operating systems , 1992, [1992] Proceedings Real-Time Systems Symposium.

[112]  David G. Korn.  Porting UNIX to windows NT , 1997 .

[113]  Richard W. Watson,et al.  Gaining efficiency in transport services by appropriate design and implementation choices , 1987, TOCS.

[114]  Rene Leonardo Cruz.  A Calculus for Network Delay and a Note on Topologies of Interconnection Networks , 1987 .

[115]  Larry L. Peterson,et al.  The x-Kernel: An Architecture for Implementing Network Protocols , 1991, IEEE Trans. Software Eng..

[116]  Larry L. Peterson,et al.  Increasing network throughput by integrating protocol layers , 1993, TNET.

[117]  Trevor Blackwell.  Speeding up Protocols for Small Messages , 1996, SIGCOMM.

[118]  K. K. Ramakrishnan,et al.  Performance Considerations in Designing Network Interfaces , 1993, IEEE J. Sel. Areas Commun..

[119]  David C. Feldmeier.  A SURVEY OF HIGH PERFORMANCE PROTOCOL IMPLEMENTATION TECHNIQUES , 1994 .

[120]  Bruce S. Davie,et al.  Integration of Real-time Services in an IP-ATM Network Architecture , 1995, RFC.

[121]  Carl Staelin,et al.  lmbench: Portable Tools for Performance Analysis , 1996, USENIX Annual Technical Conference.

[122]  Scott Shenker,et al.  Integrated Services in the Internet Architecture : an Overview Status of this Memo , 1994 .

[123]  Douglas C. Schmidt,et al.  Transport System Architecture Services for High-Performance Communications Systems , 1993, IEEE J. Sel. Areas Commun..

[124]  Robbert van Renesse.  Masking the Overhead of Protocol Layering , 1996, SIGCOMM.

[125]  George Eckel.  Inside Windows NT , 1993 .

[126]  Kang G. Shin,et al.  Resource management for real-time communication: making theory meet practice , 1996, Proceedings Real-Time Technology and Applications.

[127]  Peter Steenkiste,et al.  Buffer management and flow control in the Credit Net ATM host interface , 1995, Proceedings of 20th Conference on Local Computer Networks.

[128]  Van Jacobson,et al.  Link-sharing and resource management models for packet networks , 1995, TNET.

[129]  Anoop Gupta,et al.  Process control and scheduling issues for multiprogrammed shared-memory multiprocessors , 1989, SOSP '89.

[130]  Hideyuki Tokuda,et al.  Real-Time Mach: Towards a Predictable Real-Time System , 1990, USENIX MACH Symposium.

[131]  J.M. Smith,et al.  Giving applications access to Gb/s networking , 1993, IEEE Network.

[132]  Tatsuo Nakajima,et al.  Virtual memory management for interactive continuous media applications , 1997, Proceedings of IEEE International Conference on Multimedia Computing and Systems.

[133]  Larry L. Peterson,et al.  Analysis of techniques to improve protocol processing latency , 1996, SIGCOMM '96.

[134]  Jeffrey C. Mogul,et al.  The effect of context switches on cache performance , 1991, ASPLOS IV.

[135]  Mats Björkman,et al.  Locking Effects in Multiprocessor Implementations of Protocols , 1994, J. High Speed Networks.

[136]  John Zolnowsky,et al.  Realtime Scheduling in SunOS 5.0 , 1992 .

[137]  Dawson R. Engler,et al.  ASHs: Application-Specific Handlers for High-Performance Messaging , 1996, SIGCOMM.

[138]  C. Murray Woodside,et al.  Alternative software architectures for parallel protocol execution with synchronous IPC , 1993, TNET.

[139]  Larry L. Peterson,et al.  A dynamic network architecture , 1992, TOCS.

[140]  Peter Druschel,et al.  Lazy receiver processing (LRP): a network subsystem architecture for server systems , 1996, OSDI '96.

[141]  Andrew G. Malis,et al.  ATM Signaling Support for IP over ATM , 1995, RFC.

[142]  Thu D. Nguyen,et al.  Implementing Network Protocols at User Level , 1993, SIGCOMM.

[143]  José Carlos Brustoloni,et al.  Evaluation of data passing and scheduling avoidance , 1997, Proceedings of 7th International Workshop on Network and Operating System Support for Digital Audio and Video (NOSSDAV '97).

[144]  John Potter,et al.  Operating system extensions for dynamic real-time applications , 1996, 17th IEEE Real-Time Systems Symposium.

[145]  Hussein M. Abdel-Wahab,et al.  A proportional share resource allocation algorithm for real-time, time-shared systems , 1996, 17th IEEE Real-Time Systems Symposium.

[146]  Brian N. Bershad,et al.  Scheduler activations: effective kernel support for the user-level management of parallelism , 1991, TOCS.

[147]  Scott Shenker,et al.  Supporting real-time applications in an Integrated Services Packet Network: architecture and mechanism , 1992, SIGCOMM '92.

[148]  Peter B. Danzig.  An analytical model of operating system protocol processing including effects of multiprogramming , 1991, SIGMETRICS '91.

[149]  Henning Schulzrinne,et al.  An evaluation of scheduling mechanisms for providing best-effort real-time communications in wide-area networks , 1994, Proceedings of INFOCOM '94 Conference on Computer Communications.

[150]  David A. Mosher,et al.  User-Process Communication Performance in Networks of Computers , 1988, IEEE Trans. Software Eng..

[151]  Joseph Pasquale,et al.  The importance of non-data touching processing overheads in TCP/IP , 1993, SIGCOMM '93.

[152]  Domenico Ferrari,et al.  Rate-controlled static-priority queueing , 1993, IEEE INFOCOM '93 The Conference on Computer Communications, Proceedings.

[153]  Milon Mackey,et al.  An implementation of the Hamlyn sender-managed interface architecture , 1996, OSDI '96.