A dynamically reconfigurable and extensible operating system

Operating systems are constantly getting more complex in the functionality they support, due to the increasing demands made by modem hardware and software innovations. Basing the kernel design on co-operating and modular services incorportating a flexible communications infrastructure with run-time binding makes the operating system dynamically configurable and extensible. These features aid in the management of system complexity, while also resulting in several software engineering and performance benefits. Configurability gives the operating system designer and implementor the freedom to build a large number of components, which can be composed into different configurations depending upon the final system requirements. System components can be built and debugged in a user address space, and then transparently migrated into the kernel address space for performance once they have been demonstrated correct. This removes one of the major obstacles to developing kernel services, that of the necessity to reboot the system after each change to the service code. The system administrator can also reconfigure the system, providing similar advantages, and allowing dynamic system upgrades to be made, reducing system downtime. Extensibility lets new functionality be integrated into the operating system. This can be done on an application specific basis. This enables the development of in-kernel applications in cases where high performance is required, such as for dedicated file servers. It is also possible for applications to interpose specialised kernel services, allowing them to dramatically increase their performance and aggregate system throughput when the default system policies are ill-matched to their behaviour. The Kea operating system has been designed and implemented to be dynamically configurable and extensible. The design of the system features that make these features possible are described. Experimental results are shown that demonstrate that Kea offers comparable performance to a traditional operating system on the same hardware, and that extensibility can be used to increase performance for selected applications.

[1]  Brian N. Bershad,et al.  Scheduler activations: effective kernel support for the user-level management of parallelism , 1991, TOCS.

[2]  Dylan McNamee,et al.  Extending the Mach External Pager Interface to Accomodate User-Level Page Replacement Policies , 1990, USENIX MACH Symposium.

[3]  Meng Chang Chen,et al.  HiPEC: high performance external virtual memory caching , 1994, OSDI '94.

[4]  Markus Mock,et al.  Automatic Dynamic Compilation Support for Event Dispatching in Extensible Systems , 1998 .

[5]  Ronald L. Rivest,et al.  The MD5 Message-Digest Algorithm , 1992, RFC.

[6]  Brian N. Bershad,et al.  Fast, effective dynamic compilation , 1996, PLDI '96.

[7]  Norman C. Hutchinson,et al.  Kea-a dynamically extensible and configurable operating system kernel , 1996, Proceedings of International Conference on Configurable Distributed Systems.

[8]  Jay Lepreau,et al.  Evolving Mach 3.0 to A Migrating Thread Model , 1994, USENIX Winter.

[9]  William J. Dally,et al.  Hardware support for fast capability-based addressing , 1994, ASPLOS VI.

[10]  Robbert van Renesse,et al.  Performance of the world's fastest distributed operating system , 1988, OPSR.

[11]  Chorus Systemes,et al.  Overview of the CHORUS? Distributed Operating Systems , 1991 .

[12]  William A. Wulf,et al.  HYDRA , 1974, Commun. ACM.

[13]  Brian N. Bershad,et al.  Lightweight remote procedure call , 1989, TOCS.

[14]  Joseph Pasquale,et al.  Container shipping: operating system support for I/O-intensive applications , 1994, Computer.

[15]  Jim Lipkis,et al.  A Second-Generation Micro-Kernel Based UNIX: Lessons in Performance and Compatibility , 1991, USENIX Winter.

[16]  William A. Wulf,et al.  Policy/mechanism separation in Hydra , 1975, SOSP.

[17]  Dinesh C. Kulkarni.  Pi: A New Approach to Operating System Structuring for Flexibility , 1993 .

[18]  Elliott I. Organick,et al.  The multics system: an examination of its structure , 1972 .

[19]  Brian N. Bershad,et al.  Dynamic binding for an extensible system , 1996, OSDI '96.

[20]  Norman C. Hutchinson,et al.  The possibilities and limitations of heterogeneous process migration , 1998 .

[21]  Michael L. Scott,et al.  Design Rationale for Psyche a General-Purpose Multiprocessor Operating System , 1988, ICPP.

[22]  Jonathan Walpole,et al.  Modularity and Interfaces in Micro-kernel Design and Implementation: A Case Study of Chorus on the HP PA-Risc , 1992, USENIX Workshop on Microkernels and Other Kernel Architectures.

[23]  Robert Wahbe,et al.  Efficient software-based fault isolation , 1994, SOSP '93.

[24]  Michael N. Nelson,et al.  Extensible file systems in spring , 1994, SOSP '93.

[25]  Richard D. Schlichting,et al.  A system for constructing configurable high-level protocols , 1995, SIGCOMM '95.

[26]  Jochen Liedtke,et al.  The performance of μ-kernel-based systems , 1997, SOSP.

[27]  Calton Pu,et al.  Optimistic incremental specialization: streamlining a commercial operating system , 1995, SOSP.

[28]  David L. Cohn,et al.  Protected shared libraries: a new approach to modularity and sharing , 1997 .

[29]  Peter Druschel,et al.  Modularity and protection should be decoupled , 1992, [1992] Proceedings Third Workshop on Workstation Operating Systems.

[30]  David R. Cheriton,et al.  A caching model of operating system kernel functionality , 1995, OPSR.

[31]  Kai Li,et al.  Implementation and performance of application-controlled file caching , 1994, OSDI '94.

[32]  Mark Nuttall,et al.  A brief survey of systems providing process or object migration facilities , 1994, OPSR.

[33]  Brian N. Bershad,et al.  Watchdogs - Extending the UNIX File System , 1988, Comput. Syst..

[34]  Trent Jaeger,et al.  Achieved IPC performance (still the foundation for extensibility) , 1997, Proceedings. The Sixth Workshop on Hot Topics in Operating Systems (Cat. No.97TB100133).

[35]  Keith Bostic,et al.  The design and implementa-tion of the 4.4BSD operating system , 1996 .

[36]  Robert Grimm,et al.  Application performance and flexibility on exokernel systems , 1997, SOSP.

[37]  Jeffrey C. Mogul,et al.  The packer filter: an efficient mechanism for user-level network code , 1987, SOSP '87.

[38]  Michael Burrows,et al.  Performance of Firefly RPC , 1989, SOSP '89.

[39]  Peter Druschel,et al.  Efficient support for incremental customization of OS services , 1993, Proceedings Third International Workshop on Object Orientation in Operating Systems.

[40]  Sanjay Radia.  Names, contexts, and closure mechanisms in distributed computing environments , 1990 .

[41]  J. Liedtke.  On -Kernel Construction , 1995 .

[42]  Margo I. Seltzer,et al.  Dealing with disaster: surviving misbehaved kernel extensions , 1996, OSDI '96.

[43]  Brian N. Bershad,et al.  Protocol service decomposition for high-performance networking , 1994, SOSP '93.

[44]  Dan Hildebrand,et al.  An Architectural Overview of QNX , 1992, USENIX Workshop on Microkernels and Other Kernel Architectures.

[45]  Larry L. Peterson,et al.  Scout: a communications-oriented operating system , 1995, Proceedings 5th Workshop on Hot Topics in Operating Systems (HotOS-V).

[46]  Michael Stumm,et al.  HFS: a performance-oriented flexible file system based on building-block compositions , 1996, IOPADS '96.

[47]  David L. Black,et al.  Microkernel operating system architecture and Mach , 1991 .

[48]  Yasuhiko Yokote,et al.  The Apertos reflective operating system: the concept and its implementation , 1992, OOPSLA.

[49]  Graham Hamilton,et al.  The Spring Nucleus: A Microkernel for Objects , 1993 .

[50]  Margo I. Seltzer,et al.  A Comparison of OS Extension Technologies , 1996, USENIX Annual Technical Conference.

[51]  Brian N. Bershad,et al.  Safe Dynamic Linking in an Extensible Operating System , 1999 .

[52]  Larry L. Peterson,et al.  Design of the x-kernel , 1988, SIGCOMM '88.

[53]  Jay Lepreau,et al.  The Flux OSKit: a substrate for kernel and language research , 1997, SOSP.

[54]  Andrew D. Birrell,et al.  An Introduction to Programming with Threads , 1996 .

[55]  Joseph Pasquale,et al.  Improving continuous-media playback performance with in-kernel data paths , 1994, 1994 Proceedings of IEEE International Conference on Multimedia Computing and Systems.

[56]  Larry L. Peterson,et al.  Fbufs: a high-bandwidth cross-domain transfer facility , 1994, SOSP '93.

[57]  F. J. Corbató,et al.  Multics: the first seven years , 1972, AFIPS '72 (Spring).

[58]  Steven McCanne,et al.  The BSD Packet Filter: A New Architecture for User-level Packet Capture , 1993, USENIX Winter.

[59]  Brian N. Bershad,et al.  The interaction of architecture and operating system design , 1991, ASPLOS IV.

[60]  Dawson R. Engler,et al.  Exokernel: an operating system architecture for application-level resource management , 1995, SOSP.

[61]  Brian N. Bershad,et al.  Using continuations to implement thread management and communication in operating systems , 1991, SOSP '91.

[62]  Daniel G. Bobrow,et al.  Book review: The Art of the MetaObject Protocol By Gregor Kiczales, Jim des Rivieres, Daniel G. and Bobrow(MIT Press, 1991) , 1991, SGAR.

[63]  Brian N. Bershad,et al.  Extensibility safety and performance in the SPIN operating system , 1995, SOSP.

[64]  Alessandro Forin,et al.  UNIX as an Application Program , 1990, USENIX Summer.

[65]  Norman C. Hutchinson,et al.  Dynamic service reconfiguration and migration in the Kea kernel , 1998, Proceedings. Fourth International Conference on Configurable Distributed Systems (Cat. No.98EX159).

[66]  Richard F. Rashid,et al.  DOS as a Mach 3.0 Application , 1991, USENIX MACH Symposium.

[67]  Peter Druschel,et al.  Service composition in Lipto , 1991, Proceedings 1991 International Workshop on Object Orientation in Operating Systems.

[68]  Calton Pu,et al.  Fast concurrent dynamic linking for an adaptive operating system , 1996, Proceedings of International Conference on Configurable Distributed Systems.

[69]  Michael L. Nelson,et al.  Extensible file system , 1994 .

[70]  Mike Hibler,et al.  Microkernels meet recursive virtual machines , 1996, OSDI '96.

[71]  Jay Lepreau,et al.  Fast and Flexible Shared Libraries , 1993, USENIX Summer.

[72]  Anoop Gupta,et al.  The impact of architectural trends on operating system performance , 1995, SOSP.

[73]  Michael N. Nelson,et al.  An Implementation of UNIX on an Object-oriented Operating System , 1993, USENIX Winter.

[74]  Paul J. Leach,et al.  An Extensible I/O System , 1986, USENIX Summer.

[75]  Karsten Schwan,et al.  Experimentation with a reconfigurable micro-kernel , 1993 .

[76]  Jochen Liedtke,et al.  Improving IPC by kernel design , 1994, SOSP '93.

[77]  George C. Necula,et al.  Safe kernel extensions without run-time checking , 1996, OSDI '96.

[78]  Norman C. Hutchinson,et al.  Real Time Threads Interface , 1995 .

[79]  Alberto Bartoli,et al.  Wide-address spaces: exploring the design space , 1993, OPSR.

[80]  Calton Pu,et al.  Threads and input/output in the synthesis kernal , 1989, SOSP '89.

[81]  Calton Pu,et al.  The Synthesis Kernel , 1988, Comput. Syst..

[82]  Michael B. Jones,et al.  The Tiger Video Fileserver , 1996 .

[83]  Michael B. Jones,et al.  Interposition agents: transparently interposing user code at the system interface , 1994, SOSP '93.

[84]  Andrew W. Appel,et al.  Virtual memory primitives for user programs , 1991, ASPLOS IV.

[85]  Larry L. Peterson,et al.  Beyond micro-kernel design: decoupling modularity and protection in Lipto , 1992, [1992] Proceedings of the 12th International Conference on Distributed Computing Systems.

[86]  David R. Cheriton,et al.  Application-controlled physical memory using external page-cache management , 1992, ASPLOS V.

[87]  David D. Clark,et al.  The structuring of systems using upcalls , 1985, SOSP '85.

[88]  Pattie Maes,et al.  Concepts and experiments in computational reflection , 1987, OOPSLA '87.

[89]  Dawson R. Engler,et al.  Exterminate all operating system abstractions , 1995, Proceedings 5th Workshop on Hot Topics in Operating Systems (HotOS-V).

[90]  Gregor Kiczales,et al.  A new model of abstraction for operating system design , 1992, [1992] Proceedings of the Second International Workshop on Object Orientation in Operating Systems.

[91]  Larry L. Peterson,et al.  The x-kernel: a platform for accessing internet resources , 1990, Computer.

[92]  John K. Ousterhout,et al.  Why Aren't Operating Systems Getting Faster As Fast as Hardware? , 1990, USENIX Summer.

[93]  Brian N. Bershad,et al.  The increasing irrelevance of IPC Performance for Micro-kernel-Based Operating Systems , 1992, USENIX Workshop on Microkernels and Other Kernel Architectures.

[94]  Michel Banâtre,et al.  How to Design Reliable Servers using Fault Tolerant Micro-Kernel Mechanisms , 1991, USENIX MACH Symposium.