Observations on Operating System Security Vulnerabilities

This thesis presents research on computer security vulnerabilities in general-purpose operating systems. The objective is to investigate intrusions in such systems in order to find and model the underlying generic weaknesses, i.e., weaknesses that would be applicable to many different systems. An attempt is made to create a conceptual basis for the generic modeling of vulnerabilities, addressing security concepts, definitions, and terminology. The investigation of intrusions is based on empirical data collected from three different systems, UNIX, Novell NetWare, and Windows NT. The UNIX and Novell NetWare data were generated from a number of practical intrusion experiments with Masters students, while the Windows NT data resulted from a security analysis that we performed ourselves. This analysis showed that Windows NT, initially thought to be quite secure, still displayed a significant number of vulnerabilities. A comparison with earlier UNIX analyses indicates that the security differences between the systems are related more to factors such as time on market and security-by-obscurity than to inherent security performance. A different approach was taken with the Novell NetWare system. Here, the initial experiments showed quite poor security behavior and we have investigated the security evolution as later versions of the system have been released and have reached the conclusion that, even though some effort has been made to improve security, no significant difference has been achieved, primarily due to backward compatibility requirements. In summary, the work performed here represents a further step towards a full understanding of the generic weaknesses and vulnerabilities that impair commercially available operating systems. This understanding is essential to our aspiration to make these systems secure, or at least sufficiently secure. It is expected that this work, when fully accomplished, will comprise a powerful basis for improving the security of operating systems, at least to the extent that research results are taken into consideration by software developers and manufacturers

[1]  Dan M. Nessett.  Factors Affecting Distributed System Security , 1986, 1986 IEEE Symposium on Security and Privacy.

[2]  Hugo Krawczyk,et al.  A Security Architecture for the Internet Protocol , 1999, IBM Syst. J..

[3]  Bruce Christianson,et al.  Remote booting in a hostile world: to whom am I speaking? [Computer security] , 1995 .

[4]  Sushil Jajodia,et al.  Computer Security, Fault Tolerance, And Software Assurance , 1999, IEEE Concurrency.

[5]  Eugene H. Spafford,et al.  Software vulnerability analysis , 1998 .

[6]  Edward N. Adams,et al.  Optimizing Preventive Service of Software Products , 1984, IBM J. Res. Dev..

[7]  Erland Jonsson,et al.  An integrated framework for security and dependability , 1998, NSPW '98.

[8]  C. Richard Attanasio,et al.  Penetrating an Operating System: A Study of VM/370 Integrity , 1976, IBM Syst. J..

[9]  Erland Jonsson,et al.  A Preliminary Evaluation of the Security of a Non-Distributed Version of Windows NT , 1997 .

[10]  Richard H. Baker,et al.  Computer security handbook (2nd ed.) , 1991 .

[11]  H. Spiifford,et al.  Crisis and Aftermath , 1999 .

[12]  Tomas Olovsson,et al.  Towards operational measures of computer security: Experimentation and modelling , 1995 .

[13]  Charles P. Pfleeger,et al.  Security in computing , 1988 .

[14]  Cynthia E. Irvine,et al.  THE NPS CISR GRADUATE PROGRAM IN INFOSEC: SIX YEARS OF EXPERIENCE , 1997 .

[15]  Jerome H. Saltzer,et al.  A hardware architecture for implementing protection rings , 1972, CACM.

[16]  T. Olovsson,et al.  On measurement of operational security , 1994, IEEE Aerospace and Electronic Systems Magazine.

[17]  Keith Bostic,et al.  The design and implementa-tion of the 4.4BSD operating system , 1996 .

[18]  Udo W. Pooch,et al.  A Unix network protocol security study: network information service , 1992, CCRV.

[19]  Maurice J. Bach.  The Design of the UNIX Operating System , 1986 .

[20]  Helen Custer,et al.  Inside Windows NT , 1992 .

[21]  Eugene H. Spafford,et al.  Are computer hacker break-ins ethical? , 1992, J. Syst. Softw..

[22]  Matt Bishop,et al.  A Critical Analysis of Vulnerability Taxonomies , 1996 .

[23]  Georgory White,et al.  Security across the curriculum: using computer secu-rity to teach computer science principles , 1997 .

[24]  Tomas Olovsson,et al.  On the modelling of preventive security based on a PC network intrusion experiment , 1996, ACISP.

[25]  Tom Sheldon.  Windows NT Security Handbook , 1996 .

[26]  Erland Jonsson,et al.  An Analysis of the Security of Windows NT , 1998 .

[27]  Ross J. Anderson.  Why cryptosystems fail , 1994, CACM.

[28]  E. L. Harder,et al.  The Institute of Electrical and Electronics Engineers, Inc. , 2019, 2019 IEEE International Conference on Software Architecture Companion (ICSA-C).

[29]  Erland Jonsson,et al.  How to systematically classify computer security intrusions , 1997, Proceedings. 1997 IEEE Symposium on Security and Privacy (Cat. No.97CB36097).

[30]  Deborah A. Frincke,et al.  Integrating Security into the Curriculum , 1998, Computer.

[31]  Tatu Ylonen,et al.  SSH: secure login connections over the internet , 1996 .

[32]  Roger R. Schell,et al.  Evaluating Security Properties of Computer Systems , 1983, 1983 IEEE Symposium on Security and Privacy.

[33]  Elizabeth D. Zwicky,et al.  Building internet firewalls , 1995 .

[34]  Thomas A. Longstaff,et al.  A common language for computer security incidents , 1998 .

[35]  Andrew Birrell,et al.  Implementing remote procedure calls , 1984, TOCS.

[36]  Tomas Olovsson,et al.  A Quantitative Model of the Security Intrusion Process Based on Attacker Behavior , 1997, IEEE Trans. Software Eng..

[37]  NetBIOS Working Group.  Protocol standard for a NetBIOS service on a TCP/UDP transport: Detailed specifications , 1987, RFC.

[38]  George Eckel.  Inside Windows NT , 1993 .

[39]  Abraham Silberschatz,et al.  Operating System Concepts, 5th Edition , 1994 .

[40]  Tomas Olovsson,et al.  An analysis of a secure system based on trusted components , 1996, Proceedings of 11th Annual Conference on Computer Assurance. COMPASS '96.

[41]  Dan Walsh,et al.  Design and implementation of the Sun network filesystem , 1985, USENIX Conference Proceedings.

[42]  Peter D. Goldis.  Questions and Answers about Tiger Teams , 1989 .

[43]  Ulf Lindqvist,et al.  Detecting computer and network misuse through the production-based expert system toolset (P-BEST) , 1999, Proceedings of the 1999 IEEE Symposium on Security and Privacy (Cat. No.99CB36344).

[44]  Richard R. Linde,et al.  Operating system penetration , 1975, AFIPS '75.

[45]  Jeffrey I. Schiller,et al.  An Authentication Service for Open Network Systems. In , 1998 .

[46]  Kai Rannenberg.  Die Trusted Computer System Evaluation Criteria (TCSEC) , 1998 .

[47]  Peter J. Denning,et al.  The internet worm , 1991 .

[48]  Simson L. Garfinkel,et al.  Practical UNIX and Internet Security , 1996 .

[49]  Simson L. Garfinkel,et al.  Practical UNIX and internet security (2. ed.) , 1996 .

[50]  Tomas Olovsson,et al.  Security Evaluation of a PC Network based on Intrusion Experiments , 1996, CCS 1996.

[51]  Ken Thompson,et al.  Password security: a case history , 1979, CACM.

[52]  Brian Randell,et al.  A Distributed Secure System , 1983, 1983 IEEE Symposium on Security and Privacy.

[53]  S. Jajodia,et al.  Information Security: An Integrated Collection of Essays , 1994 .

[54]  Dan Farmer,et al.  Improving the Security of Your Site by Breaking Into it , 2000 .

[55]  Helen Custer,et al.  Inside the Windows NT File System , 1994 .

[56]  I. S. Herschberg.  Make the tigers hunt for you , 1988, Comput. Secur..

[57]  A. Avizienis,et al.  Fault-tolerance: The survival attribute of digital systems , 1978, Proceedings of the IEEE.

[58]  Catherine A. Meadows,et al.  Applying the dependability paradigm to computer security , 1995, Proceedings of 1995 New Security Paradigms Workshop.

[59]  Tomas Olovsson,et al.  An empirical model of the security intrusion process , 1996, Proceedings of 11th Annual Conference on Computer Assurance. COMPASS '96.

[60]  Thomas Anderson,et al.  Fault Tolerance Terminology Proposals , 1985 .

[61]  Erland Jonsson,et al.  A taxonomy and overview of information security experiments , 1997, SEC.

[62]  Paul Leach,et al.  CIFS Logon and Pass Through Authentication Preliminary Draft , 1997 .

[63]  Erland Jonsson,et al.  A practical dependability measure for degradable computer systems with non-exponential degradation , 1994 .

[64]  Bruce Schneier,et al.  Cryptographic Design Vulnerabilities , 1998, Computer.

[65]  Stephen A. Sutton,et al.  Windows Nt Security Guide , 1996 .

[66]  Jon A. Rochlis,et al.  With microscope and tweezers: the worm from MIT's perspective , 1989, Commun. ACM.

[67]  Erland Jonsson,et al.  A Map of Security Risks Associated wuth Using COTS , 1998, Computer.

[68]  Carl E. Landwehr,et al.  A Taxonomy of Computer Program Security Flaws, with Examples , 1993 .

[69]  Samuel J. Leffler,et al.  The design and implementation of the 4.3 BSD Unix operating system , 1991, Addison-Wesley series in computer science.

[70]  Ian Sommerville,et al.  Software engineering, 4th Edition , 1992, International computer science series.