ProtoMon: embedded monitors for cryptographic protocol intrusion detection and prevention

Intrusion detection systems (IDS) are responsible for monitoring and analyzing host or network activity to detect intrusions in order to protect information from unauthorized access or manipulation. There are two main approaches for intrusion detection: signature-based and anomaly-based. Signature-based detection employs pattern matching to match attack signatures with observed data making it ideal for detecting known attacks. However, it cannot detect unknown attacks for which there is no signature available. Anomaly-based detection uses machine-learning techniques to create a profile of normal system behavior and uses this profile to detect deviations from the normal behavior. Although this technique is effective in detecting unknown attacks, it has a drawback of a high false alarm rate. In this paper, we describe our anomaly-based IDS designed for detecting malicious use of cryptographic and application-level protocols. Our system has several unique characteristics and benefits, such as the ability to monitor cryptographic protocols and application-level protocols embedded in encrypted sessions, a very lightweight monitoring process, and the ability to react to protocol misuse by modifying protocol response directly.

[1]  J. Doug Tygar,et al.  A Model for Secure Protocols and Their Compositions , 1996, IEEE Trans. Software Eng..

[2]  Jonathan K. Millen,et al.  The Interrogator: Protocol Secuity Analysis , 1987, IEEE Transactions on Software Engineering.

[3]  Danny Dolev,et al.  On the security of public key protocols , 1981, 22nd Annual Symposium on Foundations of Computer Science (sfcs 1981).

[4]  Alec Yasinsac.  Dynamic analysis of security protocols , 2001, NSPW '00.

[5]  Stephanie Forrest,et al.  A sense of self for Unix processes , 1996, Proceedings 1996 IEEE Symposium on Security and Privacy.

[6]  Paul Douglas,et al.  Proceedings International Conference on Information Technology: Coding and Computing , 2002, Proceedings. International Conference on Information Technology: Coding and Computing.

[7]  Daniel Bleichenbacher,et al.  Chosen Ciphertext Attacks Against Protocols Based on the RSA Encryption Standard PKCS #1 , 1998, CRYPTO.

[8]  Vlastimil Klíma,et al.  Attacking RSA-Based Sessions in SSL/TLS , 2003, CHES.

[9]  Catherine A. Meadows,et al.  Applying Formal Methods to the Analysis of a Key Management Protocol , 1992, J. Comput. Secur..

[10]  John C. Mitchell,et al.  Automated analysis of cryptographic protocols using Mur/spl phi/ , 1997, Proceedings. 1997 IEEE Symposium on Security and Privacy (Cat. No.97CB36097).

[11]  R. Sekar,et al.  Specification-based anomaly detection: a new approach for detecting network intrusions , 2002, CCS '02.

[12]  Philip K. Chan,et al.  Identifying outliers via clustering for anomaly detection , 2003 .

[13]  Stephanie Forrest,et al.  Automated response using system-call delays , 2000 .

[14]  Salvatore J. Stolfo,et al.  Data Mining Approaches for Intrusion Detection , 1998, USENIX Security Symposium.

[15]  John Ulrich,et al.  Automated Analysis of Cryptographic Protocols Using Mur ' , 1997 .

[16]  Roger M. Needham,et al.  Using encryption for authentication in large networks of computers , 1978, CACM.

[17]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[18]  Alec Yasinsac.  An Environment for Security Protocol Intrusion Detection , 2002, J. Comput. Secur..

[19]  Paul Syverson,et al.  A Taxonomy of Replay Attacks , 1994 .

[20]  Jim Alves-Foss,et al.  NATE: Network Analysis of Anomalous Traffic Events, a low-cost approach , 2001, NSPW '01.

[21]  Stuart Staniford-Chen,et al.  Practical Automated Detection of Stealthy Portscans , 2002, J. Comput. Secur..

[22]  Bruce Schneier,et al.  Analysis of the SSL 3.0 protocol , 1996 .

[23]  Gavin Lowe,et al.  Breaking and Fixing the Needham-Schroeder Public-Key Protocol Using FDR , 1996, Softw. Concepts Tools.

[24]  John C. Mitchell,et al.  A meta-notation for protocol analysis , 1999, Proceedings of the 12th IEEE Computer Security Foundations Workshop.

[25]  Stephen R. Tate,et al.  ProtoMon: Embedded Monitors for Cryptographic Protocol Intrusion Detection and Prevention , 2004, ITCC.

[26]  David Brumley,et al.  Remote timing attacks are practical , 2003, Comput. Networks.

[27]  Serge Vaudenay,et al.  Password Interception in a SSL/TLS Channel , 2003, CRYPTO.