Behavioural correlation for malicious bot detection

Over the past few years, IRC bots, malicious programs which are remotely controlled by the attacker, have become a major threat to the Internet and its users. These bots can be used in different malicious ways such as to launch distributed denial of service (DDoS) attacks to shutdown other networks and services. New bots are implemented with extended features such as keystrokes logging, spamming, traffic sniffing, which cause serious disruption to targeted networks and users. In response to these threats, there is a growing demand for effective techniques to detect the presence of bots/botnets. Currently existing approaches detect botnets rather than individual bots. In our work we present a host-based behavioural approach for detecting bots/botnets based on correlating different activities generated by bots by monitoring function calls within a specified time window. Different correlation algorithms have been used in this work to achieve the required task. We start our work by detecting IRC bots' behaviours using a simple correlation algorithm. A more intelligent approach to understand correlating activities is also used as a major part of this work. Our intelligent algorithm is inspired by the immune system. Although the intelligent approach produces an anomaly value for the classification of processes, it generates false positive alarms if not enough data is provided. In order to solve this problem, we introduce a modified anomaly value which reduces the amount of false positives generated by the original anomaly value. We also extend our work to detect peer to peer (P2P) bots which are the upcoming threat to Internet security due to the fact that P2P bots do not have a centralized point to shutdown or traceback, thus making the detection of P2P bots a real challenge. Our evaluation shows that correlating different activities generated by IRC/P2P bots within a specified time period achieves high detection accuracy. In addition, using an intelligent correlation algorithm not only states if an anomaly is present, but it also names the culprit responsible for the anomaly.