Practical Data Mining and Analysis for System Administration

Modern networks are both complex and important, requiring excellent and vigilant system administration. System administrators employ many tools to aid them in their work, but still security vulnerabilities, misconfigurations, and unanticipated device failures occur regularly. The constant and repetitive work put into fixing these problems wastes money, time, and effort. We have developed a system to greatly reduce this waste. By implementing a practical data mining infrastructure, we are able to analyze device data and logs as part of general administrative tasks. This allows us to track security risks and identify configuration problems far more quickly and efficiently than conventional systems could by themselves. This approach gives system administrators much more knowledge about and power over their systems, saving them resources and time. The system is practical because it is more straightforward and easier to deploy than traditional data mining architectures. Generally, data analysis infrastructure is large, expensive, and used for other purposes than system administration. This has often kept administrators from applying the technology to analysis of their networks. But with our system, this problem can be overcome. We propose a lightweight, easily configurable solution that can be set up and maintained by the system administrators themselves, saving work hours and resources in the long run. One advantage to using data mining is that we can exploit behavioral analysis to help answer questions about points of failure, analyze an extremely large number of device logs, and identify device failures before they happen. Indexing the logs and parsing out the information enables system administrators to query and search for specific items, narrowing down points of failure to resolve them faster. Consequently, network and system downtime is decreased. In summary, we have found in our tests that the system decreases security response time significantly. We have also found that the system identifies configuration problems that had gone unnoticed for months or even years, problems that could be causing many other issues within the network. This system’s ability to identify struggling devices by early warning signs before they go down has proven invaluable. We feel that the benefits of this system are great enough to make it worth implementing in most any professional computer network.