The Impact of JDM on IT Infrastructure

This chapter explores the impact that data mining can have on the information technology (IT) infrastructure and what data and process administrators need to consider when mining in the large. Data mining can have a great impact on the infrastructure of IT. Mining in the small, that is, using small local datasets and maintaining a few models for short-term use, is unlikely to tax the IT environment. However, mining in the large—with multi-gigabyte and sometimes multi-terabyte datasets, hundreds or thousands of frequently changing datasets, obtaining datasets across the enterprise or beyond, and managing hundreds or thousands of models for deployment throughout the enterprise—places new demands for backup and recovery, data access, data staging, and ensuring proper levels of service. Moreover, data mining is often part of a larger business process requiring plugging into existing scheduling or workflow systems. There are also differences in how IT approaches database and non-database data mining engines (DMEs).