Anomaly Detection in Dynamic Social Systems Using Weak Estimators

Anomaly detection involves identifying observationsthat deviate from the normal behavior of a system. One ofthe ways to achieve this is by identifying the phenomena thatcharacterize “normal” observations. Subsequently, based on thecharacteristics of data learned from the “normal” observations,new observations are classified as being either “normal” or not.Most state-of-the-art approaches, especially those which belongto the family parameterized statistical schemes, work under theassumption that the underlying distributions of the observationsare stationary. That is, they assume that the distributions thatare learned during the training (or learning) phase, thoughunknown, are not time-varying. They further assume that thesame distributions are relevant even as new observations areencountered. Although such a “stationarity” assumption is relevantfor many applications, there are some anomaly detectionproblems where stationarity cannot be assumed. For example, innetwork monitoring, the patterns which are learned to representnormal behavior may change over time due to several factorssuch as network infrastructure expansion, new services, growthof user population, etc. Similarly, in meteorology, identifyinganomalous temperature patterns involves taking into accountseasonal changes of normal observations. Detecting anomaliesor outliers under these circumstances introduces several challenges.Indeed, the ability to adapt to changes in non-stationaryenvironments is necessary so that anomalous observations canbe identified even with changes in what would otherwise beclassified as “normal” behavior. In this paper, we proposed toapply a family of weak estimators for anomaly detection indynamic environments. In particular, we apply this theory tospam email detection. Our experimental results demonstrate thatour proposal is both feasible and effective for the detection ofsuch anomalous emails.