Failure Order: A Missing Piece in Disk Failure Processing of Data Centers

To avoid data loss, data centers adopt disk failure prediction (DFP) technology to raise warnings ahead of actual disk failures, and process the warnings in the order they are raised, i.e., a first-in-first-out (FIFO) warning order. The FIFO-guided warning order can process warnings timely when disk failures are rare in data centers. With the growing scale of data centers, the increasing number of disk failures leads to a complex situation that multiple warnings are raised simultaneously, where the FIFO-guided warning order neither processes warnings timely, nor manages warnings properly due to lack of the priority of warnings. Thus, a real-time and finer-grained priority guidance for warning order management is an urgent need. To this end, we turn our attention to the failures since each warning corresponds to a fail event. The key insight is that the interdependence of failures, i.e., the order failure occurred, indicates the order of warning processing. With an accurate failure order, data centers can decrease the probability of data loss and the downtime of latency-sensitive applications by processing urgent warnings in advance. In this paper, we predict the failure order with a LambdaMART model, which is a state-of-the-art ranking algorithm in information retrieval. To avoid overly concerning on the correctness of high-rank warnings in information retrieval, we design a symmetric metric to evaluate the prediction evaluation of failure order. Experiment on a public dataset, provided by the Backblaze company, shows that our model outperforms the FIFO order and the order from previous DFP models.