Computationally Efficient Approximations for Matrix-based Renyi's Entropy

The recently developed matrix-based Rényi’s α-order entropy enables measurement of information in data simply using the eigenspectrum of symmetric positive semi-definite (PSD) matrices in reproducing kernel Hilbert space, without estimation of the underlying data distribution. This intriguing property makes the new information measurement widely adopted in multiple statistical inference and learning tasks. However, the computation of such quantity involves the trace operator on a PSD matrix G to power α (i.e., tr(Gα)), with a normal complexity of nearly O(n3), which severely hampers its practical usage when the number of samples (i.e., n) is large. In this work, we present computationally efficient approximations to this new entropy functional that can reduce its complexity to even significantly less than O(n2). To this end, we first develop randomized approximations to tr(Gα) that transform the trace estimation into matrix-vector multiplications problem. We extend such strategy for arbitrary values of α (integer or non-integer). We then establish the connection between the matrix-based Renyi’s entropy and PSD matrix approximation, which enables us to exploit both clustering and block low-rank structure of G to further reduce the computational cost. We theoretically provide approximation accuracy guarantees and illustrate the properties of different approximations. Large-scale experimental evaluations on both synthetic and real-world data corroborate our theoretical findings, showing promising speedup with negligible loss in accuracy.