A near-optimal algorithm for estimating the entropy of a stream

We describe a simple algorithm for approximating the empirical entropy of a stream of <i>m</i> values up to a multiplicative factor of (1+ε) using a single pass, <i>O</i>(ε⁻² log (Δ⁻¹) log <i>m</i>) words of space, and <i>O</i>(log ε⁻¹ + log log Δ⁻¹ + log log <i>m</i>) processing time per item in the stream. Our algorithm is based upon a novel extension of a method introduced by Alon et al. [1999]. This improves over previous work on this problem. We show a space lower bound of Ω(ε⁻²/log² (ε⁻¹)), demonstrating that our algorithm is near-optimal in terms of its dependency on ε. We show that generalizing to multiplicative-approximation of the <i>k</i>th-order entropy requires close to linear space for <i>k</i>≥1. In contrast we show that additive-approximation is possible in a single pass using only poly-logarithmic space. Lastly, we show how to compute a multiplicative approximation to the entropy of a random walk on an undirected graph.