Efficient Network Measurements through Approximated Windows

Many networking applications require timely access to recent network measurements, which can be captured using a sliding window model. Maintaining such measurements is a challenging task due to the fast line speed and scarcity of fast memory in routers. In this work, we study the efficiency factor that can be gained by approximating the window size. That is, we allow the algorithm to dynamically adjust the window size between W and W (1 + τ) where τ is a small positive parameter. For example, consider the basic summing problem of computing the sum of the last W elements in a stream whose items are integers in {0, 1 . . . , R}, where R = poly(W ). While it is known that Ω(W logR) bits are needed in the exact window model, we show that approximate windows allow an exponential space reduction for constant τ . Specifically, we present a lower bound of Ω(τ−1 log(RWτ)) bits for the basic summing problem. Further, an (1 + ) multiplicative approximation of this problem requires Ω(log (W/ )+log logR) bits for constant τ . Additionally, for RW additive approximations, we show an Ω(τ−1 log b1 + τ/ c+ log (W/ )) lower bound . For all three settings, we provide memory optimal algorithms that operate in constant time. Finally, we demonstrate the generality of the approximated window model by applying it to counting the number of distinct flows in a sliding window over a network stream. We present an algorithm that solves this problem while requiring asymptotically less space than previous sliding window methods when τ = O(1).

[1]  Frédéric Giroire,et al.  Order statistics and estimating cardinalities of massive data sets , 2009, Discret. Appl. Math..

[2]  Ming Zhang,et al.  MicroTE: fine grained traffic engineering for data centers , 2011, CoNEXT '11.

[3]  Luca Trevisan,et al.  Counting Distinct Elements in a Data Stream , 2002, RANDOM.

[4]  Roy Friedman,et al.  Efficient Summing over Sliding Windows , 2016, ArXiv.

[5]  Moni Naor,et al.  Sliding Bloom Filters , 2013, ISAAC.

[6]  Yang Li,et al.  CASE: Cache-assisted stretchable estimator for high speed per-flow measurement , 2016, IEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer Communications.

[7]  Frédéric Giroire,et al.  Estimating the Number of Active Flows in a Data Stream over a Sliding Window , 2007, ANALCO.

[8]  P. Chassaing,et al.  Efficient estimation of the cardinality of large data sets , 2007, math/0701347.

[9]  Georges Hébrail,et al.  Sliding HyperLogLog: Estimating Cardinality in a Data Stream over a Sliding Window , 2010, 2010 IEEE International Conference on Data Mining Workshops.

[10]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[11]  Rong Pan,et al.  AF-QCN: Approximate Fairness with Quantized Congestion Notification for Multi-tenanted Data Centers , 2010, 2010 18th IEEE Symposium on High Performance Interconnects.

[12]  Hao Wang,et al.  DRAM-Based Statistics Counter Array Architecture With Performance Guarantee , 2012, IEEE/ACM Transactions on Networking.

[13]  Noga Alon,et al.  The Space Complexity of Approximating the Frequency Moments , 1999 .

[14]  Gabriel Maciá-Fernández,et al.  Anomaly-based network intrusion detection: Techniques, systems and challenges , 2009, Comput. Secur..

[15]  Gil Einziger,et al.  Independent counter estimation buckets , 2015, 2015 IEEE Conference on Computer Communications (INFOCOM).

[16]  Nan Hua,et al.  BRICK: A Novel Exact Active Statistics Counter Architecture , 2011, IEEE/ACM Transactions on Networking.

[17]  Piotr Indyk,et al.  Maintaining Stream Statistics over Sliding Windows , 2002, SIAM J. Comput..

[18]  Roy Friedman,et al.  Heavy hitters in streams and sliding windows , 2016, IEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer Communications.

[19]  Iddo Hanniel,et al.  Estimators also need shared values to grow together , 2012, 2012 Proceedings IEEE INFOCOM.

[20]  George Varghese,et al.  Bitmap algorithms for counting active flows on high speed links , 2003, IMC '03.

[21]  Roy Friedman,et al.  Counting With Tinytable: Every Bit Counts! , 2019, IEEE Access.

[22]  Philippe Flajolet,et al.  Probabilistic Counting Algorithms for Data Base Applications , 1985, J. Comput. Syst. Sci..

[23]  Roy Friedman,et al.  Optimal elephant flow detection , 2017, IEEE INFOCOM 2017 - IEEE Conference on Computer Communications.

[24]  Roy Friedman,et al.  Randomized admission policy for efficient top-k and frequency estimation , 2016, IEEE INFOCOM 2017 - IEEE Conference on Computer Communications.

[25]  Roy Friedman,et al.  TinyLFU: A Highly Efficient Cache Admission Policy , 2014, 2014 22nd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing.

[26]  Todd L. Heberlein,et al.  Network intrusion detection , 1994, IEEE Network.

[27]  P. Flajolet,et al.  Loglog counting of large cardinalities , 2003 .

[28]  George Varghese,et al.  An Improved Construction for Counting Bloom Filters , 2006, ESA.

[29]  Alexander Hall,et al.  HyperLogLog in practice: algorithmic engineering of a state of the art cardinality estimation algorithm , 2013, EDBT '13.

[30]  Yong Guan,et al.  Near-optimal approximate membership query over time-decaying windows , 2013, 2013 Proceedings IEEE INFOCOM.

[31]  Raja Chiky,et al.  How can sliding HyperLogLog and EWMA detect port scan attacks in IP traffic? , 2014, EURASIP J. Inf. Secur..

[32]  P. Flajolet,et al.  HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm , 2007 .

[33]  Min Chen,et al.  Counter Tree: A Scalable Counter Architecture for Per-Flow Traffic Measurement , 2015, 2015 IEEE 23rd International Conference on Network Protocols (ICNP).

[34]  R. Srikant,et al.  The Power of Slightly More than One Sample in Randomized Load Balancing , 2017, Math. Oper. Res..

[35]  George Varghese,et al.  CONGA: distributed congestion-aware load balancing for datacenters , 2015, SIGCOMM.