Towards Testing Monotonicity of Distributions Over General Posets

In this work, we consider the sample complexity required for testing the monotonicity of distributions over partial orders. A distribution $p$ over a poset is monotone if, for any pair of domain elements $x$ and $y$ such that $x \preceq y$, $p(x) \leq p(y)$. To understand the sample complexity of this problem, we introduce a new property called bigness over a finite domain, where the distribution is $T$-big if the minimum probability for any domain element is at least $T$. We establish a lower bound of $\Omega(n/\log n)$ for testing bigness of distributions on domains of size $n$. We then build on these lower bounds to give $\Omega(n/\log{n})$ lower bounds for testing monotonicity over a matching poset of size $n$ and significantly improved lower bounds over the hypercube poset. We give sublinear sample complexity bounds for testing bigness and for testing monotonicity over the matching poset. We then give a number of tools for analyzing upper bounds on the sample complexity of the monotonicity testing problem.

[1]  Dana Ron,et al.  Improved Testing Algorithms for Monotonicity , 1999, Electron. Colloquium Comput. Complex..

[2]  Alon Orlitsky,et al.  A Competitive Test for Uniformity of Monotone Distributions , 2013, AISTATS.

[3]  Yihong Wu,et al.  Minimax Rates of Entropy Estimation on Large Alphabets via Best Polynomial Approximation , 2014, IEEE Transactions on Information Theory.

[4]  Clément L. Canonne Big Data on the Rise? - Testing Monotonicity of Distributions , 2015, ICALP.

[5]  Ronitt Rubinfeld,et al.  Monotonicity testing over general poset domains , 2002, STOC '02.

[6]  Dana Ron,et al.  Strong Lower Bounds for Approximating Distribution Support Size and the Distinct Elements Problem , 2007, 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07).

[7]  Seshadhri Comandur,et al.  Optimal bounds for monotonicity and lipschitz testing over hypercubes and hypergrids , 2013, STOC '13.

[8]  Ronitt Rubinfeld,et al.  Sublinear algorithms for testing monotone and unimodal distributions , 2004, STOC '04.

[9]  Ronitt Rubinfeld,et al.  Testing monotonicity of distributions over general partial orders , 2011, ICS.

[10]  Seshadhri Comandur,et al.  A o(d) · polylog n Monotonicity Tester for Boolean Functions over the Hypergrid [n]d , 2018, SODA.

[11]  Dana Ron,et al.  On Disjoint Chains of Subsets , 2001, J. Comb. Theory, Ser. A.

[12]  Gregory Valiant,et al.  Instance optimal learning of discrete distributions , 2016, STOC.

[13]  Rocco A. Servedio,et al.  Learning k-Modal Distributions via Testing , 2011, Theory Comput..

[14]  Dana Ron,et al.  Testing Monotonicity , 2000, Comb..

[15]  Rocco A. Servedio,et al.  Testing k-Modal Distributions: Optimal Algorithms via Reductions , 2011, SODA.

[16]  Yihong Wu,et al.  Chebyshev polynomials, moment matching, and optimal estimation of the unseen , 2015, The Annals of Statistics.

[17]  Eric Blais,et al.  A polynomial lower bound for testing monotonicity , 2016, STOC.

[18]  Seshadhri Comandur,et al.  An Optimal Lower Bound for Monotonicity Testing over Hypergrids , 2013, APPROX-RANDOM.

[19]  Ludmil T. Zikatanov,et al.  Polynomial of Best Uniform Approximation to 1/x and Smoothing in Two-level Methods , 2010, Comput. Methods Appl. Math..

[20]  Gregory Valiant,et al.  Estimating the Unseen , 2017, J. ACM.

[21]  Paul Valiant Testing symmetric properties of distributions , 2008, STOC '08.

[22]  Ronitt Rubinfeld,et al.  Testing Shape Restrictions of Discrete Distributions , 2015, Theory of Computing Systems.

[23]  Ronitt Rubinfeld,et al.  The complexity of approximating the entropy , 2002, Proceedings 17th IEEE Annual Conference on Computational Complexity.

[24]  Constantinos Daskalakis,et al.  Optimal Testing for Properties of Distributions , 2015, NIPS.

[25]  Artur Czumaj,et al.  Testing Monotone Continuous Distributions on High-Dimensional Real Cubes , 2010, Property Testing.