Statistical Difference Beyond the Polarizing Regime

The polarization lemma for statistical distance (\({\text {SD}}\)), due to Sahai and Vadhan (JACM, 2003), is an efficient transformation taking as input a pair of circuits \((C_0,C_1)\) and an integer k and outputting a new pair of circuits \((D_0,D_1)\) such that if \({\text {SD}}(C_0,C_1) \ge \alpha \) then \({\text {SD}}(D_0,D_1) \ge 1-2^{-k}\) and if \({\text {SD}}(C_0,C_1) \le \beta \) then \({\text {SD}}(D_0,D_1) \le 2^{-k}\). The polarization lemma is known to hold for any constant values \(\beta < \alpha ^2\), but extending the lemma to the regime in which \(\alpha ^2 \le \beta < \alpha \) has remained elusive. The focus of this work is in studying the latter regime of parameters. Our main results are: 1. Polarization lemmas for different notions of distance, such as Triangular Discrimination (\({{\,\mathrm{TD}\,}}\)) and Jensen-Shannon Divergence (\({{\,\mathrm{JS}\,}}\)), which enable polarization for some problems where the statistical distance satisfies \( \alpha ^2< \beta < \alpha \). We also derive a polarization lemma for statistical distance with any inverse-polynomially small gap between \( \alpha ^2 \) and \( \beta \) (rather than a constant). 2. The average-case hardness of the statistical difference problem (i.e., determining whether the statistical distance between two given circuits is at least \(\alpha \) or at most \(\beta \)), for any values of \(\beta < \alpha \), implies the existence of one-way functions. Such a result was previously only known for \(\beta < \alpha ^2\). 3. A (direct) constant-round interactive proof for estimating the statistical distance between any two distributions (up to any inverse polynomial error) given circuits that generate them. Proofs of closely related statements have appeared in the literature but we give a new proof which we find to be cleaner and more direct.