The phase diagram of approximation rates for deep neural networks

We explore the phase diagram of approximation rates for deep neural networks. The phase diagram describes theoretically optimal accuracy-complexity relations and their qualitative properties. Our contribution is three-fold. First, we generalize the existing result on the existence of deep discontinuous phase in ReLU networks to functional classes of arbitrary positive smoothness, and identify the boundary between the feasible and infeasible rates. Second, we demonstrate that standard fully-connected architectures of a fixed width independent of smoothness can adapt to smoothness and achieve almost optimal rates. Finally, we discuss how the phase diagram can change in the case of non-ReLU activation functions. In particular, we prove that using both sine and ReLU activations theoretically leads to very fast, nearly exponential approximation rates, thanks to the emerging capability of the network to implement efficient lookup operations.