Dimension-Independent Rates of Approximation by Neural Networks

To characterize sets of functions that can be approximated by neural networks of various types with dimension-independent rates of approximation we introduce a new norm called variation with respect to a family of functions. We derive its basic properties and give upper estimates for functions satisfying certain integral equations. For a special case, variation with respect to characteristic functions of half-spaces, we give a characterization in terms of orthogonal flows throught layers corresponding to discretized hyperplanes. As a result we describe sets of functions that can be approximated with dimension-independent rates by sigmoidal perceptron networks.