The unreasonable effectiveness of neural network approximation

Results concerning the approximation rates of neural networks are of particular interest to engineers. The results reported in the literature have “slow approximation rates” (of the order of 1/√m, where m is the number of parameters in the neural network). However many empirical studies report that neural network approximation is quite effective in practice. Here we give an explanation of this unreasonable effectiveness by proving the existence of a sequence of approximations that converge at a faster rate by using methods from number theory