Why bigger is not always better: on finite and infinite neural networks