Nonlinear instabilities of a non-self-similar boundary layer on an airfoil: Experiments, DNS, and theory