Anomalous and anisotropic nonlinear susceptibility in the proximate Kitaev magnet α-RuCl3