Nonlinear nanomechanical resonators approaching the quantum ground state