A generalized nonlinear failure criterion for frictional materials