Interplay of grating and nonlinearity in mode coupling