Abstract The measured data and simulation test phenomenon of surrounding rock deformation and failure at the project site indicate that shear failure which firstly occurs in surrounding rock, block slip and second shear failure are the root cause of deformation and damage of supporting structure of the surrounding rock at a large scale. We derived limit load of surrounding rock shear slip failure and reasonable support resistance of given load by means of shear slip line field theory, discussed the main factors which influence the limit load of surrounding rock. Shear slip line field and limit load of circular tunnel surrounding rock were obtained by means of physical simulation test, which agreed well with the theoretical analysis results. Based on the theoretical analysis and physical simulation test, the cause deformation and failure at large scale of Xinshanghai No. 1 coal mine big section ingate was analyzed, and the shear failure resistance and block slip in surrounding rock were proposed as the core technical supporting ideas. Proper range of supporting resistance which came from calculation was suggested. The support scheme which is mainly composed of large grouting anchor, sprayed anchor net support technique and full-face grille concrete finally ended the dilemma of repeated failure and mending of ingate and created critical conditions for smooth production in the coal mine.