During horizontal directional drilling, the instability of borehole wall may lead to serious accidents such as too large torque, failure pullback of pipeline, abandonment of bore, etc. Based on the theory of elasticity mechanics and some assumptions, the mechanical model taking into account of initial ground stress and mud pressure is established. Under the practical conditions, cross-section of borehole is noncircular that is mainly caused by the subsidence of reamer. The real cross-section can be equivalent to a series of circles through curve fitting. According to superposition principle and the conversion relationship from polar coordinate system to rectangular coordinate system, the stress formula of noncircular cross-section is given. And then the finite element method is used to calculate and analyze the stress of borehole in the specific engineering. The results show the distribution and variation tendency of stress which can provide some beneficial reference for borehole wall stability study of horizontal directional drilling.