Semantic segmentation is one of the most important tasks in remote sensing image processing. According to task requirements, the semantic depth given to the same remote sensing image can be different, and many people have studied it through a pyramid or multilayer structure. The Markov random field (MRF) is widely used in single-layer modeling due to its outstanding spatial relationship capturing ability and feature description ability, but it is not sufficient enough to mine the interlayer information, and the way of information transmission between layers is relatively simple direct projection segmentation results. To solve this problem, new dual-track information transmission is proposed in this letter. The proposed method first constructs a triple-multi (multiresolution, multiregion adjacency graph (RAG), and multisemantic)-pyramid (TMP) structure with the original resolution image as the middle layer in the pyramid. Then, the MRF model is defined on each layer; its likelihood function and the prior function that are related to the adjacent layer are constructed. Finally, the dual-track information transmission circulation is carried out to traverse the entire pyramid structure starting from the original resolution layer. The proposed method is tested on different remote sensing images obtained by the SPOT5, Gaofen-2, and unmanned aerial vehicle (UAV) sensors. Experimental results show that the proposed method has better segmentation performance than other multilayer MRF methods.