Vegetation height estimation plays a pivotal role in forest mapping, which significantly promotes the study of environment and climate. This paper develops a general forest structure model for vegetation height estimation using polarimetric interferometric synthetic aperture radar (PolInSAR) data. In simple terms, the temporal decorrelation factor of the random volume over ground model with volumetric temporal decorrelation (RVoG-vtd) is first modeled by random motions of forest scatterers to solve the problem of ambiguity. Then, a novel four-stage algorithm is proposed to improve accuracy in forest height estimation. In particular, to compensate for the temporal decorrelation mainly caused by changes between multiple observations, one procedure of temporal decorrelation adaptive estimation via Expectation-Maximum (EM) algorithm is added into the novel method. On the other hand, to extract the features of amplitude and phase more effectively, in the proposed method, we also convert Euclidean distance to a generalized distance for the first time. Assessments of different algorithms are given based on the repeat-pass PolInSAR data of Gabon Lope Park acquired in AfriSAR campaign of German Aerospace Center (DLR). The experimental results show that the proposed method presents a significant improvement of vegetation height estimation accuracy with a root mean square error (RMSE) of 6.23 m and a bias of 1.28 m against LiDAR heights, compared to the results of the three-stage method (RMSE: 8.69 m, bias: 4.81 m) and the previous four-stage method (RMSE: 7.72 m, bias: −2.87 m).