Using analytical calculations and computer simulations, we consider both the lateral diffusion of a membrane protein and the fluctuation spectrum of the membrane in which the protein is embedded. The membrane protein interacts with the membrane shape through its spontaneous curvature and bending rigidity. The lateral motion of the protein may be viewed as diffusion in an effective potential, hence, the effective mobility is always reduced compared to the case of free diffusion. Using a rigorous path-integral approach, we derive an analytical expression for the effective diffusion coefficient for small ratios of temperature and bending rigidity, which is the biologically relevant limit. Simulations show very good quantitative agreement with our analytical result. The analysis of the correlation functions contributing to the diffusion coefficient shows that the correlations between the stochastic force of the protein and the response in the membrane shape are responsible for the reduction. Our quantitative analysis of the membrane height correlation spectrum shows an influence of the protein-membrane interaction causing a distinctly altered wave-vector dependence compared to a free membrane. Furthermore, the time correlations exhibit the two relevant time scales of the system: that of membrane fluctuations and that of lateral protein diffusion with the latter typically much longer than the former. We argue that the analysis of the long-time decay of membrane height correlations can thus provide a new means to determine the effective diffusion coefficient of proteins in the membrane.