Synthetic aperture radar (SAR) as a wideband radar system is subject to complicated interferences, such as radio frequency interference or other narrowband interferences (NBIs). In order to suppress the NBI, voluminous literature focused on its signal models and characteristics, such as the sinusoidal model and relatively constant frequencies. However, in practice, the interference environment is commonly complicated. It is hard to model the interferences accurately and mitigate them clearly in an easy way, especially for the time-varying interferences. In this article, a novel graph-based algorithm is proposed to mitigate the time-varying NBIs by using graph theory, which constructs the connections between different azimuth samples of NBIs. As a result, the locally time-varying interferences can be clustered in a nonlinear low-dimensional manifold and effectively removed by the proposed algorithm. In addition, the case of the globally time-varying interference is also analyzed in detail with strict derivations to demonstrate its low-rank property. Furthermore, the matrix factorization scheme is introduced to improve the efficiency of the proposed algorithm, and the closed-form solutions are derived for each iteration. The real SAR data with measured NBIs are provided to demonstrate the effectiveness and efficiency of the proposed algorithm.