Land surface temperature (LST) is a critical parameter of surface energy fluxes and has become the focus of numerous studies. LST downscaling is an effective technique for supplementing the limitations of the coarse-resolution LST data. However, the relationship between LST and other land surface parameters tends to be nonlinear and spatially nonstationary, due to spatial heterogeneity. Nonlinearity and spatial nonstationarity have not been considered simultaneously in previous studies. To address this issue, we propose a multi-factor geographically weighted machine learning (MFGWML) algorithm. MFGWML utilizes three excellent machine learning (ML) algorithms, namely extreme gradient boosting (XGBoost), multivariate adaptive regression splines (MARS), and Bayesian ridge regression (BRR), as base learners to capture the nonlinear relationships. MFGWML uses geographically weighted regression (GWR), which allows for spatial nonstationarity, to fuse the three base learners’ predictions. This paper downscales the 30 m LST data retrieved from Landsat 8 images to 10 m LST data mainly based on Sentinel-2A images. The results show that MFGWML outperforms two classic algorithms, namely thermal image sharpening (TsHARP) and the high-resolution urban thermal sharpener (HUTS). We conclude that MFGWML combines the advantages of multiple regression, ML, and GWR, to capture the local heterogeneity and obtain reliable and robust downscaled LST data.