Fundamental factors in the performance of diffusive samplers