The spatio-temporal dynamics of neutral genetic diversity