Spectral broadening and ultrafast dynamics of a nitrogen-vacancy center ensemble in diamond