The internal clock of many-body (de-)localization