Black hole remnants are not too fast to be dark matter