The difficult early stages of embedded star clusters and the importance of the pre-gas expulsion virial ratio