Radiation cooling instabilities in laser-heated plasma