Relativistic and nonrelativistic modulational instability of a laser radiation in an unmagnetized laser-produced plasma: Kinetic theory