Observation of the ballooning mode that limits the operation space of the high-density super-dense-core plasma in the LHD