Spectroscopic studies of molecular interactions of oxygen-16 and oxygen-18 in the high-density epsilon-phase