Experimental and numerical study on multi-wave modes of H2/O2 rotating detonation combustor