We performed magnetization, resistivity, and neutron diffraction measurements under uniaxial stress applied along [1-10] direction on the tetragonal magnet CeRh$_2$Si$_2$ with commensurate magnetic orders. CeRh$_2$Si$_2$ has two successive antiferromagnetic (AF) orders in zero magnetic field. The high temperature phase (AF1 phase) has the magnetic modulation wave vector of $q = (\frac{1}{2}, \frac{1}{2}, 0)$, and the low temperature phase (AF2 phase) is characterized by the four $q$-vectors of $q = (\frac{1}{2}, \frac{1}{2}, 0), (\frac{1}{2}, -\frac{1}{2}, 0), (\frac{1}{2}, \frac{1}{2}, \frac{1}{2})$, and $(\frac{1}{2}, -\frac{1}{2}, \frac{1}{2})$. By measuring the uniaxial stress dependence of the magnetization, resistivity and the intensities of magnetic Bragg reflections, we confirmed that the AF1 phase has the single-$q$ magnetic order with two-fold rotational symmetry and the AF2 phase has the multi-$q$ magnetic order with four-fold rotational symmetry. In order to understand the origin of multi-$q$ order of CeRh$_2$Si$_2$, we also performed inelastic neutron scattering measurement on the single crystal samples. We found a magnetic excitation at the transfer energy $\hbar \omega \sim$ 8 meV. By applying the linear spin-wave theory, we found that the nearest and the next-nearest neighbor exchange interactions on the $ab$-plane, $J_1$ and $J_2$, are dominant in the AF2 phase. However, the $J_1$-$J_2$ model cannot lift the degeneracy between the single-$q$ (AF1) and multi-$q$ (AF2) phases. We suggest that it can be lifted by taking into account the biquadratic interaction derived from the perturbative expansion for the Kondo lattice Hamiltonian. [S. Hayami et al., Phys. Rev. B 95, 224424 (2017).