Relative self-adjoint operators in Hilbert space