Characterizing and quantifying the incompatibility of quantum instruments