Spectral theory for a singular linear system of Hamiltonian differential equations