Transfer matrices as nonunitary S matrices, multimode unidirectional invisibility, and perturbative inverse scattering