Non-Markovian dynamics in chiral quantum networks with spins and photons