Toys can’t play: physical agents in Spekkens’ theory