Relating on-shell and off-shell formalisms in perturbative quantum field theory