The principle of virtual work, counterfactuals, and the avoidance of physics