Counterfactual reasoning: Do Language Models need world knowledge for causal inference?