Detecting Model Dependence in Statistical Inference: A Response

When counterfactual questions are posed too far from the data, statistical inference can become highly dependent on seemingly trivial but indefensible modeling assumptions. Choosing one or only a few specifications to publish in the presence of model dependence makes conclusions largely nonempirical and subject to criticism from others who make different choices and find contradictory results. Until now, only relatively narrow attempts to assess model dependence have been used in the literature (such as trying a few different functional form specifications) and, as a result, many published inferences are far more uncertain than standard errors and confidence intervals indicate. When researchers ignore the uncertainty due to model dependence, scholarly works tend to have the flavor of merely showing that it is possible to find results consistent with ex ante hypotheses, rather than demonstrating their veracity. Consequently, the opportunities for researchers to use methods such as the ones we offer to learn new facts about existing studies and avoid problems in their research are considerable. Although the comments contain a diversity of views, we are gratified to see that all three endorse our central message. Morrow: “It is important to know when counterfactual statements drawn from statistical estimates wander far from the data used in estimates.” Schrodt: “I am not contesting the general cautions made by the authors: models with different functional forms can diverge substantially, predictions made for sets of independent variables similar in value to those used to estimate a model are more likely to be accurate than predictions for more distant values, and specification error can really mess up regression models.” Sambanis and Doyle (S&D): “statistical results should not be taken too far [from the data] and … any extrapolation depends on the model.” On some other issues, the reviewers disagree with each other or with us. Fortunately, …