Leveraging the Fisher randomization test using confidence distributions: Inference, combination and fusion learning

The flexibility and wide applicability of the Fisher randomization test (FRT) makes it an attractive tool for assessment of causal effects of interventions from modern-day randomized experiments that are increasing in size and complexity. This paper provides a theoretical inferential framework for FRT by establishing its connection with confidence distributions Such a connection leads to development of (i) an unambiguous procedure for inversion of FRTs to generate confidence intervals with guaranteed coverage, (ii) generic and specific methods to combine FRTs from multiple independent experiments with theoretical guarantees and (iii) new insights on the effect of size of the Monte Carlo sample on the results of FRT. Our developments pertain to finite sample settings but have direct extensions to large samples. Simulations and a case example demonstrate the benefit of these new developments.

[1]  D. Rubin.  Randomization Analysis of Experimental Data: The Fisher Randomization Test Comment , 1980 .

[2]  Avi Feller,et al.  Analyzing Two-Stage Experiments in the Presence of Interference , 2016, 1608.06805.

[3]  Sensitive and Sturdy $p$-Values , 1991 .

[4]  Luke Miratrix,et al.  A Conditional Randomization Test to Account for Covariate Imbalance in Randomized Experiments , 2015, 1510.06817.

[5]  Paul H. Garthwaite,et al.  CONFIDENCE INTERVALS FROM RANDOMIZATION TESTS , 1996 .

[6]  Minge Xie,et al.  Combining information from independent sources through confidence distributions , 2005, math/0504507.

[7]  K. Singh,et al.  Confidence Distribution, the Frequentist Distribution Estimator of a Parameter: A Review , 2013 .

[8]  P. Ding,et al.  General Forms of Finite Population Central Limit Theorems with Applications to Causal Inference , 2016, 1610.04821.

[9]  S. Stigler.  R. A. Fisher in the 21st Century , 2008 .

[10]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[11]  Peter M. Steiner,et al.  Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments , 2008 .

[12]  Peng Ding,et al.  Randomization Tests for Weak Null Hypotheses in Randomized Experiments , 2018, Journal of the American Statistical Association.

[13]  Allan Birnbaum,et al.  Confidence Curves: An Omnibus Technique for Estimation and Testing Statistical Hypotheses , 1961 .

[14]  B. Efron.  Bayes and likelihood calculations from confidence intervals , 1993 .

[15]  G. Imbens,et al.  Exact p-Values for Network Interference , 2015, 1506.02084.

[16]  Bradley Efron,et al.  R. A. Fisher in the 21st century (Invited paper presented at the 1996 R. A. Fisher Lecture) , 1998 .

[17]  P. Ding.  A paradox from randomization-based causal inference , 2014, 1402.0142.

[18]  Tirthankar Dasgupta,et al.  Causal inference from 2K factorial designs by using potential outcomes , 2015 .

[19]  Minge Xie,et al.  Confidence Distributions and a Unifying Framework for Meta-Analysis , 2011 .

[20]  Elias Bareinboim,et al.  Causal inference and the data-fusion problem , 2016, Proceedings of the National Academy of Sciences.

[21]  E. Pitman.  Significance Tests Which May be Applied to Samples from Any Populations , 1937 .

[22]  D. Rubin,et al.  Comments on the Neyman-Fisher Controversy and Its Consequences , 2014, 1409.2675.

[23]  R. A. Fisher,et al.  Design of Experiments , 1936 .

[25]  D. Basu.  Randomization Analysis of Experimental Data: The Fisher Randomization Test , 1980 .

[26]  J. I.  The Design of Experiments , 1936, Nature.

[27]  J. Ioannidis,et al.  Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey , 2016, British Medical Journal.

[28]  E. Suchman,et al.  The American soldier: Adjustment during army life. (Studies in social psychology in World War II), Vol. 1 , 1949 .

[29]  Guillaume W. Basse,et al.  Randomization tests of causal effects under interference , 2019, Biometrika.

[30]  R. Fisher.  Statistical methods for research workers , 1927, Protoplasma.

[31]  D. Rubin.  Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician , 1984 .

[32]  D. Cox.  Some problems connected with statistical inference , 1958 .

[33]  P. Ding,et al.  A randomization-based perspective of analysis of variance: a test statistic robust to treatment effect heterogeneity , 2016, 1608.01787.

[34]  Minge Xie,et al.  A Split-and-Conquer Approach for Analysis of Extraordinarily Large Data , 2014 .

[35]  Luke W. Miratrix,et al.  Beyond the Sharp Null: Randomization Inference, Bounded Null Hypotheses, and Confidence Intervals for Maximum Effects , 2017, 1709.07339.

[36]  Kari Lock Morgan,et al.  Rerandomization to improve covariate balance in experiments , 2012, 1207.5625.

[37]  Tore Schweder,et al.  Frequentist Analogues of Priors and Posteriors , 1999 .

[38]  T. Speed,et al.  On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9 , 1990 .

[39]  N. Hjort,et al.  Confidence and Likelihood * , 2002 .