Adjusting treatment effect estimates by post‐stratification in randomized experiments

Summary.  Experimenters often use post‐stratification to adjust estimates. Post‐stratification is akin to blocking, except that the number of treated units in each stratum is a random variable because stratification occurs after treatment assignment. We analyse both post‐stratification and blocking under the Neyman–Rubin model and compare the efficiency of these designs. We derive the variances for a post‐stratified estimator and a simple difference‐in‐means estimator under different randomization schemes. Post‐stratification is nearly as efficient as blocking: the difference in their variances is of the order of 1/n2, with a constant depending on treatment proportion. Post‐stratification is therefore a reasonable alternative to blocking when blocking is not feasible. However, in finite samples, post‐stratification can increase variance if the number of strata is large and the strata are poorly chosen. To examine why the estimators’ variances are different, we extend our results by conditioning on the observed number of treated units in each stratum. Conditioning also provides more accurate variance estimates because it takes into account how close (or far) a realized random sample is from a comparable blocked experiment. We then show that the practical substance of our results remains under an infinite population sampling model. Finally, we provide an analysis of an actual experiment to illustrate our analytical results.

[1]  Student,et al.  On Testing Varieties of Cereals , 1923 .

[2]  J. Neyman,et al.  Statistical Problems in Agricultural Experimentation , 1935 .

[3]  M. B. Wilk,et al.  THE RANDOMZATION ANALYSIS OF A GENERALIZED RANDOMIZED BLOCK DESIGN , 1955 .

[4]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[5]  References , 1971 .

[6]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[7]  R. Serfling.  Probability Inequalities for the Sum in Sampling without Replacement , 1974 .

[8]  G. Koch,et al.  A review of some statistical methods for covariance analysis of categorical data. , 1982, Biometrics.

[9]  R. McHugh,et al.  Post-stratification in the randomized clinical trial. , 1983, Biometrics.

[10]  P. Holland.  Statistics and Causal Inference , 1985 .

[11]  P. Meier,et al.  Choosing covariates in the analysis of clinical trials. , 1989, Controlled clinical trials.

[12]  S J Senn,et al.  Covariate imbalance and random allocation in clinical trials. , 1989, Statistics in medicine.

[13]  Carl-Erik Särndal,et al.  The weighted residual technique for estimating the variance of the general regression estimator of the finite population total , 1989 .

[14]  T. Speed,et al.  On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9 , 1990 .

[15]  Rory A. Fisher,et al.  The Arrangement of Field Experiments , 1992 .

[16]  William A. Knaus,et al.  The effectiveness of right heart catheterization in the initial care of critically ill patients. SUPPORT Investigators. , 1996, Journal of the American Medical Association (JAMA).

[17]  G G Koch,et al.  Issues for covariance analysis of dichotomous and ordered categorical data from randomized clinical trials and non-parametric strategies for addressing them. , 1998, Statistics in medicine.

[18]  Charles S. Reichardt,et al.  Justifying the use and increasing the power of a t test for a randomized experiment with a convenience sample. , 1999 .

[19]  J. Dalen.  The pulmonary artery catheter-friend, foe, or accomplice? , 2001, JAMA.

[20]  S. Pocock,et al.  Subgroup analysis, covariate adjustment and baseline comparisons in clinical trial reporting: current practiceand problems , 2002, Statistics in medicine.

[21]  Rolf Sundberg,et al.  Conditional statistical inference and quantification of relevance , 2003 .

[22]  James A. Russell,et al.  FEATURE ARTICLES , 2004, Critical care medicine.

[23]  M. Singer,et al.  Assessment of the clinical effectiveness of pulmonary artery catheters in management of patients in intensive care ( PAC-Man ) : a randomised controlled trial , 2022 .

[24]  Didier Payen,et al.  Use of the pulmonary artery catheter is not associated with worse outcome in the ICU. , 2005, Chest.

[25]  Guido W. Imbens,et al.  Estimation of the Conditional Variance in Paired Experiments , 2008 .

[26]  D. Freedman.  Statistical Models and Causal Inference: On Regression Adjustments in Experiments with Several Treatments , 2008, 0803.3757.

[27]  Kosuke Imai,et al.  (www.interscience.wiley.com) DOI: 10.1002/sim.3337 Variance identification and efficiency analysis in randomized experiments under the matched-pair design , 2022 .

[28]  Adjusting Experimental Data , 2008 .

[29]  M. Davidian,et al.  Covariate adjustment for two‐sample treatment comparisons in randomized clinical trials: A principled yet flexible approach , 2008, Statistics in medicine.

[30]  David A. Freedman,et al.  On regression adjustments to experimental data , 2008, Adv. Appl. Math..

[31]  Gary King,et al.  Misunderstandings between experimentalists and observationalists about causal inference , 2008 .

[32]  Gary King,et al.  The essential role of pair matching in cluster-randomized experiments, with application to the Mexican Universal Health Insurance Evaluation , 2009, 0910.3752.

[33]  Jasjeet S. Sekhon,et al.  Opiates for the Matches: Matching Methods for Causal Inference , 2009 .

[34]  R. Perkin,et al.  Pulmonary artery catheters , 2011, Pediatric critical care medicine : a journal of the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies.

[35]  W. Lin,et al.  Agnostic notes on regression adjustments to experimental data: Reexamining Freedman's critique , 2012, 1208.2301.

[36]  J. Sekhon,et al.  A matching method for improving covariate balance in cost-effectiveness analyses. , 2012, Health economics.

[37]  J. Sekhon,et al.  From SATE to PATT : Combining Experimental with Observational Studies to Estimate Population Treatment Effects ∗ , 2013 .