Calibrated Bayes, an alternative inferential paradigm for official statistics

I characterize the prevailing philosophy of official statistics as a design/model compromise (DMC). It is design-based for descriptive inferences from large samples, and model-based for small area estimation, nonsampling errors such as nonresponse or measurement error, and some other subfields like ARIMA modeling of time series. I suggest that DMC involves a form of “inferential schizophrenia”, and offer examples of the problems this creates. An alternative philosophy for survey inference is calibrated Bayes (CB), where inferences for a particular data set are Bayesian, but models are chosen to yield inferences that have good design-based properties. I argue that CB resolves DMC conflicts, and capitalizes on the strengths of both frequentist and Bayesian approaches. Features of the CB approach to surveys include the incorporation of survey design information into the model, and models with weak prior distributions that avoid strong parametric assumptions. I describe two applications to U.S. Census Bureau data.

[1]  D. Pfeffermann The Role of Sampling Weights when Modeling Survey Data , 1993 .

[2]  Joseph Sedransk,et al.  The morris hansen lecture 2007. Assessing the value of Bayesian methods for inference about finite population quantities , 2008 .

[3]  Ying Yuan,et al.  Model‐based estimates of the finite population mean for two‐stage cluster samples with unit non‐response , 2007 .

[4]  R. Little To Model or Not To Model? Competing Modes of Inference for Finite Population Sampling , 2004 .

[5]  Michael R Elliott,et al.  A Bayesian Approach to Census Evaluation using A C E Survey Data and Demographic Analysis , 2003 .

[6]  A. Dawid The Well-Calibrated Bayesian , 1982 .

[7]  Roderick J. A. Little,et al.  Statistical Analysis with Missing Data , 1988 .

[8]  Roderick J. A. Little,et al.  Statistical Modeling Methodology for the Voting Rights Act Section 203 Language Assistance Determinations , 2014 .

[9]  Kosuke Imai,et al.  Survey Sampling , 1998, Nov/Dec 2017.

[10]  T. Postelnicu,et al.  Foundations of inference in survey sampling , 1977 .

[11]  George E. P. Box,et al.  Sampling and Bayes' inference in scientific modelling and robustness , 1980 .

[12]  David Draper,et al.  Assessment and Propagation of Model Uncertainty , 2011 .

[13]  H. W. Peers On Confidence Points and Bayesian Probability Points in the Case of Several Parameters , 1965 .

[14]  D. Rubin Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[15]  W. A. Ericson Subjective Bayesian Models in Sampling Finite Populations , 1969 .

[16]  W. A. Ericson,et al.  9 Bayesian inference in finite populations , 1988 .

[17]  David Draper,et al.  Calibration results for Bayesian model specication , 2010 .

[18]  Risto Lehtonen,et al.  Research and Development in Official Statistics and Scientific Co-operation with Universities: An Empirical Investigation , 2002 .

[19]  Ying Yuan,et al.  Parametric and Semiparametric Model‐Based Estimates of the Finite Population Mean for Two‐Stage Cluster Samples with Item Nonresponse , 2007, Biometrics.

[20]  A. Agresti,et al.  Approximate is Better than “Exact” for Interval Estimation of Binomial Proportions , 1998 .

[21]  Malay Ghosh,et al.  Bayesian Methods for Finite Population Sampling , 1997 .

[22]  R. Royall On finite population sampling theory under certain linear regression models , 1970 .

[23]  J. Heckman The Common Structure of Statistical Models of Truncation, Sample Selection and Limited Dependent Variables and a Simple Estimator for Such Models , 1976 .

[24]  Risto Lehtonen,et al.  Research and Development in Official Statistics and Scientific Co-operation with Universities: A Follow-Up Study , 2009 .

[25]  David Firth,et al.  Robust models in probability sampling , 1998 .

[26]  S. Fienberg Bayesian Models and Methods in Public Policy and Government Settings , 2011, 1108.2177.

[27]  L. Kish,et al.  Inference from Complex Samples , 1974 .

[28]  Leonard J. Savage,et al.  The foundations of statistical inference : a discussion , 1962 .

[29]  K. Brewer A Class of Robust Sampling Designs for Large-Scale Surveys , 1979 .

[30]  F. Breidt,et al.  Model-Assisted Estimation of Forest Resources With Generalized Additive Models , 2007 .

[31]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[32]  J. Robins,et al.  Inference for imputation estimators , 2000 .

[33]  M. Kendall Theoretical Statistics , 1956, Nature.

[34]  D. Rubin Multiple Imputation After 18+ Years , 1996 .

[35]  L. Brown,et al.  Interval Estimation for a Binomial Proportion , 2001 .

[36]  Graham Kalton,et al.  Models in the Practice of Survey Sampling , 1983 .

[37]  D. Holt,et al.  Regression Analysis of Data from Complex Surveys , 1980 .

[38]  Richard Valliant,et al.  Finite population sampling and inference : a prediction approach , 2000 .

[39]  C. T. Isaki,et al.  Survey Design under the Regression Superpopulation Model , 1982 .

[40]  Roderick J. A. Little,et al.  Weighting and Prediction in Sample Surveys , 2008 .

[41]  J. N. K. Rao,et al.  Impact of Frequentist and Bayesian Methods on Survey Sampling Practice: A Selective Appraisal , 2011, 1108.2356.

[42]  R. Little,et al.  Inference for the Population Total from Probability-Proportional-to-Size Samples Based on Predictions from a Penalized Spline Nonparametric Model , 2003 .

[43]  P. Gustafson,et al.  Conservative prior distributions for variance parameters in hierarchical models , 2006 .

[44]  W. DuMouchel,et al.  Using Sample Survey Weights in Multiple Regression Analyses of Stratified Samples , 1983 .

[45]  D. Wasserman,et al.  To weight or not to weight ... that is the question. , 1989, Journal of occupational medicine. : official publication of the Industrial Medical Association.

[46]  Roderick J. A. Little,et al.  Estimating a Finite Population Mean from Unequal Probability Samples , 1983 .

[47]  Roderick Little,et al.  Calibrated Bayes, for Statistics in General, and Missing Data in Particular , 2011, 1108.1917.

[48]  M. H. Hansen An Evaluation of Model-Dependent and Probability-Sampling Inferences in Sample Surveys , 1983 .

[49]  David R. Cox,et al.  The Choice between Alternative Ancillary Statistics , 1971 .

[50]  Luca Trevisan,et al.  To Weight or Not to Weight: Where is the Question? , 1996, ISTCS.

[51]  K. R. W. Brewer,et al.  THE EFFECT OF SAMPLE STRUCTURE ON ANALYTICAL SURVEYS1,2 , 1973 .

[52]  Xiao-Li Meng,et al.  Multiple-Imputation Inferences with Uncongenial Sources of Input , 1994 .

[53]  H. S. Konijn Regression Analysis in Sample Surveys , 1962 .

[54]  Nathaniel Schenker,et al.  Combining information from multiple surveys to enhance estimation of measures of health , 2007, Statistics in medicine.

[55]  R. Sugden,et al.  Ignorable and informative designs in survey sampling inference , 1984 .

[56]  R. Fay,et al.  Estimates of Income for Small Places: An Application of James-Stein Procedures to Census Data , 1979 .

[57]  R. Fay Alternative Paradigms for the Analysis of Imputed Survey Data , 1996 .

[58]  R. Little,et al.  Bayesian penalized spline model-based inference for finite population proportion in unequal probability sampling. , 2010, Survey methodology.

[59]  David A. Binder,et al.  Non-parametric Bayesian Models for Samples from Finite Populations , 1982 .

[60]  B. Efron Why Isn't Everyone a Bayesian? , 1986 .

[61]  Xiao-Li Meng,et al.  POSTERIOR PREDICTIVE ASSESSMENT OF MODEL FITNESS VIA REALIZED DISCREPANCIES , 1996 .

[62]  Andrew Gelman,et al.  Struggles with survey weighting and regression modeling , 2007, 0710.5005.

[63]  Carl-Erik Särndal,et al.  Model Assisted Survey Sampling , 1997 .

[64]  Alastair Scott Large-Sample Posterior Distributions for Finite Populations , 1971 .

[65]  D. Horvitz,et al.  A Generalization of Sampling Without Replacement from a Finite Universe , 1952 .

[66]  R. Little,et al.  Penalized Spline Nonparametric Mixed Models for Inference About a Finite Population Mean from Two-Stage Samples , 2003 .

[67]  Robert M. Groves,et al.  Total Survey Error: Past, Present, and Future , 2010 .

[68]  B. L. Welch On Comparisons between Confidence Point Procedures in the Case of a Single Parameter , 1965 .

[69]  J. Rao On Variance Estimation with Imputed Survey Data , 1996 .

[70]  William G. Cochran,et al.  Sampling Techniques, 3rd Edition , 1963 .

[71]  Dawei Xie Combining information from multiple surveys for small area estimation: Bayesian approaches. , 2004 .

[72]  Roderick J. A. Little The Bayesian Approach to Sample Survey Inference , 2003 .

[73]  Danny Pfeffermann,et al.  Small Area Estimation , 2011, International Encyclopedia of Statistical Science.

[74]  R. Little Post-Stratification: A Modeler's Perspective , 1993 .

[75]  Morris H. Hansen,et al.  An Evaluation of Model-Dependent and Probability-Sampling Inferences in Sample Surveys: Rejoinder , 1983 .

[76]  Danny Pfeffermann,et al.  Robustness considerations in the choice of a method of inference for regression analysis of survey data , 1985 .

[77]  Wayne A. Fuller,et al.  On the bias of the multiple‐imputation variance estimator in survey sampling , 2006 .

[78]  D. Rubin Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician , 1984 .

[79]  D. Basu,et al.  An Essay on the Logical Foundations of Survey Sampling, Part One* , 2011 .

[80]  Ying Yuan,et al.  Model-based inference for two-stage cluster samples subject to nonignorable item nonresponse , 2008 .