Tackling the Problem of Self Selection in the Integration of Different Data Collection Techniques.

A lot of studies are showing an increased tendency to use more than one mode of administration to collect data for a particular analysis ([10];[1];[16];[9]). Thus, understand if different data collection methods influence answers becomes a concern. The problem of self selection into different interview modes demands attention especially when the assignment to one or another collection method is not randomly controlled and respondents might self select in one data collection method over another. This paper shows an empirical case concerning the evalua- tion of two different data collection method: the CAWI (Computer Assisted Web Interview) method and the CATI (Computer Assisted Telephone Interview) method. If self-selection exists both mode of data collection and characteristics of the re- spondents influence answers. Hence, the mode effect may be confounded. In order to estimate an unbiased mode effect, this paper proposes a data driven multivariate approach to monitor self-selection that allows to disentangle interview’s modes ef- fects on answers from the effect of self-selection. We will work through the use of the monitoring system with an empirical case. In particular, we will use AlmaLaurea data and compare results of our approach to PS adjustment method that AlmaLaurea usually applies to control data quality as documented in various reports and analysis conducted by the AlmaLaurea Consortium .

[1]  Matthias Schonlau,et al.  Selection Bias in Web Surveys and the Use of Propensity Scores , 2006 .

[2]  Gary King,et al.  The Dangers of Extreme Counterfactuals , 2006, Political Analysis.

[3]  B. Janssen.  Web data collection in a mixed mode approach: an experiment , 2006 .

[4]  S. Ghiselli,et al.  Graduates’ Employment and Employability after the 'Bologna Process' Reform: Evidence from the Italian Experience and Methodological Issues , 2010 .

[5]  Jelke Bethlehem,et al.  Selection Bias in Web Surveys , 2010 .

[6]  J. Borkan.  Mixed Methods Studies: A Foundation for Primary Care Research , 2004, The Annals of Family Medicine.

[7]  W. G. Cochran.  The effectiveness of adjustment by subclassification in removing bias in observational studies. , 1968, Biometrics.

[8]  Don A. Dillman,et al.  Understanding differences in people's answers to telephone and mail surveys , 1996 .

[9]  Gary King,et al.  Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference , 2007, Political Analysis.

[10]  Josep Daunis-i-Estadella,et al.  Distribution of the inter and intra inertia in conditional MCA , 2005, Comput. Stat..

[11]  Sunghee Lee.  Propensity score adjustment as a weighting scheme for volunteer panel web surveys , 2006 .

[12]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[13]  Ida D'Attoma,et al.  A multivariate strategy to measure and test global imbalance in observational studies , 2011, Expert Syst. Appl..

[14]  M. Elliott,et al.  Conducting Research Surveys Via E-Mail and The Web , 2001 .

[15]  Ida D'Attoma,et al.  A new data mining approach to estimate causal effects of policy interventions , 2010, Expert Syst. Appl..

[16]  Geert Molenberghs,et al.  A Method for Evaluating Mode Effects in Mixed-mode Surveys , 2010 .

[17]  Bethlehem.  UvA-DARE ( Digital Academic Repository ) Selection bias in web surveys , 2010 .

[18]  Laura R. Peck,et al.  A Promising New Approach to Eliminating Selection Bias , 2009, Canadian Journal of Program Evaluation.

[19]  Brigitte Escofier.  Analyse des correspondances multiples conditionnelle , 1990, Monde des Util. Anal. Données.

[20]  Henry F. Woltman,et al.  A Comparison of Three Mixed-Mode Interviewing Procedures in the National Crime Survey , 1980 .

[21]  E. Leeuw,et al.  To mix or not to mix data collection modes in surveys. , 2005 .

[22]  F. Kreuter,et al.  Social Desirability Bias in CATI, IVR, and Web Surveys The Effects of Mode and Question Sensitivity , 2008 .

[23]  S. Ghiselli,et al.  AlmaLaurea Inter-University Consortium Graduates' employment and employability after the "Bologna Process" reform. Evidence from the Italian experience and methodological issues , 2009 .