The use of paradata to monitor and manage survey data collection

Paradata are automatic data collected about the survey data collection process captured during computer assisted data collection, and include call records, interviewer observations, time stamps, keystroke data, travel and expense information, and other data captured during the process. Increasingly such data are being used in real time to both monitor and manage large scale data collection processes. In this paper we use a statistical process control perspective to describe how such data can be used to monitor the survey process. Process control charts and statistical models can be used to identify areas of concern during data collection, and can lead to further investigation of the problem and (if necessary) intervention. We describe the data and analyses that are available and present several case studies of paradata use in difierent types of surveys and organizations.

[1]  E. Ziegel,et al.  Nonresponse In Household Interview Surveys , 1998 .

[2]  Annelies G. Blom,et al.  Nonresponse bias adjustments: what can process data contribute? , 2009 .

[3]  Tom W. Smith.  Estimating Nonresponse Bias with Temporary Refusals , 1984 .

[4]  David Morganstein,et al.  Continuous Quality Improvement in Statistical Agencies , 1997 .

[5]  Massachusetts Ave,et al.  The Use of Paradata for Evaluating Interviewer Training and Performance , 2000 .

[6]  Dirk Heerwegh,et al.  Explaining Response Latencies and Changing Answers Using Client-Side Paradata from a Web Survey , 2003 .

[7]  Lilli Japec.  Quality Issues in Interview Surveys- Some Contributions , 2006 .

[8]  J. Wagner.  The Fraction of Missing Information as a Tool for Monitoring the Quality of Survey Data , 2010 .

[9]  Richard A. Kulka,et al.  OPTIMAL CALL SCHEDULING FOR A TELEPHONE SURVEY , 1987 .

[10]  Peter Lynn,et al.  PEDAKSI: Methodology for Collecting Data about Survey Non-Respondents , 2003 .

[11]  Robert M. Groves,et al.  Responsive design for household surveys: tools for actively controlling survey errors and costs , 2006 .

[12]  W. Edwards Deming,et al.  Out of the Crisis , 1982 .

[13]  Frauke Kreuter,et al.  Analyzing contact sequences in call record data. Potential and limitations of sequence indicators for nonresponse adjustments in the European Social Survey , 2009 .

[14]  K. Olson,et al.  When do nonresponse follow-ups improve or reduce data quality? A meta-analysis and review of the existing literature , 2008 .

[15]  Shawn A. Ross,et al.  Survey Methodology , 2005, The SAGE Encyclopedia of the Sociology of Religion.

[16]  Mick P. Couper,et al.  Designing a Strategy for Reducing “No Opinion” Responses in Web-Based Surveys , 2002 .

[17]  Mick P. Couper,et al.  Technology Trends in Survey Data Collection , 2005 .

[18]  Emilia Peytcheva,et al.  Relationship between Measurement Error and Unit Nonresponse in Household Surveys : An Approach in the Absence of Validation Data , 2007 .

[19]  Mick P. Couper,et al.  Usability Evaluation of Computer-Assisted Survey Instruments , 2000 .

[20]  Annette Jäckle,et al.  Understanding cross-national differences in unit non-response: the role of contact data , 2008 .

[21]  Frauke Kreuter,et al.  Multiple Auxiliary Variables in Nonresponse Adjustment , 2011 .

[22]  James Wagner,et al.  Use of Paradata in a Responsive Design Framework to Manage a Field Data Collection , 2012 .

[23]  Robert M. Groves,et al.  Using proxy measures and other correlates of survey outcomes to adjust for non‐response: examples from multiple surveys , 2010 .

[24]  Michael J. Stern,et al.  The Use of Client-side Paradata in Analyzing the Effects of Visual Layout on Changing Responses in Web Surveys , 2008 .

[25]  Paul B. Sheatsley,et al.  Nonresponse Bias for Attitude Questions , 1981 .

[26]  Roger Tourangeau,et al.  Taking the Audio Out of Audio-CASI , 2009 .

[27]  Paul P. Biemer,et al.  The Evolution of Audio Recording in Field Surveys , 2008 .

[28]  Eric R. Ziegel,et al.  Survey measurement and process quality , 1998 .

[29]  J. N. Bassili,et al.  Response-time measurement in survey research: A method for CATI and a new look at nonattitudes. , 1991 .

[30]  Pei-Lu Chiu,et al.  A COMPARISON OF CHARACTERISTICS BETWEEN LATE/DIFFICULT AND NON- LATE/DIFFICULT INTERVIEWS IN THE NATIONAL HEALTH INTERVIEW SURVEY , 2001 .

[31]  Joseph Moses Juran,et al.  Quality Planning and Analysis: From Product Development Through Usage , 1982 .

[32]  R. Tourangeau,et al.  Fast times and easy questions: the effects of age, experience and question complexity on web survey response times , 2008 .

[33]  Paul Biemer,et al.  Continuous quality improvement for survey operations: some general principles and applications , 1994 .

[34]  Matt E. Jans,et al.  Verbal Paradata and Survey Error: Respondent Speech, Voice, and Question-Answering Behavior Can Predict Income Item Nonresponse. , 2010 .

[35]  Daniel Thorburn.  Control of the coding operation in statistical investigations : some contributions , 2005 .

[36]  N. Nayab.  Kaizen: The key to Japan‚s competitive success , 2009 .

[37]  K. Green.  Reluctant Respondents: Differences between Early, Late and Nonresponders to a Mail Survey. , 1991 .

[38]  Fred Spiring,et al.  Introduction to Statistical Quality Control , 2007, Technometrics.

[39]  Jelke Bethlehem,et al.  Indicators for the representativeness of survey response , 2009 .

[40]  G. Loosveldt,et al.  The History of the Contact Procedure and Survey Cooperation : Applying Demographic Methods to European Social Survey Contact Forms Round 2 in Belgium , 2006 .

[41]  F. Kreuter.  Kriminalitätsfurcht: Messung und methodische Probleme , 2002 .

[42]  A. Peytchev,et al.  Not All Survey Effort is Equal: Reduction of Nonresponse Bias and Nonresponse Error , 2009 .

[43]  Geert Loosveldt,et al.  Minimizing survey refusal and noncontact rates: do our efforts pay off? , 2007 .

[44]  I. Stoop,et al.  Estimation of nonresponse bias in the European Social Survey: using information from reluctant respondents , 2007 .

[45]  Kristen Olson,et al.  Survey Participation, Nonresponse Bias, Measurement Error Bias, and Total Bias , 2006 .

[46]  David R. Morganstein,et al.  Survey operations processes: the key to quality improvement , 1991 .

[47]  Paul P. Biemer,et al.  Introduction to Survey Quality , 2003 .

[48]  Gerard Nelson.  To agree or not to agree , 2011 .

[49]  S. Nunn.  The Hunt for the Last Respondent by I. Stoop , 2006 .

[50]  Mick Couper,et al.  The Use of Paradata in Survey Research , 2022 .

[51]  James M. Dahlhamer,et al.  Privacy concerns, too busy, or just not interested: using doorstep concerns to predict survey nonresponse , 2008 .

[52]  Mick P. Couper,et al.  MEASURING SURVEY QUALITY IN A CASIC ENVIRONMENT , 2002 .

[53]  Kenneth G. Manton,et al.  Correcting for nonavailability bias in surveys by weighting based on number of callbacks , 1993 .

[54]  P. Biemer,et al.  Using Callback Models to Adjust for Nonignorable Nonresponse in Face-to-Face Surveys , 2007 .

[55]  Adaptive Contact Strategies in a Telephone Survey , 2009 .

[56]  Susan Purdon,et al.  Can you hear me knocking? and investigation into the impact of interviewers on survey response rates , 1997 .

[57]  George Minton,et al.  Inspection and Correction Error in Data Processing , 1969 .

[58]  Willem E. Saris,et al.  Mixed mode designs: finding the balance between nonresponse bias and mode effects , 2005 .

[59]  Carl E. Pierchala,et al.  Control charts as a tool for data quality control , 2009 .

[60]  Mandi Yu.  Are Nonrespondents Necessarily Bad Reporters ? Using Imputation Techniques to Investigate Measurement Error of Nonrespondents in an Alumni Survey , 2007 .

[61]  Lars Kaczmirek,et al.  Cognitive burden of survey questions and response times: A psycholinguistic experiment , 2010 .

[62]  Paul Clarke,et al.  The effects of extended interviewer effort on non-response bias , 2001 .