A Test of a Computer-adaptive Survey using Online Reviews

© 26th European Conference on Information Systems: Beyond Digitization - Facets of Socio-Technical Change, ECIS 2018. All Rights Reserved. Traditional surveys are excellent instruments for establishing the correlational relationship between two constructs. However, they are unable to identify reasons why such correlations exist. Computer-Adaptive Surveys (CAS) are multi-dimensional instruments where questions asked of respondents depend on the previous questions asked. Assessing the validity of CAS is an underexplored research area as CAS differs from traditional surveys. Therefore, validating a CAS requires different techniques. This study attempts to validate the conclusion validity of a CAS about cafe customer satisfaction using online customer reviews. For our CAS to have conclusion validity, there should be a high correspondence where most respondents in CAS and online reviewers both agree that certain constructs are the cause of their dissatisfaction. We created a Computer-Adaptive Survey (CAS) of cafe satisfaction and used online customer reviews to assess its conclusion validity. Our research thus contributes to the measurement literature in two ways, one, we demonstrate that CAS captures the same criticisms of cafes as that in online reviews, and two, CAS captures problems about customer satisfaction at a deeper level than that found in online reviews.

[1]  K. Pearson.  On the χ 2 Test of Goodness of Fit , 1922 .

[2]  F. Massey.  The Kolmogorov-Smirnov Test for Goodness of Fit , 1951 .

[3]  Z. Birnbaum.  Numerical Tabulation of the Distribution of Kolmogorov's Statistic for Finite Sample Size , 1952 .

[4]  Edgar A. Smith.  Devereux Readability Index , 1961 .

[5]  J. Klotz.  Asymptotic Efficiency of the Two Sample Kolmogorov-Smirnov Test , 1967 .

[6]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[7]  A. Pettitt,et al.  The Kolmogorov-Smirnov Goodness-of-Fit Statistic with Discrete and Grouped Data , 1977 .

[8]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[9]  F. Lord.  Applications of Item Response Theory To Practical Testing Problems , 1980 .

[10]  Jean D. Gibbons,et al.  Kolmogorov-Smirnov Two-Sample Tests , 1981 .

[11]  Peter M. Bentler,et al.  Practical Issues in Structural Modeling , 1987 .

[12]  David Thissen,et al.  Data analysis using item response theory. , 1988 .

[13]  R. Peterson,et al.  Measuring customer satisfaction: Fact and artifact , 1992 .

[14]  A. Strauss,et al.  Grounded theory methodology: An overview. , 1994 .

[15]  S. Lankford,et al.  Response Bias and Wave Analysis of Mailed Questionnaires in Tourism Impact Assessments , 1995 .

[16]  Scott E. Sampson.  Gathering customer feedback via the Internet: instruments and prospects , 1998 .

[17]  S. Embretson,et al.  Item response theory for psychologists , 2000 .

[18]  Joel D. Wisner,et al.  Comparing practices for capturing bank customer feedback ‐ Internet versus traditional banking , 2001 .

[19]  Gordon W. Cheung,et al.  Evaluating Goodness-of-Fit Indexes for Testing Measurement Invariance , 2002 .

[20]  Richard T. Watson,et al.  Q-sorting and MIS Research: A Primer , 2002, Commun. Assoc. Inf. Syst..

[21]  J. Pratten.  Customer satisfaction and waiting staff , 2004 .

[22]  Stephen R. Porter,et al.  Multiple Surveys of Students and Survey Fatigue. , 2004 .

[23]  Bing Liu,et al.  Mining and summarizing customer reviews , 2004, KDD.

[24]  Martin Wetzels,et al.  Response Rate and Response Quality of Internet-Based Surveys: An Experimental Study , 2004 .

[25]  Detmar W. Straub,et al.  Validation Guidelines for IS Positivist Research , 2004, Commun. Assoc. Inf. Syst..

[26]  T. Shanka,et al.  Assessment of university campus café service: The students' perceptions , 2005 .

[27]  R. Groves.  Nonresponse Rates and Nonresponse Bias in Household Surveys , 2006 .

[28]  Geert Loosveldt,et al.  An experimental study on the effects of personalization, survey length statements, progress indicators, and survey sponsor logos in web surveys , 2006 .

[29]  Gita Z. Wilder,et al.  Test Takers' Attitudes and Beliefs About the Graduate Management Admission Test , 2006 .

[30]  Patrick J. Moreo,et al.  Customers' Satisfaction Factors Regarding University Food Court Service , 2006 .

[31]  S. Rogelberg,et al.  Introduction Understanding and Dealing With Organizational Survey Nonresponse , 2007 .

[32]  Bing Liu,et al.  Analyzing and Detecting Review Spam , 2007, Seventh IEEE International Conference on Data Mining (ICDM 2007).

[33]  Peter Tymms,et al.  Identifying reading problems with computer-adaptive assessments , 2007, J. Comput. Assist. Learn..

[34]  Fang Wu,et al.  How Public Opinion Forms , 2008, WINE.

[35]  Ingoo Han,et al.  The effect of negative online consumer reviews on product attitude: An information processing view , 2008, Electron. Commer. Res. Appl..

[36]  H. Vorst,et al.  Computerized Adaptive Testing of Personality Traits , 2008 .

[37]  Kisang Ryu,et al.  DINESCAPE: A Scale for Customers' Perception of Dining Environments , 2008 .

[38]  Evangelos Grigoroudis,et al.  Customer Satisfaction Evaluation: Methods for Measuring and Implementing Service Quality , 2009 .

[39]  M. Galesic,et al.  Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web Survey , 2009 .

[40]  X. Liang,et al.  Investigation of customer satisfaction in student food service: An example of student cafeteria in NHH , 2009 .

[41]  D. Weiss,et al.  Termination Criteria in Computerized Adaptive Tests: Variable-Length CATs Are Not Biased , 2009 .

[42]  Pattarachai Lalitrojwong,et al.  Mining Feature-Opinion in Online Customer Reviews for Opinion Summarization , 2010, J. Univers. Comput. Sci..

[43]  Jinsoo Hwang,et al.  Factors Influencing Customer Satisfaction or Dissatisfaction in the Restaurant Business Using AnswerTree Methodology , 2010 .

[44]  David Schuff,et al.  What Makes a Helpful Review? A Study of Customer Reviews on Amazon.com , 2010 .

[45]  Mattias Elg,et al.  Continuous learning using dissatisfaction feedback in new product development contexts , 2010 .

[46]  David J. Weiss,et al.  A Framework for the Development of Computerized Adaptive Tests. , 2011 .

[47]  Stephen Shaoyi Liao,et al.  Mining comparative opinions from customer reviews for Competitive Intelligence , 2011, Decis. Support Syst..

[48]  Riccardo Silvi,et al.  A framework for business analytics in performance management , 2012 .

[49]  Bing Liu,et al.  Sentiment Analysis and Opinion Mining , 2012, Synthesis Lectures on Human Language Technologies.

[50]  Chingwei David Shin,et al.  A Comparison of Three Content Balancing Methods for Fixed and Variable Length Computerized Adaptive Tests. , 2012 .

[51]  Tsung-Han Ho,et al.  Item Selection and Ability Estimation Procedures for a Mixed-Format Adaptive Test , 2012 .

[52]  Markus Zanker,et al.  Classification of Customer Reviews based on Sentiment Analysis , 2012, ENTER.

[53]  Anil Bilgihan,et al.  An analysis of user‐generated content for hotel experiences , 2013 .

[54]  Christos Faloutsos,et al.  Why people hate your app: making sense of user feedback in a mobile app store , 2013, KDD.

[55]  Björn W. Schuller,et al.  New Avenues in Opinion Mining and Sentiment Analysis , 2013, IEEE Intelligent Systems.

[56]  Erkan Sağlik.  Service Quality and Customer Satisfaction Relationship : A Research in Erzurum Ataturk , 2014 .

[57]  Claudia López,et al.  Analysis of local online review systems as digital word-of-mouth , 2014, WWW.

[58]  T. M. Tripp,et al.  Managing social media crises with your customers: The good, the bad, and the ugly , 2015 .

[59]  Cecil Eng Huang Chua,et al.  A Q-Sorting Methodology for Computer-adaptive Surveys - Style "Research" , 2016, ECIS.

[60]  Cecil Eng Huang Chua,et al.  A Threshold for a Q-Sorting Methodology for Computer-adaptive Surveys , 2017, ECIS.

[61]  Cecil Eng Huang Chua,et al.  Computer-Adaptive Surveys (CAS) as a Means of Answering Questions of Why , 2017, PACIS.