Benefits from Using Continuous Rating Scales in Online Survey Research

Likert-type scales require individuals to choose between a limited number of choices, and have been criticized in the scholarly literature for causing loss of information, allowing the researcher to affect responses by determining the range, and being ordinal in nature. The use of online surveys allows for the easy implementation of continuous rating scales, which have a long history in psychophysical measurement but are rarely used in IS surveys. This type of measurement requires survey participants to express their opinion in a visual form. That not only solves the problems of information loss, but also allows for applying advanced robust statistical analyses. We use a real-world sample and a simulation to illustrate how noise impacts our data set. A noise level of 10% has only a small effect on both classical and robust estimates, but when 20% of noise is added, the classical estimators become severely distorted.

[1]  Peter Filzmoser,et al.  Exploratory factor analysis revisited: How robust methods support the detection of hidden multivariate data structures in IS research , 2010, Inf. Manag..

[2]  M. Power,et al.  Literacy affected ability to adequately discriminate among categories in multipoint Likert Scales. , 2009, Journal of clinical epidemiology.

[3]  Danny Weathers,et al.  Assessing Three Sources of Misresponse to Reversed Likert Items , 2008 .

[4]  Nicola Stanley,et al.  Watch What I Do! - Using graphical input controls in web surveys , 2007 .

[5]  R. Grover.  The Handbook of Marketing Research: Uses, Misuses, and Future Advances , 2006 .

[6]  V. Yohai,et al.  Robust Statistics: Theory and Methods , 2006 .

[7]  Chad R. Allred,et al.  Advanced Techniques and Technologies in Online Research , 2006 .

[8]  Terry Anthony Byrd,et al.  A methodology for construct development in MIS research , 2005, Eur. J. Inf. Syst..

[9]  Robert I. Jennrich,et al.  Gradient Projection Algorithms and Software for Arbitrary Rotation Criteria in Factor Analysis , 2005 .

[10]  Ming-Hui Huang,et al.  Web performance scale , 2005, Inf. Manag..

[11]  R. Steve McCallum,et al.  A comparison of the pencil-and-paper and computer-administered Minnesota Multiphasic Personality Inventory–Adolescent , 2005 .

[12]  Edward Haletky.  11 – An Investigative Approach , 2005 .

[13]  Scott M. Smith,et al.  Fundamentals of Marketing Research , 2004 .

[14]  Victoria J. Hodge,et al.  A Survey of Outlier Detection Methodologies , 2004, Artificial Intelligence Review.

[15]  I. Brace,et al.  Questionnaire Design: How to Plan, Structure and Write Survey Material for Effective Market Research , 2004 .

[16]  S. Gosling,et al.  Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. , 2004, The American psychologist.

[17]  Horst Treiblmaier,et al.  Antecedents of the Adoption of E-Payment Services in the Public Sector , 2004, ICIS.

[18]  Detmar W. Straub,et al.  Validation Guidelines for IS Positivist Research , 2004, Commun. Assoc. Inf. Syst..

[19]  Christopher D. Allport,et al.  A Research Note Regarding the Development of the Consensus on Appropriation Scale , 2003, Inf. Syst. Res..

[20]  P. Rousseeuw,et al.  Robust factor analysis , 2003 .

[21]  T. B. Rogers,et al.  Measurement in psychology: A critical history of a methodological concept , 2002 .

[22]  Christopher J. McCarthy,et al.  Paper-and-Pencil Or Online? , 2002, Assessment.

[23]  Jonathan W. Palmer,et al.  Web Site Usability, Design, and Performance Metrics , 2002, Inf. Syst. Res..

[24]  W. Shofner,et al.  Pitch strength and Stevens’s power law , 2002, Perception & psychophysics.

[25]  Wynne W. Chin,et al.  Research Report: Better Theory Through Measurement-Developing a Scale to Capture Consensus on Appropriation , 2002, Inf. Syst. Res..

[26]  Peter Caputi,et al.  Introduction to Quantitative Research Methods: An Investigative Approach , 2001 .

[27]  Michael R. Harwell,et al.  Rescaling Ordinal Data to Interval Data in Educational Research , 2001 .

[28]  Abhay Shah,et al.  Goodness-of-fit tests for rating scale data: Applying the minimum chi-square method , 2001 .

[29]  Peter Caputi,et al.  Introduction to quantitative research methods , 2001 .

[30]  D. Kaplan.  Structural Equation Modeling: Foundations and Extensions , 2000 .

[31]  H. Schutz.  Consumer data—sense and nonsense , 1999 .

[32]  J. Michell.  Measurement in psychology: A critical history of a methodological concept. , 1999 .

[33]  Katrien van Driessen,et al.  A Fast Algorithm for the Minimum Covariance Determinant Estimator , 1999, Technometrics.

[34]  K. Yuan,et al.  9. Structural Equation Modeling with Robust Covariances , 1998 .

[35]  M. Pett.  Nonparametric Statistics for Health Care Research: Statistics for Small Samples and Unusual Distributions , 1997 .

[36]  Wynne W. Chin,et al.  Panel 17 Resolved: Surveys Have Outlived Their Usefulness in IS Research , 1996, ICIS.

[37]  Juhani Iivari,et al.  Why do individuals use computer technology? A Finnish case study , 1995, Inf. Manag..

[38]  A. Basilevsky.  Statistical Factor Analysis and Related Methods: Theory and Applications , 1994 .

[39]  Alain Pinsonneault,et al.  Survey Research Methodology in Management Information Systems: An Assessment , 1993, J. Manag. Inf. Syst..

[40]  Kent W. Smith,et al.  Survey Responses: An Evaluation of Their Validity , 1993 .

[41]  Izak Benbasat,et al.  Development of an Instrument to Measure the Perceptions of Adopting an Information Technology Innovation , 1991, Inf. Syst. Res..

[42]  Detmar W. Straub,et al.  Validating Instruments in MIS Research , 1989, MIS Q..

[43]  H. Marsh,et al.  A new, more powerful approach to multitrait-multimethod analyses: Application of second-order confirmatory factor analysis. , 1988 .

[44]  Joseph Moses Juran.  Juran on planning for quality , 1988 .

[45]  M. Hazewinkel.  Encyclopaedia of mathematics , 1987 .

[46]  Bruno Neibecker,et al.  The Validity of Computer-Controlled Magnitude Scaling to Measure Emotional Impact of Stimuli , 1984 .

[47]  Milton Lodge,et al.  Magnitude Scaling: Quantitative Measurement of Opinions , 1981 .

[48]  G. Albaum,et al.  The Effect of Varying Response Intervals on the Stability of Factor Solutions of Rating Scale Data , 1979 .

[49]  S. S. Stevens,et al.  Psychophysics: Introduction to Its Perceptual, Neural and Social Prospects , 1975 .

[50]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[51]  S S Stevens,et al.  On the Theory of Scales of Measurement. , 1946, Science.