Differentiating Response Styles and Construct-Related Responses: A New IRT Approach Using Bifactor and Second-Order Models

Response styles (RS) can interfere if rating or Likert scales are used to assess personality traits, biasing survey results and affecting the validity of the measurement. Based on a new approach using multidimensional item response theory (MIRT) models (Bockenholt, U., Psychological Methods, 2012, doi: 10.1037/a0028111), rating data can be analyzed and RS can be controlled for by a decomposition of the ordinal response into different sequential processes using a binary decision tree. Expanding on this approach (using multiscale questionnaires), Khorramdel and von Davier (in press) showed that the obtained RS measures are not always unidimensional but may be confounded with trait-related responses. The current paper addresses this problem by applying bifactor and second-order item response theory (IRT) models to pseudo items, examining extreme RS (ERS) and midpoint RS (MRS). In contrast to simple-structure MIRT models, these models allow item responses to depend on two factors: a general factor (RS) and a specific factor (personality trait). Findings from two empirical applications using questionnaires which measure the Big Five personality dimensions and a five-point Likert scale show a superior model fit of the bifactor model with regard to ERS and MRS. While the ERS measure clearly appears to be a distinct RS measure, the MRS appears not to be, but both response style measurements show high model-based reliabilities.

[1]  D. Paulhus.  Measurement and control of response bias. , 1991 .

[2]  Frank Rijmen,et al.  Hierarchical factor item response theory models for PIRLS: capturing clustering effects at multiple levels , 2011 .

[3]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[4]  Y. Poortinga,et al.  Response Styles in Rating Scales , 2004 .

[5]  Eric T. Bradlow,et al.  Testlet Response Theory and Its Applications , 2007 .

[6]  H. Akaike.  A new look at the statistical model identification , 1974 .

[7]  U. Böckenholt.  Modeling multiple response processes in judgment and choice. , 2012, Psychological methods.

[8]  Chuansheng Chen,et al.  Response Style and Cross-Cultural Comparisons of Rating Scales Among East Asian and North American Students , 1995 .

[9]  T. Karner.  The Volunteer Effect of Answering Personality Questionnaires , 2002 .

[10]  Matthias von Davier,et al.  Multivariate and Mixture Distribution Rasch Models , 2007 .

[11]  I. Borg,et al.  Correcting Big Five Personality Measurements for Acquiescence: An 18–Country Cross–Cultural Study , 2013 .

[12]  S. Heine,et al.  Cultural differences in response styles: The role of dialectical thinking , 2008 .

[13]  Lale Khorramdel,et al.  Measuring Response Styles Across the Big Five: A Multiscale Extension of an Approach Using Multinomial Processing Trees , 2014, Multivariate behavioral research.

[14]  Bert Weijters,et al.  Assessing response styles across modes of data collection , 2008 .

[15]  Bettina Grün,et al.  Response Style Contamination of Student Evaluation Data , 2009 .

[16]  Georg Rasch,et al.  Probabilistic Models for Some Intelligence and Attainment Tests , 1981, The SAGE Encyclopedia of Research Design.

[17]  Donald Hedeker,et al.  Full-information item bi-factor analysis , 1992 .

[18]  Frank Rijmen,et al.  Three Multidimensional Models for Testlet-Based Tests: Formal Relations and an Empirical Comparison. Research Report. ETS RR-09-37. , 2009 .

[19]  Ingwer Borg,et al.  The measurement equivalence of Big Five factor markers for persons with different levels of education. , 2010, Journal of research in personality.

[20]  L. A. Goodman.  On Quasi-Independence and Quasi-Dependence in Contingency Tables, with Special Reference to Ordinal Triangular Contingency Tables , 1994 .

[21]  John P. Robinson,et al.  Measures Of Personality And Social Psychological Attitudes , 1991 .

[22]  Daniel M. Bolt,et al.  On the Use of Factor-Analytic Multinomial Logit Item Response Models to Account for Individual Differences in Response Style , 2010 .

[23]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[24]  Matthias von Davier,et al.  Introduction: Extending the Rasch Model , 2007 .

[25]  Jean-Paul Fox,et al.  Using Item Response Theory to Measure Extreme Response Style in Marketing Research: A Global Investigation , 2008 .

[26]  Anne-Wil Harzing.  Response Styles in Cross-national Survey Research , 2006 .

[27]  Daniel M. Bolt,et al.  Addressing Score Bias and Differential Item Functioning Due to Individual Differences in Response Style , 2009 .

[28]  M. Geuens,et al.  The stability of individual response styles. , 2008, Psychological methods.

[29]  Stephen E. Fienberg,et al.  Quasi-independence and Maximum Likelihood Estimation in Incomplete Contingency Tables , 1970 .

[30]  Matthias von Davier,et al.  A GENERAL DIAGNOSTIC MODEL APPLIED TO LANGUAGE TESTING DATA , 2005 .

[31]  Melvin R. Novick,et al.  Some latent train models and their use in inferring an examinee's ability , 1966 .

[32]  M. Gail.  Mixed Quasi-Independent Models for Categorical Data , 1972 .

[33]  B. Ripley,et al.  An “Unfolding” Latent Variable Model for Likert Attitude Data , 2007 .

[34]  John B. Campbell,et al.  Extreme Response Style in Cross-Cultural Research , 1974 .

[35]  Patrick M. O'Malley,et al.  Yea-Saying, Nay-Saying, and Going to Extremes: Black-White Differences in Response Styles , 1984 .

[36]  Gregory R. Hancock,et al.  Advances in Latent Variable Mixture Models , 2007 .

[37]  Hans Baumgartner,et al.  Response Styles in Marketing Research: A Cross-National Investigation , 2001 .

[38]  P. Costa,et al.  Validation of the five-factor model of personality across instruments and observers. , 1987, Journal of personality and social psychology.

[39]  Lale Khorramdel,et al.  The Effect of Speediness on Personality Questionnaires: An Experiment on Applicants within a Job Recruiting Procedure , 2006 .

[40]  David Thissen,et al.  On the relationship between the higher-order factor model and the hierarchical factor model , 1999 .

[41]  Daniel M. Bolt,et al.  Multiscale Measurement of Extreme Response Style , 2011 .

[42]  John A. Johnson,et al.  The international personality item pool and the future of public-domain personality measures ☆ , 2006 .

[43]  Harry C. Triandis,et al.  Effects of Culture and Response Format on Extreme Response Style , 1989 .

[44]  Matthias von Davier,et al.  Multivariate and Mixture Distribution Rasch Models: Extensions and Applications , 2006 .

[45]  M. Davier.  Hierarchical mixtures of diagnostic models , 2010 .

[46]  J. Buckley.  Cross-National Response Styles in International Educational Assessments: Evidence from PISA 2006 , 2009 .

[47]  J. Vermunt,et al.  The Impact of Controlling for Extreme Responding on Measurement Equivalence in Cross-Cultural Research , 2012 .

[48]  Stephen G. Sireci,et al.  ON THE RELIABILITY OF TESTLET‐BASED TESTS , 1991 .

[49]  Young Ik Cho,et al.  The Relation Between Culture and Response Styles , 2005 .