What happens if we compare chopsticks with forks? The impact of making inappropriate comparisons in cross-cultural research.

It is a common practice to export instruments developed in one culture to another. Little is known about the consequences of making inappropriate comparisons in cross-cultural research. Several studies were conducted to fill in this gap. Study 1 examined the impact of lacking factor loading invariance on regression slope comparisons. When factor loadings of a predictor are higher in the reference group (e.g., United States), for which the scale was developed, than in the focal group (e.g., China), into which the scale was imported, the predictive relationship (e.g., self-esteem predicting life satisfaction) is artificially stronger in the reference group but weaker in the focal group, creating a bogus interaction effect of predictor by group (e.g., self-esteem by culture); the opposite pattern is found when the reference group has higher loadings in an outcome variable. Studies 2 and 3 examined the impact of lacking loading and intercept (i.e., point of origin) invariance on factor means, respectively. When the reference group has higher loadings or intercepts, the mean is overestimated in that group but underestimated in the focal group, resulting in a pseudo group difference.

[1]  R. Larsen,et al.  The Satisfaction with Life Scale , 1985, Journal of personality assessment.

[2]  Gordon W. Cheung,et al.  Cross-cultural comparisons using non-invariant measurement items , 1998 .

[3]  R. P. McDonald,et al.  Structural Equations with Latent Variables , 1989 .

[4]  Heather M. Coon,et al.  Rethinking individualism and collectivism: evaluation of theoretical assumptions and meta-analyses. , 2002, Psychological bulletin.

[5]  Dorothy T. Thayer,et al.  Differential Item Performance and the Mantel-Haenszel Procedure. , 1986 .

[6]  Larissa L. Smith,et al.  Gender differences on negative affectivity: an IRT study of differential item functioning on the Multidimensional Personality Questionnaire Stress Reaction Scale. , 1998, Journal of personality and social psychology.

[7]  D. J. Lee.  Society and the Adolescent Self-Image , 1969 .

[8]  A. A. Lumsdaine,et al.  Evaluation and Experiment: Some Critical Issues in Assessing Social Programs , 1976 .

[9]  C. Moore,et al.  The Chinese Mind: Essentials of Chinese Philosophy and Culture , 1978 .

[10]  Y. Poortinga.  Equivalence of cross-cultural data: an overview of basic issues. , 1989, International journal of psychology : Journal international de psychologie.

[11]  S. Reise,et al.  Exploring the measurement invariance of psychological instruments: Applications in the substance use domain. , 1997 .

[12]  S. Oishi,et al.  The mediating role of parental expectations in culture and well-being. , 2005, Journal of personality.

[13]  S. Fiske,et al.  The Handbook of Social Psychology , 1935 .

[14]  Howard T. Everson,et al.  Methodology Review: Statistical Approaches for Assessing Measurement Bias , 1993 .

[15]  R. Nisbett,et al.  Validity problems comparing values across cultures and possible solutions. , 1997 .

[16]  M. Brewer,et al.  Where (who) are collectives in collectivism? Toward conceptual clarification of individualism and collectivism. , 2007, Psychological review.

[17]  Michael Windle,et al.  The science of prevention: methodological advances from alcohol and substance abuse research. , 1997 .

[18]  Virginia S. Y. Kwan,et al.  The Construct of Individuation: More Complex in Collectivist than in Individualist Cultures , 2002 .

[19]  J. Horn,et al.  When is invariance not invarient: A practical scientist's look at the ethereal concept of factor invariance. , 1983 .

[20]  R. Boruch,et al.  3 – Making the Case for Randomized Assignment to Treatments by Considering the Alternatives: Six Ways in Which Quasi-Experimental Evaluations In Compensatory Education Tend to Underestimate Effects , 1975 .

[21]  F. Chen.  Sensitivity of Goodness of Fit Indexes to Lack of Measurement Invariance , 2007 .

[22]  F. V. D. van de Vijver,et al.  Methodological Issues in Psychological Research on Culture , 2000 .

[23]  J. Uleman,et al.  Variations in Collectivism and Individualism by Ingroup and Culture: Confirmatory Factor Analyses , 1996 .

[24]  R. Cattell,et al.  Structural Equation Models of Factorial Invariance in Parallel Proportional Profiles and Oblique Confactor Problems. , 1994, Multivariate behavioral research.

[25]  M. Rosenberg.  Society and the adolescent self-image , 1966 .

[26]  John W. Berry,et al.  ON CROSS‐CULTURAL COMPARABILITY , 1969 .

[27]  B. Byrne,et al.  Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. , 1989 .

[28]  Kwok Leung,et al.  Methods and data analysis of comparative research , 1997 .

[29]  R. Millsap.  Four Unresolved Problems in Studies of Factorial Invariance. , 2005 .

[30]  D. R. Lehman,et al.  Psychology and culture. , 2004, Annual review of psychology.

[31]  J. Steenkamp,et al.  Assessing Measurement Invariance in Cross-National Consumer Research , 1998 .

[32]  S. West,et al.  Teacher's Corner: Testing Measurement Invariance of Second-Order Factor Models , 2005 .

[33]  C. Maslach,et al.  Individuation: Conceptual analysis and assessment. , 1985 .

[34]  W. Meredith.  Measurement invariance, factor analysis and factorial invariance , 1993 .

[35]  T. Little.  Mean and Covariance Structures (MACS) Analyses of Cross-Cultural Data: Practical and Theoretical Issues. , 1997, Multivariate behavioral research.

[36]  D. R. Lehman,et al.  What's wrong with cross-cultural comparisons of subjective Likert scales?: The reference-group effect. , 2002, Journal of personality and social psychology.

[37]  S. Embretson,et al.  Item response theory for psychologists , 2000 .

[38]  Robert J. Vandenberg,et al.  A Central Question in Cross-Cultural Research: Do Employees of Different Cultures Interpret Work-related Measures in an Equivalent Manner?: , 1994 .

[39]  R. Nisbett,et al.  The cultural matrix of social psychology , 1998 .

[40]  R. H. Moorman,et al.  A meta‐analytic review and empirical test of the potential confounding effects of social desirability response sets in organizational behaviour research , 1992 .

[41]  Dag Sörbom,et al.  An alternative to the methodology for analysis of covariance , 1978 .

[42]  Keith F Widaman,et al.  Confirmatory factor analysis and item response theory: two approaches for exploring measurement invariance. , 1993, Psychological bulletin.

[43]  H. Triandis,et al.  Measurement in Cross-Cultural Psychology , 1985 .

[44]  Barbara M. Byrne,et al.  Cross-Cultural Comparisons and the Presumption of Equivalent Measurement and Theoretical Structure , 1999 .

[45]  Chuansheng Chen,et al.  Response Style and Cross-Cultural Comparisons of Rating Scales Among East Asian and North American Students , 1995 .

[46]  Karl G. Jöreskog,et al.  Lisrel 8: User's Reference Guide , 1997 .

[47]  David Watkins,et al.  The Issue Of Measurement Invariance Revisited , 2003 .

[48]  B. Tabachnick,et al.  Using Multivariate Statistics , 1983 .

[49]  Hsieh Yu-wei.  Filial Piety and Chinese Society , 1959 .

[50]  R. Millsap,et al.  Evaluating the impact of partial factorial invariance on selection in two populations. , 2004, Psychological methods.

[51]  K. G. J8reskoC,et al.  Simultaneous Factor Analysis in Several Populations , 2007 .

[52]  K. Jöreskog.  Simultaneous factor analysis in several populations , 1971 .

[53]  Gordon W. Cheung,et al.  Assessing Extreme and Acquiescence Response Sets in Cross-Cultural Research Using Structural Equations Modeling , 2000 .

[54]  H. Markus,et al.  Culture and the self: Implications for cognition, emotion, and motivation. , 1991 .

[55]  S. West,et al.  Measuring individualism and collectivism: The importance of considering differential components, reference groups, and measurement invariance , 2008 .

[56]  D. R. Lehman,et al.  Is there a universal need for positive self-regard? , 1999, Psychological review.