Examining the Measurement Invariance between Paper-and-Pencil and Internet-Administered Tests in China

Concerns about the quality of internet-based tests have been brought by the increasing applications of such tests in psychological research. Over the past years, a large body of studies has been conducted to examine the equivalence of internet-administered tests to their paper-and-pencil counterparts. Although studies based on Classic Test Theory (CTT) showed that internet-administered tests were trustable, studies based on measurement invariance tests produced mixed findings. What is more, most studies so far have been conducted in individualistic cultures. Given these,, the present study aimed to examine the equivalence between internet-based and paper-and-pencil tests in a collectivistic culture, particularly, in China. To this end, we employed Confirmatory Factor Analysis (CFA) to examine the measurement invariance of the selected scale: Satisfaction with Life Scale (SWLS) across modes. SWLS was administered via internet and the paper-and-pencil modes. Five items were rated on a 7-point likert scale ranging from 1 ("strongly disagree") to 7 ("strongly agree"). A total of 418 self-selected college students from 26 provinces in China took the internet-based test. And a total of 288 college students at Sun Yat-Sen University were sampled to take the paper-and-pencil test in classroom. For the internet sample, the age ranges from 18 to 24 years old with a mean of 21.31 (SD= 1.38); for the paper-and-pencil sample, the age ranges from 18 to 24 years old with a mean of 20.20 (SD=0.81). Multi-group CFA was employed to test measurement invariance between the internet administered and the paper-and-pencil SWLS. Results showed weak measurement invariance held across these two test modes, indicating metric similarity between the tests; partial strong measurement invariance and partial strict measurement invariance also held, suggesting that response bias existed in some items across modes; further analysis revealed that the paper-and-pencil test included more noise arising from administering environment. In terms of mean comparisons, significant differences between modes were found in observed scores but not in latent scores. For the variances, no significant differences were found between modes in either latent scores or observed scores. These findings suggested that administering environments produced potential impacts on observed scores. As the first examination of the measurement invariance in Chinese samples, the study provided initial evidence that internet-based tests have equivalent metrics with paper-and-pencil tests. Further, the results from the partial strong invariance and partial strict invariance may indicate the sensitivity of Chinese people to environments that may be resulted from collectivistic culture. Taken together, the findings from this study suggest that although internet-based tests are trustable in China, cautions of response biases should be kept in mind when conducting cross-groups (or modes) comparisons. Also, the findings underscore the importance of examining measurement invariance when a test is applied across multi-groups (or modes).