Validating the Quality of Crowdsourced Psychometric Personality Test Items

Personality questionnaire items are normally written by experts in personality theory or psychometrics. This makes personality test development expensive, especially since as many as 40% of the items are discarded after quantitative analysis. Could the crowd create personality items that are as good as the ones developed by professionals? We analyze the psychometric quality of 164 Big Five personality items generated by the crowd (n=82) and show that non-expert item writers are capable of designing high quality personality items. The analysis revealed that 59% of the crowdsourced items had adequate psychometric properties, and thus could be used as an alternative instrument to measure Big Five personality traits. We conclude that crowdsourcing personality items is a possible strategy to generate new items rapidly. We also discuss some of the challenges of crowdsourcing general psychological items.

[1]  Gjergji Kasneci,et al.  Crowd IQ: measuring the intelligence of crowdsourcing platforms , 2012, WebSci '12.

[2]  Jian Tang,et al.  You better be honest: Discouraging free-riding and false-reporting in mobile crowdsourcing , 2014, 2014 IEEE Global Communications Conference.

[3]  S. Reise,et al.  Factor analysis and scale revision. , 2000, Psychological assessment.

[4]  Frank Kleemann,et al.  Un(der)paid innovators: the commercial utilization of consumer work through crowdsourcing , 2008 .

[5]  J. Nunnally.  Psychometric Theory (2nd ed), New York: McGraw-Hill. , 1978 .

[6]  Murray R. Barrick,et al.  THE BIG FIVE PERSONALITY DIMENSIONS AND JOB PERFORMANCE: A META-ANALYSIS , 1991 .

[7]  Ronald K. Hambleton,et al.  Advances in Criterion-Referenced Measurement , 1991 .

[8]  Charles W. Stansfield,et al.  Test translation and adaptation in public education in the USA , 2003 .

[9]  H. Rogers,et al.  Naive versus sophisticated item-writers for the assessment of anxiety. , 1985, Journal of clinical psychology.

[10]  Thomas M. Haladyna,et al.  A technology for test-item writing , 1981 .

[11]  Jason W. Osborne,et al.  Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. , 2005 .

[12]  R. Cattell.  The Scientific Use of Factor Analysis in Behavioral and Life Sciences , 2012 .

[13]  S. West,et al.  The investigation of personality structure: Statistical models , 1997 .

[14]  J. Ruscio,et al.  Determining the number of factors to retain in an exploratory factor analysis using comparison data of known factorial structure. , 2012, Psychological assessment.

[15]  Michael C. Rodriguez,et al.  Developing and Validating Test Items , 2013 .

[16]  Joar Vittersø,et al.  Personality traits and subjective well-being: emotional stability, not extraversion, is probably the important predictor , 2001 .

[17]  B. Thompson.  Exploratory and Confirmatory Factor Analysis: Understanding Concepts and Applications , 2004 .

[18]  David Eichelberger,et al.  Handbook Of Psychological Testing , 2016 .

[19]  David J. Weiss,et al.  Using computerized adaptive testing to reduce the burden of mental health assessment. , 2008, Psychiatric services.

[20]  P. Sheehan,et al.  A Model for Integrating the Public into Scientific Research , 2003 .

[21]  I. Deary,et al.  The NEO-FFI: emerging British norms and an item-level analysis suggest N, A and C are more reliable than O and E , 2000 .

[22]  Cornelis A.W. Glas,et al.  Computerized adaptive testing : theory and practice , 2000 .

[23]  W F Velicer,et al.  Factors Influencing Four Rules For Determining The Number Of Components To Retain. , 1982, Multivariate behavioral research.

[24]  J. Horn.  A rationale and test for the number of factors in factor analysis , 1965, Psychometrika.

[25]  D. Watson,et al.  Constructing validity: Basic issues in objective scale development , 1995 .

[26]  Christos H. Papadimitriou,et al.  Free-riding and whitewashing in peer-to-peer systems , 2004, IEEE Journal on Selected Areas in Communications.

[27]  T. Haladyna.  Developing and Validating Multiple-Choice Test Items , 1994 .

[28]  Daren C. Brabham.  Crowdsourcing as a Model for Problem Solving , 2008 .

[29]  P. Costa,et al.  Revised NEO Personality Inventory (NEO-PI-R) and NEO-Five-Factor Inventory (NEO-FFI) , 1992 .

[30]  Duncan J. Watts,et al.  Financial incentives and the "performance of crowds" , 2009, HCOMP '09.

[31]  Robert Cudeck,et al.  Exploratory Factor Analysis , 2000 .

[32]  Matthew Courtney,et al.  Determining the Number of Factors to Retain in EFA: Using the SPSS R-Menu v2 0 to Make More Judicious Estimations , 2013 .

[33]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[34]  Victoria A. Miller,et al.  Challenges in Measuring a New Construct: Perception of Voluntariness for Research and Treatment Decision Making , 2009, Journal of empirical research on human research ethics : JERHRE.

[35]  John A. Johnson,et al.  The international personality item pool and the future of public-domain personality measures ☆ , 2006 .

[36]  Aniket Kittur,et al.  A comparison of social, learning, and financial strategies on crowd engagement and output quality , 2014, CSCW.

[37]  Rick H. Hoyle,et al.  Confirmatory Factor Analysis , 1983 .

[38]  A Case Study on an Item Writing Process: Use of Test Specifications, Nature of Group Dynamics, and Individual Item Writers' Characteristics , 2010 .

[39]  Mihaela van der Schaar,et al.  Reputation-based incentive protocols in crowdsourcing applications , 2011, 2012 Proceedings IEEE INFOCOM.

[40]  Pedro M. Valero-Mora,et al.  Determining the Number of Factors to Retain in EFA: An easy-to-use computer program for carrying out Parallel Analysis , 2007 .

[41]  Lawrence M. Rudner,et al.  Implementing the Graduate Management Admission Test Computerized Adaptive Test , 2009 .

[42]  Michael Vitale,et al.  The Wisdom of Crowds , 2015, Cell.

[43]  J. Ramalho-Santos,et al.  Cronbach's alpha: a tool for assessing the reliability of scales , 1999 .

[44]  Panagiotis G. Ipeirotis,et al.  STEP: A Scalable Testing and Evaluation Platform , 2014, HCOMP.