Cultural consensus theory for continuous responses: A latent appraisal model for information pooling

Abstract A Cultural Consensus Theory approach for continuous responses is developed, leading to a new model called the Continuous Response Model (CRM). It is a cognitive psychometric model that is applicable to consensus data, in which respondents (informants) have answered questions (items) regarding a shared knowledge or belief domain, and where a consensus (a latent set of ‘true’ answers applicable to the group) may exist. The model estimates the consensus answers to items, item difficulty, informant knowledge and response biases. The model can handle subcultures that have different consensuses from one another in the data, and can both detect and cluster respondents into these subcultures; it thus provides one of the first forms of model-based clustering for multicultural consensus data of the continuous response type. The model is demonstrated on both simulated and real multi-cultural data, using the hierarchical Bayesian framework for inference; two posterior predictive checks are developed to verify the central assumptions of the model; and software is provided to facilitate the application of the model by other researchers.

[1]  Brandon M. Turner,et al.  Evaluating Probabilistic Forecasts with Bayesian Signal Detection Models , 2014, Risk analysis : an official publication of the Society for Risk Analysis.

[2]  Cajo J. F. ter Braak,et al.  A Markov Chain Monte Carlo version of the genetic algorithm Differential Evolution: easy Bayesian computing for real parameter spaces , 2006, Stat. Comput..

[3]  Richard Gonzalez,et al.  On the Shape of the Probability Weighting Function , 1999, Cognitive Psychology.

[4]  Hang Zhang,et al.  Ubiquitous Log Odds: A Common Representation of Probability and Frequency Distortion in Perception, Action, and Cognition , 2012, Front. Neurosci..

[5]  Brandon M. Turner,et al.  A Wisdom of the Crowd Approach to Forecasting , 2011 .

[6]  Brandon M. Turner,et al.  A method for efficiently sampling from distributions with correlated dimensions. , 2013, Psychological methods.

[7]  William H Batchelder,et al.  Cultural Consensus Theory for the Ordinal Data Case , 2015, Psychometrika.

[8]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[9]  Brandon M. Turner,et al.  Forecast aggregation via recalibration , 2014, Machine Learning.

[10]  Stephen L. France,et al.  Maximum Likelihood Item Easiness Models for Test Theory Without an Answer Key , 2015, Educational and psychological measurement.

[11]  W. Batchelder,et al.  Cultural consensus theory for multiple consensus truths , 2012 .

[12]  Aki Vehtari Discussion to "Bayesian measures of model complexity and fit" by Spiegelhalter, D.J., Best, N.G., Carlin, B.P., and van der Linde, A. , 2002 .

[13]  M. Stephens Dealing with label switching in mixture models , 2000 .

[14]  S. Weller Cultural Consensus Theory: Applications and Frequently Asked Questions , 2007 .

[15]  Mark Steyvers,et al.  A Psychological Model for Aggregating Judgments of Magnitude , 2011, SBP.

[16]  Martyn Plummer,et al.  JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling , 2003 .

[17]  William H. Batchelder,et al.  Cultural Consensus Theory: Aggregating Continuous Responses in a Finite Interval , 2010, SBP.

[18]  William H. Batchelder,et al.  New Results in Test Theory Without an Answer Key , 1989 .

[19]  Craig R. Fox,et al.  Ambiguity Aversion and Comparative Ignorance , 1995 .

[20]  Georg Rasch,et al.  Probabilistic Models for Some Intelligence and Attainment Tests , 1981, The SAGE Encyclopedia of Research Design.

[21]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[22]  W. Batchelder,et al.  Test theory without an answer key , 1988 .

[23]  M. Lee How cognitive modeling can benefit from hierarchical Bayesian models. , 2011 .

[24]  A. Comrey The Minimum Residual Method of Factor Analysis , 1962 .

[25]  William H. Batchelder,et al.  Cultural Consensus Theory: Comparing different concepts of cultural truth , 2012 .

[26]  W. Batchelder,et al.  Culture as Consensus: A Theory of Culture and Informant Accuracy , 1986 .

[27]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .