A Probabilistic Latent Class Model for Assessing Inter-Judge Reliability.

Increasingly behavioral researchers are soliciting cognitive responses in addition to standard attitudinal measures when attempting to assess the effects of persuasive communications. The coding of the elicited cognitive responses generally involves some sort of categorization, typically undertaken by independent judges, and the quality of the data is, to a large degree, evaluated in terms of some reliability coefficient which reflects the extent to which the independent judges agreed. The purpose of this paper is to present and illustrate a probabilistic model for assessing inter-judge reliability. The proposed probabilistic model allows one to (a) use formal test statistics to evaluate the extent and character of inter-judge reliability, (b) estimate the assignment error rates and their standard errors, and (c) test for simultaneous agreement for more than two judges. The probabilistic model is operationalized in terms of restricted latent class models.

[1]  Charles H. Proctor,et al.  A probabilistic formulation and statistical analysis of guttman scaling , 1970 .

[2]  J. R. Landis,et al.  A one-way components of variance model for categorical data , 1977 .

[3]  Peter Wright,et al.  Factors Affecting Cognitive Resistance to Advertising , 1975 .

[4]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[5]  S. Chaiken,et al.  Communication modality as a determinant of persuasion: The role of communicator salience. , 1983 .

[6]  S. K. Mitchell.  Interobserver agreement, reliability, and generalizability of data collected in observational studies. , 1979 .

[7]  K. Kaye,et al.  Estimating False Alarms and Missed Events From Interobserver Agreement: A Rationale , 1980 .

[8]  P. Wright.  The Cognitive Processes Mediating Acceptance of Advertising , 1973 .

[9]  C. Mitchell Dayton,et al.  A Two-Stage Conditional Estimation Procedure for Unrestricted Latent Class Models , 1980 .

[10]  C. Mitchell Dayton,et al.  A scaling model with response errors and intrinsically unscalable respondents , 1980 .

[11]  Richard J. Lutz,et al.  Integrating Cognitive Structure and Cognitive Response Approaches to Monitoring Communications Effects , 1977 .

[12]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[13]  William R. Dillon,et al.  Causal Analysis and Latent Class Models: An Application to a Communication Hierarchy of Effects Model , 1982 .

[14]  Jacob Cohen,et al.  Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. , 1968 .

[15]  J. Fleiss.  Measuring nominal scale agreement among many raters. , 1971 .