Using a New Inter-rater Reliability Statistic

!      "!#    $      %       &'(     (           %  ( (  &( '(  %   )% !   "'#   ")% *++#   $  (    $! &,  & -   ! ( '          %' !     (    $      &'(  %  (   ' .        &/.     (  $  $  &'    (   $  %   0 &  ( !       .             (! &&  % $     &'    $             %      %    &1      (               2     (   $ ((    (    $ &      %   $ 3,  $   & 3, (2     (     (  $ (        .   (  $    (  $ % $$(  &   (        ( $ 3, $   & (      2  %      (      '  .      &

[1]  Y H Chan,et al.  Biostatistics 104: correlational analysis. , 2003, Singapore medical journal.

[2]  D. Ross Jeffery,et al.  An exploratory study of process enactment as input to software process improvement , 2006, WoSQ '06.

[3]  J. H. Pollard,et al.  Introductory statistics with applications in general insurance , 1999 .

[4]  B. Thompson.  What Future Quantitative Social Science Research Could Look Like: Confidence Intervals for Effect Sizes , 2002 .

[5]  Susan T. Dumais,et al.  Data-driven approaches to information access , 2003, Cogn. Sci..

[6]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[7]  A. Feinstein,et al.  High agreement but low kappa: I. The problems of two paradoxes. , 1990, Journal of clinical epidemiology.

[8]  Dustin Hillard,et al.  Automated classification of congressional legislation , 2006, DG.O.

[9]  Preslav Nakov,et al.  Towards Deeper Understanding of the LSA Performance , 2003 .

[10]  Marian Petre,et al.  A Research Taxonomy for Latent Semantic Analysis- Based Educational Applications , 2005 .

[11]  andy. luecking,et al.  Assessing Reliability on Annotations (1): Theoretical Considerations , 2005 .

[12]  Fabrizio Sebastiani,et al.  Machine learning in automated text categorization , 2001, CSUR.

[13]  Susan T. Dumais,et al.  Improving the retrieval of information from external sources , 1991 .

[14]  C. Aberson,et al.  Interpreting Null Results: Improving Presentation and Conclusions with Confidence Intervals , 2022 .

[15]  Arthur C. Graesser,et al.  Using Latent Semantic Analysis to Evaluate the Contributions of Students in AutoTutor , 2000, Interact. Learn. Environ..

[16]  Curtis F. Gerald.  Applied numerical analysis , 1970 .

[17]  Peter W. Foltz,et al.  Supporting Content-Based Feedback in On-Line Writing Evaluation with LSA , 2000, Interact. Learn. Environ..

[18]  Peter W. Foltz,et al.  The Measurement of Textual Coherence with Latent Semantic Analysis. , 1998 .

[19]  Derek Rowntree,et al.  Statistics without tears : a primer for non-mathematicians , 1982 .

[20]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[21]  K. Gwet.  Inter-Rater Reliability: Dependency on Trait Prevalence and Marginal Homogeneity , 2002 .

[22]  Christine P. Dancey,et al.  Statistics Without Maths for Psychology: Using Spss for Windows , 2005 .

[23]  Eileen Kintsch,et al.  Summary Street: Interactive Computer Support for Writing , 2004 .

[24]  Grace Hui Yang,et al.  Next steps in near-duplicate detection for eRulemaking , 2006, DG.O.

[25]  Peter W. Foltz.  Using latent semantic indexing for information filtering , 1990 .

[26]  Carlo Strapparava,et al.  Automatic Assessment of Students' Free-Text Answers Underpinned by the Combination of a BLEU-Inspired Algorithm and Latent Semantic Analysis , 2005, FLAIRS Conference.

[27]  Wc Taylor,et al.  Special Issue , 2000, International Journal of Recent Technology and Engineering.

[28]  Marian Petre,et al.  Measuring improvement in latent semantic analysis-based marking systems: using a computer to mark questions about HTML , 2007 .

[29]  Peter M. Wiemer-Hastings,et al.  How Latent is Latent Semantic Analysis? , 1999, IJCAI.

[30]  A. Graesser,et al.  Improving an intelligent tutor ’ s comprehension of students with Latent Semantic Analysis ∗ , 1999 .

[31]  Barbara Di Eugenio,et al.  Squibs and Discussions: The Kappa Statistic: A Second Look , 2004, CL.

[32]  Kevin F. Spratt,et al.  Disagreement on Agreement: Two Alternative Agreement Coefficients , 2007 .

[33]  K. Gwet.  Kappa Statistic is not Satisfactory for Assessing the Extent of Agreement Between Raters , 2002 .