Pitfalls in Corpus Research

Abstract. This paper discusses some pitfalls in corpus research and suggests solutions on the basis of examples and computer simulations. We first address reliability problems in language transcriptions, agreement between transcribers, and how disagreements can be dealt with. We then show that the frequencies of occurrence obtained from a corpus cannot always be analyzed with the traditional χ2 test, as corpus data are often not sequentially independent and unit independent. Next, we stress the relevance of the power of statistical tests, and the sizes of statistically significant effects. Finally, we point out that a t-test based on log odds often provides a better alternative to a χ2 analysis based on frequency counts.

[1]  C. J. Burke,et al.  The use and misuse of the chi-square test. , 1949, Psychological bulletin.

[2]  C. A. Boneau,et al.  The effects of violations of assumptions underlying the test. , 1960, Psychological bulletin.

[3]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[4]  J. Gart,et al.  On the bias of various estimators of the logit and its variance with application to quantal bioassay. , 1967, Biometrika.

[5]  J. Fleiss.  Measuring nominal scale agreement among many raters. , 1971 .

[6]  H. T. Reynolds,et al.  The analysis of cross-classifications , 1977 .

[7]  W. W. Daniel.  Applied Nonparametric Statistics , 1979 .

[8]  Stephen E. Fienberg,et al.  The analysis of cross-classified categorical data , 1980 .

[9]  Patricia M. E. Altham.  Detecting Relationships between Categorical Variables observed over Time: a Problem of deflating a Chi-squared Statistic , 1979 .

[10]  David Sankoff,et al.  Time Warps, String Edits, and Macromolecules: The Theory and Practice of Sequence Comparison , 1983 .

[11]  K. Delucchi.  The use and misuse of chi-square: Lewis and Burke revisited. , 1983 .

[12]  A. Liebetrau.  Measures of association , 1983 .

[13]  H. Schouten,et al.  Statistical measurement of interobserver agreement [: Analysis of agreements and disagreements between observers] , 1985 .

[14]  P. V. Reenen.  Probleme der phonetischen Transkription , 1990 .

[15]  S. Shott,et al.  Nonparametric Statistics , 2018, The Encyclopedia of Archaeological Sciences.

[16]  Alan Agresti,et al.  Categorical Data Analysis , 1991, International Encyclopedia of Statistical Science.

[17]  T. Wickens,et al.  Multiway Contingency Tables Analysis for the Social Sciences , 1992 .

[18]  Ted Dunning,et al.  Accurate Methods for the Statistics of Surprise and Coincidence , 1993, CL.

[19]  C. Cucchiarini,et al.  Phonetic transcription: a methodological and empirical study , 1993 .

[20]  Thomas D. Wickens,et al.  Analysis of contingency tables with between-subjects variability. , 1993 .

[21]  Toni Rietveld,et al.  Statistical Techniques for the Study of Language and Language Behaviour , 1993 .

[22]  Jean Carletta,et al.  Assessing Agreement on Classification Tasks: The Kappa Statistic , 1996, CL.

[23]  C. Cucchiarini.  Assessing transcription agreement: Methodological aspects , 1996 .

[24]  Cecile T. L. Kuijpers,et al.  The Influence of Rhythmic Context on Schwa Epenthesis and Schwa Deletion in Dutch , 1998 .

[25]  M. Ernestus.  Voice Assimilation and Segment Reduction in Casual Dutch. A Corpus-based Study of the Phonology-phonetics Interface , 2000 .

[26]  A. Kilgarriff.  Comparing Corpora , 2001 .

[27]  H. Van de Velde,et al.  The devoicing of fricatives in a reading task , 2001 .

[28]  Petra Hendriks,et al.  Initial coordination and the Law of Coordination of Likes , 2001 .

[29]  Audra Dainora.  Does Intonational Meaning Come From Tones or Tunes ? Evidence Against a Compositional Approach , 2002 .

[30]  Robert Schreuder,et al.  Processing reduced word forms: The suffix restoration effect , 2004, Brain and Language.