Using a Type Hierarchy to Characterize Reliability of Coding Schemas for Dialogue Moves

Problem An important issue in choosing a coding schema is reliability. Is the schema reliable , i.e. can the coding results be replicated by researchers others than those who have devised the schema? If so, then (arguably) the results derived from coding are more likely to reeect facts. Unfortunately, many schemas do not include useful measures of reliability. This problem has only recently been recognized and a proposed solution, the Kappa statistic ((6]), seems to have been generally accepted. The advantage of Kappa is that it normalizes pairwise agreement between coders with respect to the expected random agreement, i.e. the amount of agreement that the coders would reach if they annotated a dialogue by chance Since the theory of speech acts was conceived, there has been numerous proposals of which speech acts (dialogue moves) there are, how they are to be deened and how they are related to each other. In the last few years, several taxonomies in the form of coding schemas for dialogue moves (or speech acts) have been developed more-or-less independently and used for coding dialogues. HCRC have developed a schema with three complementary structural levels (move, game and transaction) for coding dialogue structure in the Map Task corpus. Linkk oping University (Ahrenberg, Dahlbb ack, JJ onsson) have coded a corpus of WOZ-dialogues using two diierent schemas-one very simple (henceforth referred to as LINLIN1) and one slightly more complex (LINLIN2). In connection with the TRAINS project 8], Traum ((9]) has developed ve complementary coding schemas for spoken dialogue structure: coherence, grounding, surface form, illocutionary function and argumen-tation structure. A part of the TRAINS corpus has been annotated using these schemes. Allwood 2] also gives an account of communication management which subsequently has been reened and speciied in two complementary coding schemas, Own Communication Management (OCM) and Interaction Management (IM). Finally , The DAMSL (Dialogue Act Markup in Several Layers) schema is the product of the Discourse Resource Initiative (DRI), consisting of researchers from several dialogue projects worldwide. The goal of DRI is to provide a standard for coding of dialogue acts, which if necessary can be augmented with further subdivisions of the given categories. There are signs indicating that the DAMSL schema may become a \standard" schema for dialogue move coding. Such a standard has several advantages, e.g. increased re-usability and comparability of research results and a common framework for research on dialogue moves. However, it is …