A Framework for Assessment of Student Project Groups On-Line and Off-Line

Assessment of difficulties within group processes, especially through automatic means, is a problem of great interest to the broader CSCL community. Group difficulties can be revealed through interaction processes that occur during group work. Whether these patterns are encoded in speech recorded from face-to-face interactions or in text from on-line interactions, the language communication that flows between group members is an important key to understanding how better to support group functions and therefore be in a better position to design effective group learning environments. With the capability of monitoring and then influencing group processes when problems are detected, it is possible to intervene in order to facilitate the accomplishment of a higher quality product. In this chapter we address this research problem of monitoring group work processes in a context where project course instructors are making assessments of student group work. Thus, our purpose is to support those instructors in their task. We describe the mixed methods approach that we took, which combines both an interview study and a classroom study. Three research questions are answered: (1) What do instructors want to know about their student groups? (2) Is the desired information observable, and can it be reliably tracked by human annotators? (3) Can the desired information be automatically tracked using machine learning techniques to produce a summary report that instructors can use? Based on interviews with nine instructors, we identified five process assessment categories with subcategories at the group and individual level: namely, goal setting, group and individual progress, knowledge contribution, participation, and teamwork. We verified that these assessment categories can be reliably coded during group meetings with a reliability of r = 0.80 at the group level and r = 0.64 at the individual level using carefully constructed human assessment instruments. We present work in progress towards automation of this assessment framework.