Assessing QuADEM: preliminary notes on a new method for evaluating online language learning courseware

In this article, we set out to assess QuADEM (Quality Assessment of Digital Educational Material), one of the latest methods for evaluating online language learning courseware. What is special about QuADEM is that the evaluation is based on observing the actual usage of the online courseware and that, from a checklist of 12 different components, the evaluator is free to pick and choose one or more. In particular, we focus on the QuADEM evaluation of a module of the digital environment Deutsch-Uni Online (DUO) that aims at preparing B1/B2 students for a study semester in Germany. DUO is meant for self-study supported by an online tutor. For our assessment, we observed two respondents during their activities in the online learning module, using think-aloud protocol, video registration, and keystroke logging, and we conducted semistructured postintervention interviews with them. Zooming in on usability, we found that this QuADEM component lacks assessment criteria regarding feedback and task design, both of which turned out to play an important motivational role in our assessment. While both could be added to the QuADEM usability dimension under the denomination “didactic usability,” we suggest that it might be worth reconsidering QuADEM's pick-and-choose approach.

[1]  Lesley Shield,et al.  Usability and Pedagogical Design: are Language Learning Websites Special? , 2004 .

[2]  Martin Oliver,et al.  An introduction to the Evaluation of Learning Technology , 2000, J. Educ. Technol. Soc..

[3]  Larry Ambrose,et al.  The power of feedback. , 2002, Healthcare executive.

[4]  Ann Barnes,et al.  Beyond the “wow” factor—Evaluating multimedia language learning software from a pedagogical viewpoint , 1998 .

[5]  Steve Cushion,et al.  Evaluation of a Web-based language learning environment: the importance of a user-centred design approach for CALL , 2001, ReCALL.

[6]  Bernard Susser.  A defense of checklists for courseware evaluation , 2001, ReCALL.

[7]  Priscilla Garrido Comer,et al.  A Methodology for Software Evaluation , 1998 .

[8]  Michael Levy,et al.  Computer applications in second language acquisition : Foundations for teaching , testing and research , 2009 .

[9]  Marion Niehoff.  Fremdsprachenlernen mit Multimedia: Anforderungen aus Sicht der NutzerInnen ; eine qualitative Untersuchung zum selbstorganisierten Lernen , 2003 .

[10]  Dominique Hémard,et al.  Enhancing online CALL design: The case for evaluation , 2004, ReCALL.

[11]  Philip Hubbard,et al.  An Integrated Framework for CALL Courseware Evaluation , 2013 .

[12]  Geert Jacobs,et al.  Quadem: manual for the quality assessment of digital educational material , 2010 .

[13]  Dorothea Thomé.  Kriterien zur Bewertung von Lernsoftware: mit einer exemplarischen Beurteilung von Deutsch-Lernprogrammen , 1989 .

[14]  Mike Levy,et al.  Call Dimensions: Options and Issues in Computer Assisted Language Learning (ESL & Applied Linguistics Professional Series) , 2006 .

[15]  Philip L. Hubbard,et al.  Evaluating CALL Software , 2005 .

[16]  M T O'Toole.  Evaluating instructional software. , 1993, Nursing dynamics.

[17]  Jozef Colpaert,et al.  Pedagogy-driven Design for Online Language Teaching and Learning , 2013 .

[18]  Robert A. Fischer.  How do we Know what Students are Actually Doing? Monitoring Students' Behavior in CALL , 2007 .

[19]  Wilfried Decoo.  AN APPLICATION OF DIDACTIC CRITERIA TO COURSEWARE EVALUATION , 2013 .