FLOP: A User-Friendly System for Automated Program Assessment

Currently, automated systems for program submission and assessment play a central role in the teaching of programming. A number of such systems have been developed in the last two decades. However, their adoption in regular programming teaching presents an obstacle: the overhead work required for the design of each problem, for compilation of problem collections, and for mundane management tasks. An open challenge is to make these systems easier to use and to reduce to a minimum the number of management tasks. In this article we present the FLOP system, which was developed to satisfy this goal. The contribution of the article is twofold. On the one hand, we present the FLOP system itself and its user-friendly features. On the other hand, we present in detail the user-centered design process used to design and enhance the ease of use of FLOP. Several actions were undertaken to inquire users' concerns and needs, with a usability evaluation of FLOP conducted with students being the most fruitful action.

[1]  J. Ángel Velázquez-Iturbide,et al.  Student perception and usage of an automated programming assessment tool , 2014, Comput. Hum. Behav..

[2]  J. Carter,et al.  How shall we assess this? , 2003, ITiCSE-WGR '03.

[3]  Sebastian G. Elbaum,et al.  Bug Hunt: Making Early Software Testing Lessons Engaging and Affordable , 2007, 29th International Conference on Software Engineering (ICSE'07).

[4]  Christopher Douce,et al.  Automatic test-based assessment of programming: A review , 2005, JERC.

[5]  Petri Ihantola,et al.  Review of recent systems for automatic assessment of programming assignments , 2010, Koli Calling.

[6]  Katerina Georgouli,et al.  Integrating an Automatic Judge into an Open Source LMS , 2009 .

[7]  Enrique Martin-Martin,et al.  FLOP, a free laboratory of programming , 2012, Koli Calling.

[8]  Kirsti Ala-Mutka,et al.  A Survey of Automated Assessment Approaches for Programming Assignments , 2005, Comput. Sci. Educ..

[9]  Neil Brown,et al.  Repositories of teaching material and communities of use: nifty assignments and the greenroom , 2010, ICER '10.

[10]  Lauri Malmi,et al.  Experiences on automatically assessed algorithm simulation exercises with different resubmission policies , 2005, JERC.

[11]  José Paulo Leal,et al.  Mooshak: a Web‐based multi‐site programming contest system , 2003, Softw. Pract. Exp..

[12]  D. M. Ha,et al.  A gentle introduction , 2006 .

[13]  Kent L. Beck,et al.  Test-driven Development - by example , 2002, The Addison-Wesley signature series.

[14]  J. Ángel Velázquez-Iturbide,et al.  Experiences in Usability Evaluation of Educational Programming Tools , 2015 .

[15]  Ruth Martinez,et al.  Ejercicios De Programación: Creativos Y Recreativos Con C++ , 2002 .

[16]  Russell K. Schutt,et al.  Research Methods in Education , 2011 .

[17]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[18]  Heinz Ulrich Hoppe,et al.  Learning to Program with COALA, a Distributed Computer Assisted Environment , 2009, J. Univers. Comput. Sci..

[19]  David S. Janzen,et al.  Implications of integrating test-driven development into CS1/CS2 curricula , 2009, SIGCSE '09.

[20]  George Siemens,et al.  Learning analytics: envisioning a research discipline and a domain of practice , 2012, LAK.

[21]  Salvador Roura,et al.  Jutge.org: an educational programming judge , 2012, SIGCSE '12.

[22]  Diane Kelly,et al.  More testing should be taught , 2001, CACM.

[23]  Judithe Sheard,et al.  Tools for "contributing student learning" , 2010, ITiCSE-WGR '10.

[24]  J. Ángel Velázquez-Iturbide,et al.  EXercita. A System for Archiving and Publishing Programming Exercises , 2001, Computers and Education. Towards an Interconnected Society.

[25]  Nathan Griffiths,et al.  The boss online submission and assessment system , 2005, JERC.

[26]  Stephen H. Edwards,et al.  Using software testing to move students from trial-and-error to reflection-in-action , 2004, SIGCSE '04.

[27]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[28]  Genevieve Orr,et al.  Interactive program demonstration as a form of student program assessment , 2002 .

[29]  J. A. Salvato.  John wiley & sons. , 1994, Environmental science & technology.

[30]  Athanasios Tsintsifas,et al.  Automated assessment and experiences of teaching programming , 2005, JERC.

[31]  Lauri Malmi,et al.  Visual Algorithm Simulation Exercise System with Automatic Assessment: TRAKLA2 , 2004, Informatics Educ..