Infinite looping problems that keep student programs from termi-nating may occur in many kinds of programming assignments. While non-terminating code is easier to diagnose interactively, it poses different concerns when software tests are being run auto-matically in batch. The common strategy of using a timeout to preemptively kill test runs that execute for too long has limita-tions, however. When one test case gets stuck in an infinite loop, forcible termination prevents any later test cases from running. Worse, when test results are buffered inside a test execution framework, forcible termination may prevent any information from being produced. Further, overly generous timeouts can de-lay the availability of results, and when tests are executed on a shared server, one non-terminating program can delay results for many people. This paper describes an alternative strategy that uses a fine-grained timeout on the execution of each individual test case in a test suite, and that adaptively adjusts this timeout dynamically based on the termination behavior of test cases com-pleted so far. By avoiding forcible termination of test runs, this approach allows all test cases an opportunity to run and produce results, even when infinite looping behaviors occur. Such fine-grained timeouts also result in faster completion of entire test runs when non-terminating code is present. Experimental results from applying this strategy to 4,214 student-written programs are dis-cussed, along with experiences from live deployment in the class-room where 8,926 non-termination events were detected over the course of one academic year.
[1]
Thomas A. Henzinger,et al.
Proving non-termination
,
2008,
POPL '08.
[2]
Koushik Sen,et al.
Looper: Lightweight Detection of Infinite Loops at Runtime
,
2009,
2009 IEEE/ACM International Conference on Automated Software Engineering.
[3]
D. S. Morris.
Automatic grading of student's programming assignments: an interactive process and suite of programs
,
2003,
33rd Annual Frontiers in Education, 2003. FIE 2003..
[4]
Michael Striewe,et al.
Using run time traces in automated programming tutoring
,
2011,
ITiCSE '11.
[5]
Georgios Evangelidis,et al.
WIPE: a programming environment for novices
,
2005,
ITiCSE '05.
[6]
David Hovemeyer,et al.
Software repository mining with Marmoset: an automated programming project snapshot and testing system
,
2005,
ACM SIGSOFT Softw. Eng. Notes.
[7]
Janet Verbyla,et al.
An Online Programming Assessment Tool
,
2003,
ACE.
[8]
David Jackson,et al.
Grading student programs using ASSYST
,
1997,
SIGCSE '97.
[9]
Philipp Rümmer,et al.
Non-termination Checking for Imperative Programs
,
2008,
TAP.
[10]
David Hovemeyer,et al.
Software repository mining with Marmoset
,
2005,
MSR.
[11]
Koushik Sen,et al.
CUTE: a concolic unit testing engine for C
,
2005,
ESEC/FSE-13.
[12]
Stephen H. Edwards.
Rethinking computer science education from a test-first perspective
,
2003,
OOPSLA '03.