Using Property-Based Testing to Generate Feedback for C Programming Exercises

This paper reports on the use of property-based testing for providing feedback to C programming exercises. Test cases are generated automatically from properties specified in a test script; this not only makes it possible to conduct many tests (thus potentially find more mistakes), but also allows simplifying failed tests cases automatically. We present some experimental validation gathered for an introductory C programming course during the fall semester of 2018 that show significant positive correlations between getting feedback during the semester and the student’s results in the final exam. We also discuss some limitations regarding feedback for undefined behaviors in the C language. 2012 ACM Subject Classification Social and professional topics → Student assessment; Software and its engineering → Software testing and debugging; Software and its engineering → Domain specific languages

[1]  John Hughes,et al.  Testing telecoms software with quviq QuickCheck , 2006, ERLANG '06.

[2]  F. Yates.  Contingency Tables Involving Small Numbers and the χ2 Test , 1934 .

[3]  Gene Fisher,et al.  Making Formal Methods More Relevant to Software Engineering Students via Automated Test Generation , 2016, ITiCSE.

[4]  Lars-Åke Fredlund,et al.  Automatic Grading of Programming Exercises using Property-Based Testing , 2016, ITiCSE.

[5]  Steven S. Lumetta,et al.  Automated Feedback Framework for Introductory Programming Courses , 2016, ITiCSE.

[6]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2000, ICFP.

[7]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2011, SIGP.

[8]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[9]  John Hughes,et al.  Experiences with QuickCheck: Testing the Hard Stuff and Staying Sane , 2016, A List of Successes That Can Change the World.

[10]  Johan Jeuring,et al.  Towards a Systematic Review of Automated Feedback Generation for Programming Exercises , 2016, ITiCSE.

[11]  Raymond Lister,et al.  On the Number of Attempts Students Made on Some Online Programming Exercises During Semester and their Subsequent Performance on Final Exam Questions , 2016, ITiCSE.

[12]  Calton Pu,et al.  Buffer overflows: attacks and defenses for the vulnerability of the decade , 2000, Foundations of Intrusion Tolerant Systems, 2003 [Organically Assured and Survivable Information Systems].