How difficult are exams?: a framework for assessing the complexity of introductory programming exams

Student performance on examinations is influenced by the level of difficulty of the questions. It seems reasonable to propose therefore that assessment of the difficulty of exam questions could be used to gauge the level of skills and knowledge expected at the end of a course. This paper reports the results of a study investigating the difficulty of exam questions using a subjective assessment of difficulty and a purpose-built exam question complexity classification scheme. The scheme, devised for exams in introductory programming courses, assesses the complexity of each question using six measures: external domain references, explicitness, linguistic complexity, conceptual complexity, length of code involved in the question and/or answer, and intellectual complexity (Bloom level). We apply the scheme to 20 introductory programming exam papers from five countries, and find substantial variation across the exams for all measures. Most exams include a mix of questions of low, medium, and high difficulty, although seven of the 20 have no questions of high difficulty. All of the complexity measures correlate with assessment of difficulty, indicating that the difficulty of an exam question relates to each of these more specific measures. We discuss the implications of these findings for the development of measures to assess learning standards in programming courses.

[1]  Robert McCartney,et al.  Making sense of data structures exams , 2010, ICER '10.

[2]  Simon,et al.  Assignment and sequence: why some students can't recognise a simple swap , 2011, Koli Calling.

[3]  Rainer Bromme,et al.  Task Complexity, Epistemological Beliefs and Metacognitive Calibration: An Exploratory Study , 2006 .

[4]  Nell B. Dale,et al.  Most difficult topics in CS1: results of an online survey of educators , 2006, SGCS.

[5]  Angela Carbone,et al.  Reliably classifying novice programmer exam responses using the SOLO taxonomy , 2008 .

[6]  D. Campbell.  Task Complexity: A Review and Analysis , 1988 .

[7]  Judy Kay,et al.  Toward a shared understanding of competency in programming: an invitation to the BABELnot project , 2012, ACE 2012.

[8]  Robert McCartney,et al.  Applying data structures in exams , 2011, SIGCSE.

[9]  Per Øivind Braarud,et al.  Subjective Task Complexity and Subjective Workload: Criterion Validity for Complex Team Tasks , 2001 .

[10]  Errol Thompson,et al.  Bloom's taxonomy for CS assessment , 2008, ACE '08.

[11]  Benjamin S. Bloom,et al.  Taxonomy of Educational Objectives: The Classification of Educational Goals. , 1957 .

[12]  Daryl J. D'Souza,et al.  Can computing academics assess the difficulty of programming examination questions? , 2012, Koli Calling.

[13]  Eddie Blass.  The Australian Academic Profession in Transition , 2013 .

[14]  M. Banerjee,et al.  Beyond kappa: A review of interrater agreement measures , 1999 .

[15]  John Sweller,et al.  Cognitive Load During Problem Solving: Effects on Learning , 1988, Cogn. Sci..

[16]  James Skene,et al.  Introductory programming: examining the exams , 2012, ACE 2012.

[17]  Thorvald Haerem,et al.  The influence of degree of expertise and objective task complexity on perceived task complexity and performance. , 2007, The Journal of applied psychology.

[18]  Carsten Schulte,et al.  What do teachers teach in introductory programming? , 2006, ICER '06.

[19]  Daryl J. D'Souza,et al.  Exploring programming assessment instruments: a classification scheme for examination questions , 2011, ICER.

[20]  Raymond Lister,et al.  Relationships between reading, tracing and writing skills in introductory programming , 2008, ICER '08.

[21]  Judy Kay,et al.  Coming to terms with Bloom: an online tutorial for teachers of programming fundamentals , 2012, ACE 2012.

[22]  Angela Carbone,et al.  Going SOLO to assess novice programmers , 2008, ITiCSE.

[23]  Daniel Zingaro,et al.  Reviewing CS1 exam question content , 2011, SIGCSE '11.

[24]  Sophie Arkoudis,et al.  The Australian academic profession in transition , 2011 .

[25]  S. Mennin.  Small-Group Problem-Based Learning as a Complex Adaptive System. , 2007 .