A Framework for the Computer Based Assessment of Diagram Based Coursework

This research investigates the feasibility and usefulness of designing an authoring environment for diagram-based Computer Based Assessment (CBA). CBA systems present students with an environment that permits them to solve exercises. Upon completion, solutions to the exercises are marked automatically, and the results, along with instructional feedback, are returned to the students. CBA exercises are authored by educators who gain practical and pedagogic benefits from automating the assessment of their students. Until now, facilities to develop CBA have only been implemented for exercises where the answers are text-based. The deliverables described in this dissertation implement novel solutions to problems in three areas. The first area concerns the customisation of graphic editors to the learning material. The proposed solution is an object-oriented framework for visually specifying and authoring diagram editors. The second area presents techniques for describing the marking criteria of exercises. The proposed solution is a generic marking system designed to support the development of pluggable marking tools for the numerous and inherently different types of diagram domains. The third area is concerned with the integration of the two previous designs with the functions of a CBA system. For this reason provisions were made in the design and implementation of the CourseMaster CBA system. CourseMaster was implemented to improve and replace the Ceilidh CBA system. Ceilidh has been successfully used throughout academia for the assessment of coursework in computer programming. Evaluation results on diagram-based domains, such as circuit design and software design, indicate that the automation of the assessment of diagrams can be as effective and useful as that of programs. CourseMaster has been available within academia since 1999, providing support for the marking of programming coursework. The diagram authoring extension, entitled DATsys, was integrated into CourseMaster in May 2000. Together they provide a novel and realistic foundation towards authoring, running, and administering diagram-based CBA.

[1]  Ray M. Rottmann,et al.  Computer Grading As an Instructional Tool. , 1983 .

[2]  Oscar Nierstrasz,et al.  Design Guidelines for Tailorable Frameworks , 1997 .

[3]  Sushil Jajodia,et al.  On the Representation of Relational Structures by Entity-Relationship Diagrams , 1983, ER.

[4]  Erich Gamma,et al.  ET++—a portable, homogeneous class library and application framework , 1995 .

[5]  Ralph E. Johnson,et al.  Frameworks = (components + patterns) , 1997, CACM.

[6]  簡聰富,et al.  物件導向軟體之架構(Object-Oriented Software Construction)探討 , 1989 .

[7]  Sharam Hekmatpour.  Templa Graphica: A Generic Graphical Editor for the MacIntosh , 1991 .

[8]  Ralph E. Johnson,et al.  Creating Tools in HotDraw by Composition , 1994, TOOLS.

[9]  Joanna Bull.  Update on the National TLTP3 Project: The implementation and evaluation of computer-assisted assessment , 1999 .

[10]  Krzysztof Czarnecki,et al.  Generative Programming , 2001, ECOOP Workshops.

[11]  Jo Coldwell-Neilson,et al.  Automated assessment and marking of spreadsheet concepts , 1997, ACSE '97.

[12]  Michael Luck,et al.  Effective electronic marking for on-line assessment , 1998, ITiCSE '98.

[13]  John English,et al.  Experience with an automatically assessed course , 2000, ITiCSE '00.

[14]  Mark Minas,et al.  DiaGen: a generator for diagram editors providing direct manipulation and execution of diagrams , 1995, Proceedings of Symposium on Visual Languages.

[15]  Manfred Broy,et al.  State Transition Diagrams , 1996 .

[16]  Simson L. Garfinkel,et al.  Practical UNIX and Internet Security , 1996 .

[17]  Simson L. Garfinkel,et al.  Practical UNIX and internet security (2. ed.) , 1996 .

[18]  F. W. Day.  Computer Aided Software Engineering (CASE) , 1983, 20th Design Automation Conference Proceedings.

[19]  James Coplien,et al.  Advanced C++ Programming Styles and Idioms , 1991, Proceedings. Technology of Object-Oriented Languages and Systems, TOOLS 25 (Cat. No.97TB100239).

[20]  Athanasios Tsintsifas,et al.  The marking system for CourseMaster , 2002, ITiCSE '02.

[21]  B. Shneiderman,et al.  Flowchart techniques for structured programming , 1973, SIGP.

[22]  Volker Haarslev,et al.  GenEd-an editor with generic semantics for formal reasoning about visual notations , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[23]  Kathi Fisler,et al.  Exploiting the potential of diagrams in guiding hardware reasoning , 1996 .

[24]  Niklaus Wirth,et al.  Automatic grading programs , 1965, CACM.

[25]  B. Bloom.  Taxonomy of educational objectives , 1956 .

[26]  Iraj Hirmanpour,et al.  A student system development diagrammer , 1988, SIGCSE '88.

[27]  Charlie Daly,et al.  RoboProf and an introductory computer programming course , 1999, ITiCSE '99.

[28]  Peter J. Denning.  A debate on teaching computing science , 1989, CACM.

[29]  Stephen R. Schach.  Software engineering (2. ed.) , 1993 .

[30]  J. Christie.  Automated essay marking - for style and content , 1999 .

[31]  Mahesh H. Dodani,et al.  Pedagogical patterns—successes in teaching object technology: a workshop from OOPSLA '96 , 1996, SIGP.

[32]  David Wood,et al.  Using the REDEEM ITS authoring environment in naval training , 2001, Proceedings IEEE International Conference on Advanced Learning Technologies.

[33]  Ivar Jacobson,et al.  The unified modeling language reference manual , 2010 .

[34]  Billy B. L. Lim,et al.  DBTool: a graphical database design tool for an introductory database course , 1992, SIGCSE '92.

[35]  Ivan E. Sutherland,et al.  Sketchpad: a man-machine graphical communication system , 1899, AFIPS '63 (Spring).

[36]  David M. Arnow,et al.  On-line programming examinations using Web to teach , 1999, ITiCSE '99.

[37]  Margaret M. Burnett.  Visual object-oriented programming , 1993, OOPSLA '93.

[38]  J. Craig Cleaveland.  Program Generators with XML and Java , 2001 .

[39]  Ali Mili,et al.  Reusing Software: Issues and Research Directions , 1995, IEEE Trans. Software Eng..

[40]  Nicolas Van labeke.  Nicolas VanLabeke, Prise en compte de l'usager enseignant dans la conception des EIAO. Illustration dans Calques 3D. Thèse de l'université Henri Poincaré - Nancy I, 17 décembre 1999 , 1999 .

[41]  Evelyn Goldsmith.  Research into Illustration:An Approach and a Review , 1984 .

[42]  Denise M. Woit,et al.  Integrating technology into computer science examinations , 1998, SIGCSE '98.

[43]  Ian Douglas Sanders,et al.  A fundamentals-based curriculum for first year computer science , 2000, SIGCSE '00.

[44]  Krzysztof Czarnecki,et al.  Generative programming - methods, tools and applications , 2000 .

[45]  Arthur J. Riel,et al.  Object-Oriented Design Heuristics , 1996 .

[46]  Karl J. Lieberherr,et al.  Object-oriented design , 1996, CSUR.

[47]  Larry Rosenstein.  MacApp: first commercially successful framework , 1995 .

[48]  N. G. Miller,et al.  A Diagrammatic Formal System for Euclidean Geometry , 2001 .

[49]  E. Burton Swanson,et al.  Problems in application software maintenance , 1981, CACM.

[50]  Steve Benford,et al.  The Ceilidh system for the automatic grading of students on programming courses , 1995, ACM-SE 33.

[51]  Brad A. Myers,et al.  A brief history of human-computer interaction technology , 1998, INTR.

[52]  Martin Fowler,et al.  Refactoring - Improving the Design of Existing Code , 1999, Addison Wesley object technology series.

[53]  Antti-Pekka Tuovinen.  Towards a Framework for Developing Processors for Visual Languages , 1997 .

[54]  E. B. Page.  Computer Grading of Student Prose, Using Modern Concepts and Software , 1994 .

[55]  Colin Higgins,et al.  PRAM: prolog automatic marker , 1998, ITiCSE '98.

[56]  Lorin M. Hitt,et al.  Beyond the productivity paradox , 1998, CACM.

[57]  Adele Goldberg,et al.  Personal Dynamic Media , 1977, Computer.

[58]  Neff Walker,et al.  A classification of visual representations , 1994, CACM.

[59]  Konstantinos Tourlas,et al.  Diagrammatic representations in domain-specific languages , 2002 .

[60]  Zenon Kulpa,et al.  DIAGRAMMATIC REPRESENTATION AND REASONING , 1994 .

[61]  Matthew C. Clarke.  Possible models diagrams: a visual alternative to truth tables , 1993, SIGCSE '93.

[62]  Peter Coad,et al.  Object-oriented patterns , 1992, CACM.

[63]  Vincent J. Hardy.  Java 2D API Graphics , 1999 .

[64]  John M. Jeffrey.  Using Petri nets to introduce operating system concepts , 1991, SIGCSE '91.

[65]  Maurice H. Halstead,et al.  Elements of software science , 1977 .

[66]  Gerhard Viehstaedt.  A generation for diagram editors , 1995 .

[67]  James D. Hollan,et al.  Strategic directions in human-computer interaction , 1996, CSUR.

[68]  D. Whittington,et al.  Approaches to the computerized assessment of free text responses , 1999 .

[69]  Ben Shneiderman,et al.  Direct Manipulation: A Step Beyond Programming Languages , 1983, Computer.

[70]  David Jackson,et al.  Grading student programs using ASSYST , 1997, SIGCSE '97.

[71]  Alan C. Kay.  The Computer Revolution Hasn't Happened Yet , 2002, Mensch & Computer.

[72]  Murray Silverstein,et al.  A Pattern Language , 1977 .

[73]  Robert L. Myers.  Computerized Grading of Freshman Chemistry Laboratory Experiments. , 1986 .

[74]  Sinan Si Alhir.  UML in a Nutshell , 1998 .

[75]  Lauri Malmi,et al.  Algorithm simulation with automatic assessment , 2000, ITiCSE '00.

[76]  Roel Wieringa,et al.  A survey of structured and object-oriented software specification methods and techniques , 1998, CSUR.

[77]  Alan F. Blackwell,et al.  Metaphor in diagrams , 1998 .

[78]  Erik Hilsdale.  Aspect-Oriented Programming with Aspectj , 2001, TOOLS.

[79]  Peter Sommerlad,et al.  Pattern-Oriented Software Architecture: A System of Patterns: John Wiley & Sons , 1987 .

[80]  Martin Fowler,et al.  Analysis patterns - reusable object models , 1996, Addison-Wesley series in object-oriented software engineering.

[81]  Barry W. Boehm,et al.  A spiral model of software development and enhancement , 1986, Computer.

[82]  William F. Opdyke,et al.  Refactoring object-oriented frameworks , 1992 .

[83]  Paul R. Calder,et al.  Composing user interfaces with InterViews , 1989, Computer.

[84]  Brian R. Gaines,et al.  Concept maps as hypermedia components , 1995, Int. J. Hum. Comput. Stud..

[85]  Karl Lieberherr,et al.  Adaptive Object-Oriented Software , 1995 .

[86]  Robert J. Rossheim.  Report on proposed American standard flowchart symbols for information processing , 1963, CACM.

[87]  Peter G. Neumann,et al.  Risks of e-education , 1998, CACM.

[88]  David A. Duce,et al.  Scalable Vector Graphics SVG 1.0 Specification , 2000 .

[89]  Brad A. Myers,et al.  Past, Present and Future of User Interface Software Tools , 2000, TCHI.

[90]  Steve Benford,et al.  Early experiences of computer-aided assessment and administration when teaching computer programming , 1993 .

[91]  Sallie M. Henry,et al.  The evaluation of software systems' structure using quantitative software metrics , 1984, Softw. Pract. Exp..

[92]  Cristina V. Lopes,et al.  Aspect-oriented programming , 1999, ECOOP Workshops.

[93]  John A. Taylor,et al.  Constructed‐response, computer‐graded homework , 1976 .

[94]  Ralph Johnson.  Documenting frameworks using patterns , 1992, OOPSLA 1992.

[95]  Erich Gamma,et al.  ET++ - An Object-Oriented Application Framework in C++ , 1988, OOPSLA.

[96]  Mark A. Linton,et al.  Generalized graphical object editing , 1990 .

[97]  Bill Venners,et al.  Inside the Java Virtual Machine , 1997 .

[98]  James Martin,et al.  Object-oriented methods : a foundation , 1995 .

[99]  J. Christie.  Automated Essay Marking – for both Style and Content , 1999 .

[100]  Steven Holzner.  Inside XML , 2000 .

[101]  Doug Lea,et al.  Christopher Alexander: an introduction for object-oriented designers , 1998, SOEN.

[102]  Rebecca Wirfs-Brock,et al.  Designing object-oriented software , 1990 .

[103]  Chris Power.  Designer—a logic diagram design tool , 1999, ITiCSE '99.

[104]  Ivar Jacobson,et al.  Object-oriented software engineering - a use case driven approach , 1993, TOOLS.

[105]  Ole Lehrmann Madsen.  Strategic research directions in object-oriented programming , 1996, CSUR.

[106]  Bernard J. Jansen,et al.  Digital video in education , 1999, SIGCSE '99.

[107]  Herbert A. Simon,et al.  Why a Diagram is (Sometimes) Worth Ten Thousand Words , 1987, Cogn. Sci..

[108]  Ivan Herman,et al.  Graph Visualization and Navigation in Information Visualization: A Survey , 2000, IEEE Trans. Vis. Comput. Graph..

[109]  Terry Winograd,et al.  Beyond programming languages , 1979, CACM.

[110]  Tim J. Brailsford,et al.  Towards a Framework to Effective Web-based Distributed Learning , 2001, WWW Posters.

[111]  Dana L. G. Anthony,et al.  Patterns for classroom education , 1996 .

[112]  Bruce Eckel.  Thinking in Java , 1998 .

[113]  Jan Rune Holmevik.  Compiling SIMULA: a historical study of technological genesis , 1994, IEEE Annals of the History of Computing.

[114]  James Martin,et al.  Diagramming techniques for analysts and programmers , 1984 .

[115]  Grady Booch,et al.  Object-Oriented Design with Applications , 1990 .

[116]  Ted G. Lewis.  Object oriented application frameworks , 1995 .

[117]  James O. Coplien,et al.  A development process generative pattern language , 1994 .

[118]  Ellen Yi-Luen Do,et al.  Thinking with Diagrams in Architectural Design , 2001, Artificial Intelligence Review.

[119]  Edward Yourdon,et al.  Object-oriented analysis , 2012 .

[120]  Dennis Tsichritzis,et al.  Reengineering the university , 1999, Commun. ACM.

[121]  Jon A. Preston,et al.  A system for improving distance and large-scale classes , 1998, ITiCSE '98.

[122]  Tony Greening.  Pedagogically sound responses to economic rationalism , 2000, SIGCSE '00.

[123]  Yuri Engelhardt,et al.  A Meta-Taxonomy for Diagram Research , 2002, Diagrammatic Representation and Reasoning.

[124]  Charles F. Goldfarb,et al.  SGML handbook , 1990 .

[125]  David Harel,et al.  On visual formalisms , 1988, CACM.

[126]  Patricia Bylebyl Van Verth.  A system for automatically grading program quality (metrics, software metrics, program complexity) , 1985 .

[127]  Joseph J. Moder,et al.  Project Management with CPM and PERT , 1964 .

[128]  Alan Borning,et al.  ThingLab: a constraint-oriented simulation laboratory , 1979 .

[129]  Elliott J. Chikofsky,et al.  Computer-Aided Software Engineering (Case) , 1994 .

[130]  C. Petri.  Kommunikation mit Automaten , 1962 .

[131]  Fintan Culwin.  Web hosted assessment—possibilities and policy , 1998, ITiCSE '98.

[132]  John Grundy,et al.  Vixels, createthroughs, dragthroughs and attachment regions in BuildByWire , 1998, Proceedings 1998 Australasian Computer Human Interaction Conference. OzCHI'98 (Cat. No.98EX234).

[133]  Christopher Alexander,et al.  The Timeless Way of Building , 1979 .

[134]  Mike A. Lockyer,et al.  An automated student diagram assessment system , 1998, ITiCSE '98.

[135]  Wendy Doubé.  Distance teaching workloads , 2000, SIGCSE '00.

[136]  Zarina Shukur,et al.  The automatic assessment of Z specifications , 1997, ITiCSE-WGR '97.

[137]  James Dalziel,et al.  Next generation computer assisted assessment software: the design and implementation of WebMCQ , 1999 .

[138]  T. Kuhn,et al.  The Structure of Scientific Revolutions. , 1964 .

[139]  Ralph E. Johnson,et al.  Surveying current research in object-oriented design , 1990, CACM.

[140]  David Jackson.  A semi-automated approach to online assessment , 2000, ITiCSE '00.

[141]  Steven L. Tanimoto,et al.  VIVA: A visual language for image processing , 1990, J. Vis. Lang. Comput..

[142]  Duane Buck,et al.  Design early considered harmful: graduated exposure to complexity and structure based on levels of cognitive development , 2000, SIGCSE '00.

[143]  Ann L. Brown,et al.  How people learn: Brain, mind, experience, and school. , 1999 .

[144]  Terry Scott,et al.  Automated grading assistance for student programs , 1994, SIGCSE '94.

[145]  John Vlissides,et al.  Pattern hatching: design patterns applied , 1998 .

[146]  Philipp Ackermann,et al.  Developing object-oriented multimedia software : based on the MET++ application framework , 1996 .

[147]  Andrew Hunter,et al.  MetaBuilder: The Diagrammer's Diagrammer , 2000, Diagrams.

[148]  Denise M. Woit,et al.  Providing mark-up and feedback to students with online marking , 1999, SIGCSE '99.

[149]  Nanette J. Eaton.  Microsoft Visio Version 2002 Inside Out , 2001 .

[150]  Ernest C. Ackermann,et al.  Computer aided program design experiments: diagrammatic versus textual material , 1989, SIGCSE '89.

[151]  Athanasios Tsintsifas,et al.  The CourseMaster CBA System , 2001 .

[152]  Philipp Ackermann,et al.  Combining 2D User Interface Components and Interactive 3D Graphics in ET++ , 1993, TOOLS.

[153]  Krishnan Rangarajan,et al.  Product quality framework: a vehicle for focusing on product quality goals , 2001, SOEN.

[154]  Robert Williams.  Automated essay grading: an evaluation of four conceptual models , 2001 .

[155]  John D. McGregor,et al.  Understanding object-oriented: a unifying paradigm , 1990, CACM.

[156]  Thomas J. Mowbray,et al.  AntiPatterns: Refactoring Software, Architectures, and Projects in Crisis , 1998 .

[157]  David A. Wilson,et al.  C++ Programming With Macapp , 1990 .

[158]  Brad A. Myers.  A new model for handling input , 1990, TOIS.

[159]  Richard Gisselquist.  Engineering in software , 1998, CACM.

[160]  Andre Vladimirescu,et al.  The Spice Book , 1994 .

[161]  James O. Coplien.  A Pattern Language for Writers’ Workshops , 1997 .

[162]  Philip Machanick.  Experience of Applying Bloom’s Taxonomy in Three Courses , 2000 .

[163]  Ralph Johnson,et al.  design patterns elements of reusable object oriented software , 2019 .

[164]  Nicolas van Labeke.  Prise en compte de l'usager enseignant dans la conception des EIAO. Illustration dans calques 3D , 1999 .

[165]  Russell L. Shackelford,et al.  Using software to solve problems in large computing courses , 1998, SIGCSE '98.

[166]  Alan Kay,et al.  The reactive engine , 1969 .

[167]  Rajat P. Garg,et al.  JSPICE: a component‐based distributed Java front‐end for SPICE , 1998 .

[168]  Wim Codenie,et al.  From custom applications to domain-specific frameworks , 1997, CACM.

[169]  Kurt Nørmark.  A suite of WWW-based tools for advanced course management , 2000, ITiCSE '00.

[170]  Edward A. Fox,et al.  Online evaluation in WWW-based courseware , 1997, SIGCSE '97.

[171]  James O. Coplien,et al.  A generative development-process pattern language , 1995 .

[172]  Enrique Ivan Oviedo.  Control flow, data flow and program complexity , 1984 .

[173]  Ian Sommerville.  Software engineering, 5th Edition , 1996 .