Implementation of computer assisted assessment: lessons from the literature

This paper draws attention to literature surrounding the subject of computer-assisted assessment (CAA). A brief overview of traditional methods of assessment is presented, highlighting areas of concern in existing techniques. CAA is then defined, and instances of its introduction in various educational spheres are identified, with the main focus of the paper concerning the implementation of CAA. Through referenced articles, evidence is offered to inform practitioners, and direct further research into CAA from a technological and pedagogical perspective. This includes issues relating to interoperability of questions, security, test construction and testing higher cognitive skills. The paper concludes by suggesting that an institutional strategy for CAA coupled with staff development in test construction for a CAA environment can increase the chances of successful implementation.

[1]  Douglas M. Walker,et al.  A Note on Multiple Choice Exams, with Respect to Students' Risk Preference and Confidence , 2001 .

[2]  Niall Sclater,et al.  User requirements of the "ultimate" online assessment engine , 2003, Comput. Educ..

[3]  P. Cox,et al.  Computer Aided Assessment. , 1979 .

[4]  Jerrell C. Cassady,et al.  Cognitive Test Anxiety and Academic Performance , 2002 .

[5]  Nick Reid,et al.  Computer assisted assessment: Designing online quiz questions to assess a range of cognitive skills , 2002 .

[6]  M. Paxton.  A Linguistic Perspective on Multiple Choice Questioning , 2000 .

[7]  Glenn K. Baggott,et al.  Focus Group. TRIADS experiences and developments. A panel discussion. , 2002 .

[8]  Ray Harper.  Allowing for guessing and for the expectations from the learning outcomes in computer-based assessments , 2002 .

[9]  Hugh C. Davis,et al.  Rethinking assessment through learning technologies , 2002, ASCILITE.

[10]  Emma Duke-Williams,et al.  Using computer-aided assessment to test higher level learning outcomes , 2001 .

[11]  Joanna Bull,et al.  Cost-effectiveness analysis of computer-based assessment , 2003 .

[12]  Roy B. Clariana,et al.  Paper-based versus computer-based assessment: key factors associated with the test mode effect , 2002, Br. J. Educ. Technol..

[13]  B. Crisp,et al.  Assessment methods in social work education: A review of the literature , 2002 .

[14]  Benjamin S. Bloom,et al.  Taxonomy of Educational Objectives: The Classification of Educational Goals. , 1957 .

[15]  Tadahiko Fukuda,et al.  Glare from monitors measured with subjective scales and eye movements , 1999 .

[16]  G. Sim,et al.  Strategies for large-scale assessment: an institutional analysis of research and practice in a virtual university , 2003 .

[17]  T. Haladyna.  Writing Test Items to Evaluate Higher Order Thinking , 1996 .

[18]  Gráinne Conole,et al.  CAA in UK HEIs – the state of the art? , 2003 .

[19]  R. Bennett,et al.  USING MULTIMEDIA IN LARGE‐SCALE COMPUTER‐BASED TESTING PROGRAMS , 1997 .

[20]  Gwen Nugent,et al.  On-line Multimedia Assessment for K-4 Students , 2003 .

[21]  Blaine A. Price,et al.  Remote electronic examinations: student experiences , 2002, Br. J. Educ. Technol..

[22]  Min Liu,et al.  Exploring the use of multimedia examination formats in undergraduate teaching: results from the fielding testing , 2001, Comput. Hum. Behav..

[23]  Robert Frohlich.  Keeping the wolves from the doors… wolves in sheep’s clothing, that is , 2000 .

[24]  Benjamin S. Bloom,et al.  Handbook On Formative and Summative Evaluation of Student Learning , 1971 .

[25]  Stephen Hinde.  Careless about privacy , 2003, Comput. Secur..

[26]  Chris Sealey,et al.  ‘At the Coal Face’ : experiences of computer-based exams , 2003 .

[27]  Eva Heinrich,et al.  Online Marking of Essay-type Assignments , 2003 .

[28]  Stephen Mason.  Electronic Security is a Continuous Process , 2003 .

[29]  A. Stevenson,et al.  Integrating CAA within the University of Ulster , 2002 .

[30]  R. Burton,et al.  Statistical Modelling of Multiple-Choice and True/False Tests: Ways of Considering, and of Reducing, the Uncertainties Attributable To Guessing. , 1999 .

[31]  Richard A. Bacon.  Assessing the use of a new QTI assessment tool within Physics , 2003 .

[32]  P. Knight,et al.  Does Grading Method Influence Honours Degree Classification? , 2002 .

[33]  Robert S. Tannenbaum,et al.  Theoretical foundations of multimedia , 1998, UBIQ.

[34]  Peter Twining,et al.  Assessing activity-based learning for a networked course , 2002, Br. J. Educ. Technol..

[35]  Lawrie Phipps,et al.  Computer Assisted Assessment and Disabilities , 2001 .

[36]  P. Davies.  “There’s no Confidence in Multiple-Choice Testing, ……” , 2002 .

[37]  Joanna Bull,et al.  Computer-assisted assessment , 2004 .

[38]  David O'Hare,et al.  Empirical prediction of the measurement scale and base Level ‘Guess factor’ for advanced computer-based assessments , 2002 .

[39]  A. Baranchik,et al.  Correcting grade deflation caused by multiple-choice scoring , 2000 .

[40]  Chris Ricketts,et al.  What factors affect student opinions of computer-assisted assessment? , 2002 .

[41]  Win Hornby,et al.  Assessing Using Grade-related Criteria: A single currency for universities? , 2003 .

[42]  Kenneth C. Cox,et al.  The use of formative quizzes for deep learning , 1998 .

[43]  David O'Hare,et al.  Finding appropriate methods to assure quality Computer-Based Assessment development in UK Higher Education , 2003 .

[44]  Salvatore Valenti,et al.  Computer Based Assessment Systems Evaluation via the ISO9126 Quality Model , 2002, J. Inf. Technol. Educ..

[45]  M. J. Pollock,et al.  Evaluating the costs and benefits of changing to CAA , 2000 .

[46]  Elaine Chapman,et al.  Computerised adaptive testing , 2002, Br. J. Educ. Technol..

[47]  R. L. Ebel,et al.  Essentials of educational measurement , 1972 .

[48]  Michael McCabe,et al.  CAA scoring strategies for partial credit and confidence levels , 2003 .

[49]  N. Sabar,et al.  TOWARD PRINCIPLED PRACTICE IN EVALUATION: LEARNING FROM INSTRUCTORS' DILEMMAS IN EVALUATING GRADUATE STUDENTS , 2002 .

[50]  David Callear,et al.  Using computer-based tests for information science , 1997 .

[51]  Den Pain,et al.  WebCT and Online Assessment: The best thing since SOAP? , 2003, J. Educ. Technol. Soc..

[52]  Alan P. Boyle,et al.  The use of PGP to provide secure email delivery of CAA results , 2002 .

[53]  C. Steven,et al.  Computer-based assessment: quality assurance issues, the hub of the wheel , 2003 .

[54]  Richard E. Mayer,et al.  A taxonomy for computer-based assessment of problem solving , 2002, Comput. Hum. Behav..

[55]  Niall Sclater,et al.  Question and test interoperability: an update on national and international developments , 2000 .

[56]  D. Lloyd,et al.  The Introduction of Computer-Based Testing on an Engineering Technology Course. , 1996 .

[57]  Mrj Morgan.  MCQ: An interactive computer program for multiple-choice self-testing , 1979 .

[58]  Poul Erik Philipsen,et al.  Chat as Media in Exams , 2001 .

[59]  Helen Sian Ashton,et al.  Piloting Summative Web Assessment in Secondary Education , 2003 .

[60]  B. R. Dawson,et al.  Experiences of using computer assisted assessment in engineering mathematics , 2001, Comput. Educ..

[61]  Niall Sclater,et al.  Focus Group. Building a Compliance Audit for BS7988 "Code of Practice for the Use of Information Technology (IT) for the Delivery of Assessments , 2002 .

[62]  Paul Gathercoal.  Principles of Assessment , 1995 .

[63]  Richard F. Burton,et al.  Quantifying the Effects of Chance in Multiple Choice and True/False Tests: Question selection and guessing of answers , 2001 .

[64]  Su White,et al.  Creating large-scale test banks: a briefing for participative discussion of issues and agendas , 2000 .

[65]  J. Seale,et al.  Access all areas: disability, technology and learning , 2002 .

[66]  Benjamin S. Bloom,et al.  A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives , 2000 .

[67]  Ros L O'Leary,et al.  Wading through treacle: CAA at the University of Bristol , 2001 .

[68]  Michael Luck,et al.  A Secure On-line Submission System , 1999, Softw. Pract. Exp..

[69]  Mary C. Dyson,et al.  The legibility of screen formats: are three columns better than one? , 1997, Comput. Graph..

[70]  John Kleeman,et al.  A practical look at delivering assessments to BS 7988 recommendations , 2002 .

[71]  Joanna Bull,et al.  Quality assurance of computer‐assisted assessment: practical and strategic issues , 2000 .

[72]  Simon Ball,et al.  Constructing accessible CBA: minor works or major renovations? , 2003 .

[73]  J. S. Paterson.  Linking on-line assessment in mathematics to cognitive skills , 2002 .

[74]  R. Harper.  Correcting computer-based assessments for guessing , 2003, J. Comput. Assist. Learn..

[75]  Martin Chodorow,et al.  Stumping e-rater: challenging the validity of automated essay scoring , 2002, Comput. Hum. Behav..

[76]  J. Christie.  Automated Essay Marking – for both Style and Content , 1999 .

[77]  Derek Stephens,et al.  Computer‐assisted Assessment: suggested guidelines for an institutional strategy , 1998 .

[78]  Jefferson M. Koonce,et al.  Comprehension and workload differences for VDT and paper-based reading , 2001 .

[79]  Sarah Maughan,et al.  On-line formative assessment item banking and learning support , 2001 .

[80]  Martin Bush.  ALTERNATIVE MARKING SCHEMES FOR ON-LINE MULTIPLE- CHOICE TESTS , 1999 .

[81]  R. Dowsing.  FLEXIBILITY AND THE TECHNOLOGY OF COMPUTER-AIDED ASSESSMENT , 1998 .

[82]  John Waldron,et al.  Introductory programming, problem solving and computer assisted assessment , 2002 .

[83]  C. Ricketts,et al.  Improving Student Performance Through Computer-based Assessment: Insights from recent research , 2002 .

[84]  Joanna Bull,et al.  A Blueprint for Computer-Assisted Assessment , 2003 .

[85]  Pat Jefferies,et al.  Computer Aided Assessment using WebCT , 2000 .

[86]  Eleanor Hawe,et al.  'It's Pretty Difficult to Fail': The reluctance of lecturers to award a failing grade , 2003 .

[87]  Michael Luck,et al.  A secure on-line submission system , 1999 .

[88]  Claudia Leacock,et al.  Automated evaluation of essays and short answers , 2001 .

[89]  Abdoljavad Jafarpur,et al.  Is the test constructor a facet? , 2003 .

[90]  Jon H Sims Williams,et al.  Computer-Assisted Assessment in Higher Education , 1999 .