Comparison of oral examination and electronic examination using paired multiple-choice questions

The aim of the present research was to compare the use of multiple-choice questions (MCQs) as an examination method against the oral examination (OE) method. MCQs are widely used and their importance seems likely to grow, due to their inherent suitability for electronic assessment. However, MCQs are influenced by the tendency of examinees to guess answers, warranting research concerning scoring rules different from the simple positive-grades-only scores rule. Alternatively, OE is used in tertiary education, since it enables the assessment of intellectual capabilities and personal traits to a level not found in most other examination formats. However, the significant resource requirements of OE, especially in structured forms, might excessively strain the resources of academic institutions. In the present study, an MCQ test was given to examinees, in the framework of a computer-based learning system. The same examinees took also an OE possessing elements of structure, with three examiners concurrently and independently grading each of the examinees. In the MCQs examination a set of pairs of MCQs was composed. The MCQs in each pair were similar concerning the same topic, but this similarity was not evident for an examinee that did not possess adequate knowledge on the topic addressed in the questions of the pair. The scoring of the paired questions avoided the procedure of mixed-scoring, i.e., both positive and negative markings, while at the same time a pair-wise bonus/penalty scoring rule was adopted. The results of the ''paired'' MCQs examination, when using the pair-wise scoring rule, were statistically indistinguishable with the grades produced by the OE, when made to the same sample of students, on the same topics and with the same levels of difficult. Both the results of the paired MCQs examination, when using the pair-wise scoring rule, and the OE results differed significantly from those obtained by scoring the same MCQs using a positive-grades-only scoring rule that ignored the pairing of MCQs.

[1]  P. McCoubrie,et al.  Single best answer MCQs: a new format for the FRCR part 2a exam. , 2008, Clinical radiology.

[2]  Eduardo Guzmán,et al.  Self-assessment in a feasible, adaptive web-based testing system , 2005, IEEE Transactions on Education.

[3]  G. E. El Shallaly,et al.  Use of video‐projected structured clinical examination (ViPSCE) instead of the traditional oral (viva) examination in the assessment of final year medical students , 2003, Education for health.

[4]  Charalampos Stergiopoulos,et al.  Evaluating Electronic Examination Methods Applied to Students of Electronics. Effectiveness and Comparison to the Paper-and-Pencil Method. , 2006, IEEE International Conference on Sensor Networks, Ubiquitous, and Trustworthy Computing (SUTC'06).

[5]  Uri Zoller,et al.  Examination-type preferences of secondary school students and their teachers in the science disciplines , 1997 .

[6]  Cornelis A.W. Glas,et al.  Computerized adaptive testing : theory and practice , 2000 .

[7]  Pete Thomas.  The evaluation of electronic marking of examinations , 2003 .

[8]  David Finkel,et al.  ITiCSE 2003 : proceedings of the 8th annual SIGCSE Conference on Innovation and Technology in Computer Science Education, June 30-July 2, 2003, Thessaloniki, Greece , 2003 .

[9]  Brent Bridgeman,et al.  Essays and multiple-choice tests as predictors of college freshman GPA , 1991 .

[10]  Richard Freeman,et al.  Planning and Implementing Assessment , 1998 .

[11]  J. Shatzer,et al.  For Personal Use Only. Reproduce with Permission from the Lancet Publishing Group. Standard Setting Assessment of Clinical Competence Medical Education Quartet , 2022 .

[12]  Roye H. Crouse,et al.  Desensitization for students with different sources and experiences of test anxiety. , 1985 .

[13]  Uri Zoller,et al.  Interaction between Examination Type, Anxiety State, and Academic Achievement in College Science; an Action-Oriented Research. , 1989 .

[14]  D. TRIANTIS,et al.  A pilot application of electronic examination applied to students of electronic engineering : Preliminary results . , 2004 .

[15]  D. Budescu,et al.  Decision making under internal uncertainty: the case of multiple-choice tests with different scoring rules. , 2003, Acta psychologica.

[16]  Donald E. Hanna.  Higher Education in an Era of Digital Competition: Choices and Challenges. , 2000 .

[17]  Ivan Pedrosa,et al.  Part I: preparing first-year radiology residents and assessing their readiness for on-call responsibilities. , 2006, Academic radiology.

[18]  N. John Castellan.  Evaluating information technology in teaching and learning , 1993 .

[19]  Oded M. Flascher,et al.  Prospect theory analysis of guessing in multiple choice tests , 2002 .

[20]  R. Hambleton,et al.  Handbook of Modern Item Response Theory , 1997 .

[21]  Howard Wainer,et al.  HOW WELL CAN WE EQUATE TEST FORMS THAT ARE CONSTRUCTED BY EXAMINEES?1 , 1991 .

[22]  Charalampos Stergiopoulos,et al.  Comparison of examination methods based on multiple-choice questions and constructed-response questions using personal computers , 2010, Comput. Educ..

[23]  E. Ghatala,et al.  Sometimes Adults Miss the Main Ideas and Do Not Realize It: Confidence in Responses to Short-Answer and Multiple-Choice Comprehension Questions. , 1990 .

[24]  L. R. Evans,et al.  The reliability, validity, and taxonomic structure of the oral examination. , 1966, Journal of medical education.

[25]  A D Walmsley,et al.  Potential of information technology in dental education. , 2008, European journal of dental education : official journal of the Association for Dental Education in Europe.

[26]  Howard Wainer,et al.  How Well Can We Compare Scores on Test Forms That Are Constructed by Examinees Choice , 1994 .

[27]  F. Weinert,et al.  Metacognition, motivation, and understanding , 1987 .

[28]  Tom Buchanan,et al.  Online assessment: Desirable or dangerous? , 2002 .

[29]  Eduardo Guzmán,et al.  A Model for Student Knowledge Diagnosis Through Adaptive Testing , 2004, Intelligent Tutoring Systems.

[30]  Lynne P. Baldwin,et al.  Assessing multiple choice question (MCQ) tests - a mathematical perspective , 2007 .

[31]  D. Anastakis,et al.  The structured oral examination as a method for assessing surgical residents. , 1991, American journal of surgery.

[32]  G. Mellsop,et al.  The oral examination: a study of academic and non‐academic factors , 1993, Medical education.

[33]  S. Embretson,et al.  Item response theory for psychologists , 2000 .

[34]  G. Norman,et al.  Content specificity and oral certification examinations , 1996 .

[35]  Zubair Amin,et al.  Medical education in Southeast Asia: emerging issues, challenges and opportunities , 2005, Medical education.

[36]  Robert Lukhele,et al.  On the Relative Value of Multiple-Choice, Constructed Response, and Examinee-Selected Items on Two Achievement Tests. Program Statistics Research Technical Report No. 93-28. , 1993 .

[37]  Randy Elliot Bennett,et al.  Equivalence of Free-Response and Multiple-Choice Items , 1991 .

[38]  Howard Wainer,et al.  ON THE RELATIVE VALUE OF MULTIPLE‐CHOICE, CONSTRUCTED‐RESPONSE, AND EXAMINEE‐SELECTED ITEMS ON TWO ACHIEVEMENT TESTS1 , 1993 .

[39]  Pete G. Thomas,et al.  The evaluation of electronic marking of examinations , 2003, ITiCSE '03.

[40]  H. Wainer,et al.  COMBINING MULTIPLE-CHOICE AND CONSTRUCTED RESPONSE TEST SCORES: TOWARD A MARXIST THEORY OF TEST CONSTRUCTION , 1992 .

[41]  Moisés Kirk de Carvalho Filho.  Confidence judgments in real classroom settings: monitoring performance in different types of tests. , 2009 .

[42]  Orly Sarid,et al.  Students’ Reactions to Three Typical Examinations in Health Sciences , 2005, Advances in health sciences education : theory and practice.

[43]  Hugh C. Davis,et al.  Rethinking assessment through learning technologies , 2002, ASCILITE.

[44]  M Verma,et al.  Assessment of clinical competence , 1994 .

[45]  Ricardo Conejo,et al.  SIETTE: A Web-Based Tool for Adaptive Testing , 2004, Int. J. Artif. Intell. Educ..

[46]  Martin Bush.  Quality assurance of multiple‐choice tests , 2006 .

[47]  Roger Neighbour,et al.  Achieving acceptable reliability in oral examinations: an analysis of the Royal College of General Practitioners membership examination's oral component , 2003, Medical education.

[48]  M. Aruguete,et al.  Are Computer-Assisted Teaching Methods Effective? , 2004 .

[49]  A. Brown.  Metacognition, executive control, self-regulation, and other more mysterious mechanisms , 1987 .

[50]  D. Nafpaktitis,et al.  Computer as a Tool in Teaching, Examining and Assessing Electronic Engineering Students , 2007, EUROCON 2007 - The International Conference on "Computer as a Tool".

[51]  Sylvia Stracke,et al.  Multimedia-driven teaching significantly improves students' performance when compared with a print medium. , 2006, The Annals of thoracic surgery.

[52]  M. Tann,et al.  Increasing examiner involvement in an objective structured clinical examination by integrating a structured oral examination. , 1997, American journal of surgery.