The impact of individual differences on the equivalence of computer-based and paper-and-pencil educational assessments

Computer-based assessment (CBA) is yet to have a significant impact on high-stakes educational assessment, but the equivalence between CBA and paper-and-pencil (P&P) test scores will become a central concern in education as CBA increases. It is argued that as CBA and P&P tests provide test takers with qualitatively different experiences, the impact of individual differences on the testing experience, and so statistical equivalence of scores, needs to be considered. As studies of score equivalence have largely ignored individual differences such as computer experience, computer anxiety and computer attitudes, the purpose of this paper is to highlight the potential effects of these. It is concluded that each of these areas is of significance to the study of equivalence and that the often inconsistent findings result from the rapid changes in exposure to technology.

[1]  Joan Jamieson,et al.  THE RELATIONSHIP BETWEEN COMPUTER FAMILIARITY AND PERFORMANCE ON COMPUTER-BASED TOEFL TEST TASKS , 1998 .

[2]  Chris Singleton,et al.  Computerised baseline assessment of literacy , 1999 .

[3]  G. Neuman,et al.  Computerization of Paper-and-Pencil Tests: When are They Equivalent? , 1998 .

[4]  T. Levin,et al.  Effect of Gender and Computer Experience on Attitudes toward Computers , 1989 .

[5]  Angela F. L. Wong,et al.  Computer anxiety and its correlates : a meta-analysis , 1999 .

[6]  Sally Brown,et al.  Computer-Assisted Assessment in Higher Education. Staff and Educational Development Series. , 1999 .

[7]  Angus S. McDonald,et al.  The Prevalence and Effects of Test Anxiety in School Children , 2001, DECP Debate.

[8]  F. Drasgow,et al.  Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis. , 1993 .

[9]  John Todman,et al.  Computer Anxiety in Primary Schoolchildren and University Students , 1992 .

[10]  John Mazzeo.  Comparability of Computer and Paper-and-Pencil Scores for Two CLEP General Examinations. College Board Report No. 91-5. , 1991 .

[11]  David Reinking,et al.  The Effects of Computer-Mediated Text on Measures of Reading Comprehension and Reading Behavior , 1985 .

[12]  Donald E. Powers,et al.  Inexperienced and Anxious Computer Users: Coping With a Computer-Administered Test of Academic Skills , 1993 .

[13]  Brandon Bennett,et al.  Reinventing Assessment: Speculations on the Future of Large Scale Educational Testing , 1998 .

[14]  Irwin S. Kirsch,et al.  Examining the Relationship Between Computer Familiarity and Performance on Computer‐Based Language Tasks , 1999 .

[15]  Bernadette Szajna,et al.  An Investigation of the Predictive Validity of Computer Anxiety and Computer Aptitude , 1994 .

[16]  Mark J. Brosnan,et al.  The impact of computer anxiety and self-efficacy upon performance , 1998, J. Comput. Assist. Learn..

[17]  R M Luecht,et al.  A comparative study of a comprehensive basic sciences test using paper-and-pencil and computerized formats. , 1998, Academic medicine : journal of the Association of American Medical Colleges.

[18]  Michael W. Eysenck,et al.  Anxiety and attention , 1988 .

[19]  M. Shermis,et al.  Effects of Computer-Based Test Administrations on Test Anxiety and Performance , 1998 .

[20]  Michael Russell,et al.  Testing Writing on Computers: An Experiment Comparing Student Performance on Tests Conducted via Computer and via Paper-and-Pencil , 1997 .

[21]  T. H. Harrell,et al.  Computerized versus traditional administration of the multidimensional aptitude battery-verbal scale: An examination of reliability and validity , 1987 .

[22]  T. Levine,et al.  Computer use, confidence, attitudes, and knowledge : A causal analysis , 1998 .

[23]  John R. Beech,et al.  The psychological assessment of reading , 1997, The Psychological Assessment of Reading.

[24]  Donna Dalton.  What Do Others Know about CD-ROMs, LANs, Modems, and More? A Survey for Staff Training. , 1994 .

[25]  Douglas F. Johnson,et al.  Effects of training on computerized test performance in the elderly. , 1980 .

[26]  W. J. Pelgrum.  Schools, Teachers, Students and Computers: A Cross-National Perspective. IEA-Comped Study Stage 2. , 1993 .

[27]  Lora Ann Vogel.  Explaining Performance on P&P versus Computer Mode of Administration for the Verbal Section of the Graduate Record Exam , 1994 .

[28]  R. Jayasuriya,et al.  A review of the construct of computer experience , 1999 .

[29]  Walter P. Vispoel,et al.  How instruction influences attitudes of college men and women towards computers , 1993 .

[30]  L.Michael Honaker,et al.  The equivalency of computerized and conventional MMPI administration: A critical review☆☆☆ , 1988 .

[31]  Steven L. Wise,et al.  The Relationship between Examinee Anxiety and Preference for Self-Adapted Testing. , 1994 .

[32]  Rob Millar,et al.  A Survey of Adolescent Worry In Northern Ireland , 1996 .

[33]  Larry D. Rosen,et al.  The psychological impact of technology from a global perspective , 1995 .

[34]  Paul H. Dimock,et al.  The Effects of Format Differences and Computer Experience on Performance and Anxiety on a Computer-Administered Test. , 1991 .

[35]  M. Humphreys,et al.  Personality, motivation, and performance: a theory of the relationship between individual differences and information processing. , 1984, Psychological review.

[36]  Fritz Drasgow,et al.  Innovations in Computerized Assessment , 1999 .

[37]  Fayneese S. Miller,et al.  The Effects of Psychosocial Factors on Indian Children's Attitudes toward Computers , 1994 .

[38]  Michael Russell,et al.  Testing On Computers , 1999 .

[39]  R. Hembree.  Correlates, Causes, Effects, and Treatment of Test Anxiety , 1988 .

[40]  Michael W. Gos.  Computer Anxiety and Computer Experience: A New Look at an Old Relationship. , 1996 .

[41]  R. Kay.  An exploration of theoretical and practical foundations for assessing attitudes toward computers: The Computer Attitude Measure (CAM) , 1993 .

[42]  David Zandvliet,et al.  A Comparison of Computer-Administered and Written Tests , 1997 .

[43]  P Wright,et al.  Computer anxiety: a comparison of pen-based personal digital assistants, conventional computer and paper assessment of mood and performance. , 1998, British journal of psychology.

[44]  Rosemary E. Sutton.  Equity and High Stakes Testing: Implications for Computerized Testing , 1997 .

[45]  Deléne Visser,et al.  Comparability of the Paper-and-Pencil and Computerised Adaptive Versions of the General Scholastic Aptitude Test (GSAT) Senior , 1998 .

[46]  F. Vijver,et al.  The incomplete equivalence of the paper-and-pencil and computerized versions of the General Aptitude Test Battery , 1994 .

[47]  Pat-Anthony Federico,et al.  Measuring recognition performance using computer-based and paper-based methods , 1991 .

[48]  Said S. Al-Gahtani,et al.  Attitudes, satisfaction and usage: Factors contributing to each in the acceptance of information technology , 1999, Behav. Inf. Technol..

[49]  Howard Wainer,et al.  Computerized Adaptive Testing: A Primer , 2000 .

[50]  Walter P. Vispoel,et al.  Individual Differences and Test Administration Procedures: A Comparison of Fixed-Item, Computerized-Adaptive, and Self-Adapted Testing. , 1994 .

[51]  T. Levine,et al.  Commitment to Learning: Effects of Computer Experience, Confidence and Attitudes , 1997 .

[52]  Jacob E. Hautaluoma,et al.  Anxiety management training as a strategy for enhancing computer user performance , 1990 .

[53]  Glenn Fulcher,et al.  Computerizing an English language placement test , 1999 .

[54]  Jo Ann Lee.  The Effects of Past Computer Experience on Computerized Aptitude Test Performance , 1986 .

[55]  M. Burke,et al.  Examinee attitudes toward computer-administered ability testing , 1987 .

[56]  Donald E. Powers.  TEST ANXIETY AND TEST PERFORMANCE: COMPARING PAPER‐BASED AND COMPUTER‐ADAPTIVE VERSIONS OF THE GRE GENERAL TEST , 1999 .