Computer- vs. paper-based tasks: Are they equivalent?

In 1992, Dillon published his critical review of the empirical literature on reading from paper vs. screen. However, the debate concerning the equivalence of computer- and paper-based tasks continues, especially with the growing interest in online assessment. The current paper reviews the literature over the last 15 years and contrasts the results of these more recent studies with Dillon's findings. It is concluded that total equivalence is not possible to achieve, although developments in computer technology, more sophisticated comparative measures and more positive user attitudes have resulted in a continuing move towards achieving this goal. Many paper-based tasks used for assessment or evaluation have been transferred directly onto computers with little regard for any implications. This paper considers equivalence issues between the media by reviewing performance measures. While equivalence seems impossible, the importance of any differences appears specific to the task and required outcomes.

[1]  Per Carlbring,et al.  Internet vs. paper and pencil administration of questionnaires commonly used in panic/agoraphobia research , 2007, Comput. Hum. Behav..

[2]  John Eustis Williams,et al.  Equivalence of standard and computerized versions of the Raven Progressive Matrices Test , 2006, Comput. Hum. Behav..

[3]  Hsiu-Mei Huang,et al.  Do print and Web surveys provide the same results? , 2006, Comput. Hum. Behav..

[4]  Erik Wästlund,et al.  Effects of VDT and paper presentation on consumption and production of information: Psychological and physiological factors , 2005, Comput. Hum. Behav..

[5]  Daniel H. Robinson,et al.  Speed and Performance Differences among Computer-Based and Paper-Pencil Tests , 2004 .

[6]  Kenneth Tait,et al.  Computer or paper? That is the question: does the medium in which assessment questions are presented affect children's performance in mathematics? , 2004 .

[7]  James W. Smither,et al.  An Examination of the Equivalence of Web-Based Versus Paper-and-Pencil Upward Feedback Ratings: Rater- and Ratee-Level Analyses , 2004 .

[8]  Scott McCoy,et al.  Electronic versus paper surveys: analysis of potential psychometric biases , 2004, 37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the.

[9]  Kate J. Garland,et al.  CRT monitors: Do they interfere with learning? , 2004, Behav. Inf. Technol..

[10]  Palmer Morrel-Samuels,et al.  Web surveys' hidden hazards. , 2003, Harvard business review.

[11]  Jan Noyes,et al.  VDT versus paper-based text: reply to Mayes, Sims and Koonce , 2003 .

[12]  Amie Goldberg,et al.  The Effect of Computers on Student Writing: A Meta-analysis of Studies from 1992 to 2002 , 2003 .

[13]  Kenneth K. Boyer,et al.  Print versus electronic surveys: A comparison of two data collection methodologies , 2002 .

[14]  David Schwartz,et al.  Social desirability and controllability in computerized and paper-and-pencil personality questionnaires , 2002, Comput. Hum. Behav..

[15]  Christopher J. McCarthy,et al.  Paper-and-Pencil Or Online? , 2002, Assessment.

[16]  Brent Bridgeman,et al.  The Effect of Computer-Based Tests on Racial-Ethnic and Gender Groups , 2002 .

[17]  Brian C. Cronk,et al.  Personality research on the Internet: A comparison of Web-based and traditional instruments in take-home and in-class settings , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[18]  Douglas F. Becker,et al.  The Score Equivalence of Paper-and-Pencil and Computerized Versions of a Speeded Test of Reading Comprehension , 2002 .

[19]  Robert MacCann,et al.  Responding to free response examination questions: computer versus pen and paper , 2002, Br. J. Educ. Technol..

[20]  Jefferson M. Koonce,et al.  Comprehension and workload differences for VDT and paper-based reading , 2001 .

[21]  Richard Turner,et al.  The myth of the paperless office , 2001 .

[22]  Jaeyool Boo,et al.  Computerized and Paper-and-Pencil Versions of the Rosenberg Self-Esteem Scale: A Comparison of Psychometric Features and Respondent Preferences , 2001 .

[23]  Alysse Weinberg Comparaison de deux versions d'un test de classement: version papier-crayon et version informatisée , 2001 .

[24]  Min Liu,et al.  Exploring the use of multimedia examination formats in undergraduate teaching: results from the fielding testing , 2001, Comput. Hum. Behav..

[25]  Tianyou Wang,et al.  Evaluating Comparability in Computerized Adaptive Testing: Issues, Criteria and an Example , 2001 .

[26]  Daniel J. Bernstein,et al.  An Examination of the Equivalence between Non-Adaptive Computer-Based and Traditional Testing , 2001 .

[27]  Walter P. Vispoel Computerized Versus Paper-and-Pencil Assessment of Self-Concept: Score Comparability and Respondent Preferences , 2000 .

[28]  L. Saxe,et al.  A comparison of paper vs computer-assisted self interview for school alcohol, tobacco, and other drug surveys , 2000 .

[29]  F. Drasgow,et al.  Does computerizing paper-and-pencil job attitude scales make a difference? New IRT analyses offer insight. , 2000, The Journal of applied psychology.

[30]  Karen Littleton,et al.  Gender, Pair Composition and Computer Versus Paper Presentations of an English Language Task , 2000 .

[31]  R. N. Davis,et al.  Web-based administration of a personality questionnaire: Comparison with traditional methods , 1999, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[32]  Michael Russell,et al.  Testing On Computers , 1999 .

[33]  Stefan E. Schulenberg,et al.  The equivalence of computerized and paper-and-pencil psychological instruments: Implications for measures of negative affect , 1999, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[34]  Richard E. Mayer,et al.  Computer-Based Assessment of Problem Solving. , 1999 .

[35]  Martina Ziefle,et al.  Effects of Display Resolution on Visual Performance , 1998, Hum. Factors.

[36]  P Wright,et al.  Computer anxiety: a comparison of pen-based personal digital assistants, conventional computer and paper assessment of mood and performance. , 1998, British journal of psychology.

[37]  S. Schwartz,et al.  Effects of authoritative structure in the measurement of identity formation: individual computer-managed versus group paper-and-pencil testing , 1998 .

[38]  G. Neuman,et al.  Computerization of Paper-and-Pencil Tests: When are They Equivalent? , 1998 .

[39]  Benjamin M. Ogles,et al.  Computerized Depression Screening and Awareness , 1998, Community Mental Health Journal.

[40]  Wesley C. King,et al.  Gender and Administration Mode Effects when Pencil-And-Paper Personality Tests are Computerized , 1998 .

[41]  Michael A. Smith,et al.  Virtual subjects: Using the Internet as an alternative source of subjects and research environment , 1997 .

[42]  Beth E. Haverkamp,et al.  Comparison of User Reaction to Two Methods of Strong Interest Inventory Administration and Report Feedback. , 1997 .

[43]  Hamish Macleod,et al.  Computer Anxiety and Measurement of Mood Change , 1997 .

[44]  David Zandvliet,et al.  A Comparison of Computer-Administered and Written Tests , 1997 .

[45]  Jane Webster,et al.  Computer-assisted versus paper-and-pencil administration of questionnaires , 1996 .

[46]  D. Dilalla Computerized Administration of the Multidimensional Personality Questionnaire , 1996 .

[47]  F. Craik,et al.  Relations among memory performance, mental workload and cardiovascular responses. , 1996, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[48]  Sven G. Carlsson,et al.  Computerized testing in a hospital setting: Psychometric and psychological effects , 1996 .

[49]  T. Pinsoneault,et al.  Equivalency of computer-assisted and paper-and-pencil administered versions of the Minnesota Multiphasic Personality Inventory-2 , 1996 .

[50]  S. Tsemberis,et al.  Expert Judgments of Computer-Based and Clinician-Written Reports , 1996 .

[51]  Willibald Ruch,et al.  A comparison of computerized and conventional administration of the German versions of the Eysenck Personality Questionnaire and the Carroll Rating Scale for Depression , 1996 .

[52]  N. Stuckless,et al.  The effects of computer versus paper-and-pencil administration on measures of anger and revenge with an inmate population , 1996 .

[53]  Edward W. Miles,et al.  A quasi-experimental assessment of the effect of computerizing noncognitive paper-and-pencil measurements: A test of measurement equivalence. , 1995 .

[54]  Shellie D. Locke,et al.  Method of psychological assessment, self-disclosure, and experiential differences: A study of computer, questionnaire, and interview assessment formats. , 1995 .

[55]  Robert W. Bell,et al.  Computerized versus standard personality measures: Equivalency, computer anxiety, and gender differences , 1994 .

[56]  N. Allen,et al.  Computerized and Written Questionnaires: Are They Equivalent? , 1994 .

[57]  Steven V. Horton,et al.  A Comparison of Two Methods of Administering Group Reading Inventories to Diverse Learners , 1994 .

[58]  G. E. Rice,et al.  Examining Constructs in Reading Comprehension Using Two Presentation Modes: Paper vs. Computer , 1994 .

[59]  A. Beck,et al.  Use of the computer-administered Beck depression inventory and hopelessness scale with psychiatric inpatients , 1994 .

[60]  Andrew Dillon,et al.  Designing Usable Electronic Text: Ergonomic Aspects Of Human Information Usage , 1994 .

[61]  T. Vansickle,et al.  Comparing paper-pencil and computer-based versions of the strong-campbell interest inventory , 1993 .

[62]  K. Kobak,et al.  Development and validation of a computer-administered version of the Hamilton Rating Scale. , 1993 .

[63]  F. Drasgow,et al.  Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis. , 1993 .

[64]  C. G. Watson,et al.  Do computer-administered Minnesota Multiphasic Personality Inventories underestimate booklet-based scores? , 1992, Journal of clinical psychology.

[65]  K. Kobak,et al.  A Computer-Administered Version of the Yale-Brown Obsessive-Compulsive Scale , 1992 .

[66]  Dennis Shasha,et al.  Information Search with Dynamic Text vs Paper Text: An Empirical Comparison , 1991, Int. J. Man Mach. Stud..

[67]  Paul Muter,et al.  Reading and skimming from computer screens and books: the paperless office revisited? , 1991 .

[68]  David J. Oborne,et al.  Reading from Screen versus Paper: There is No Difference , 1988, Int. J. Man Mach. Stud..

[69]  John D. Gould,et al.  Reading from CRT Displays Can Be as Fast as Reading from Paper , 1987 .

[70]  V Barnes,et al.  Reading Is Slower from CRT Displays than from Paper: Attempts to Isolate a Single-Variable Explanation , 1987, Human factors.

[71]  R. T. Wilkinson,et al.  Proof-reading: VDU and paper text compared for speed, accuracy and fatigue , 1987 .

[72]  W. Cushman Reading from Microfiche, a VDT, and the Printed Page: Subjective Fatigue and Performance , 1986, Human factors.

[73]  Susanne Askwall,et al.  Computer Supported Reading vs Reading Text on Paper: A Comparison of Two Reading Situations , 1985, Int. J. Man Mach. Stud..

[74]  Debra M. Switchenko Reading from CRT versus Paper: The CRT-Disadvantage Hypothesis Re-Examined , 1984 .

[75]  Paul Muter,et al.  Reading of Continuous Text on Video Screens , 1984 .

[76]  J. D. Gould,et al.  Doing the Same Work with Hard Copy and with Cathode-Ray Tube (CRT) Computer Terminals , 1984 .

[77]  A. Lickorish,et al.  Proof-reading texts on screen and paper , 1983 .

[78]  William C. Treurniet,et al.  Extended Reading of Continuous Text on Television Screens , 1982 .

[79]  Anita V. Kak Relationships Between Readability of Printed and Crt-Displayed Text , 1981 .

[80]  John D. Gould,et al.  Composing Letters with Computer-Based Text Editors , 1981 .

[81]  Wilfred J. Hansen,et al.  Why an Examination Was Slower On-Line than On Paper. , 1978 .

[82]  Gerald E. Larson,et al.  Social desirability effects on computerized and paper-and-pencil questionnaires , 2007, Comput. Hum. Behav..

[83]  Kate J. Garland,et al.  Paper-based versus computer-based assessment: is workload another test mode effect? , 2004, Br. J. Educ. Technol..

[84]  Edward W. Wolfe,et al.  Composition Medium Comparability in a Direct Writing Assessment of Non-Native English Speakers , 2004 .

[85]  H. K. Lee,et al.  A Comparative Study of ESL Writers' Performance in a Paper-Based and a Computer-Delivered Writing Test. , 2004 .

[86]  Herschel Knapp,et al.  Using pencil and paper, Internet and touch-tone phones for self-administered surveys: does methodology matter? , 2003, Comput. Hum. Behav..

[87]  Young-Ju Lee,et al.  A comparison of composing processes and written products in timed-essay tests across paper-and-pencil and computer modes , 2002 .

[88]  Alysse Weinberg Comparaison de deux versions d'une test de classement: Version papier-crayon et version informatisee (Comparison of Two Versions of a Placement Test: Paper-Pencil Version and Computer-based Version). , 2001 .

[89]  S DeAngelis,et al.  Equivalency of computer-based and paper-and-pencil testing. , 2000, Journal of allied health.

[90]  Rich Picking,et al.  Reading music from screens vs paper , 1997, Behav. Inf. Technol..

[91]  P. Bobko,et al.  Computer versus paper-and-pencil administration mode and response distortion in noncognitive selection tests. , 1997, The Journal of applied psychology.

[92]  F. Vijver,et al.  The incomplete equivalence of the paper-and-pencil and computerized versions of the General Aptitude Test Battery , 1994 .

[93]  I. Davies Whatever Happened to Political Education , 1994 .

[94]  Ron Oliver,et al.  Proof-Reading on Paper and Screens: The Influence of Practice and Experience on Performance. , 1993 .

[95]  Andrew Dillon,et al.  Reading from paper versus screens: a critical review of the empirical literature , 1992 .

[96]  Clay E. George,et al.  The effects of computerized versus paper-and-pencil administration on measures of negative affect , 1992 .

[97]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[98]  Stephen E. Newstead,et al.  Proof-reading on VDUs , 1987 .

[99]  Frank H. Heppner,et al.  Reading Performance on a Standardized Test is Better from Print than from Computer Display. , 1985 .

[100]  B. Plake,et al.  Comparing computerized versus traditional psychological assessment , 1985 .

[101]  Susan M. Belmore,et al.  Reading computer-presented text , 1985 .

[102]  Stacey A. Keenan,et al.  Effects of Chunking and Line Length on Reading Efficiency. , 1984 .