Dependability and acceptability of handheld computers in school-based data collection

Given the increasing influence of technology and the explosion in data collection demands, the acceptance and assimilation of new paradigms and technologies require today's educators, researchers, and evaluators to select appropriate tools and apply them effectively. One of these technologies—handheld computers—makes the benefits of computerized data collection more accessible to field-based researchers. Three related studies were conducted to evaluate handheld-based data collection system for use in special education settings and to highlight the acceptability factors to effectively use this emerging technology. The first study reviewed the recent literature on the dependability and willingness of teachers to adopt handheld data collection systems and emphasized five important factors: (1) perceived ease of use; (2) perceived usefulness; (3) intention to use; (4) dependability; and (5) subjective norms. The second study discussed the dependability of handheld computers used by special education teachers for collecting self-report data by addressing four dependability attributes: reliability, maintainability, availability, and safety. Data were collected from five sources: (1) self-reports of time use by 19 special education teachers using Pocket PC computers, (2) observations of time use from eight external data collectors, (3) teacher interviews, (4) technical reports prepared by the researcher, and (5) teacher satisfaction. Results indicated that data collection via handheld computers yielded accurate, complete, and timely data, and was appropriate for these four dependability attributes. The last study investigated teachers' acceptance of handheld computer use by testing the relationship among five factors that influence intention to use this technology which was based on a modified version of the technology acceptance model using the handheld computer acceptance survey responses from 45 special education teachers. The results showed that intention to use handheld computer was directly affected by the devices' perceived usefulness and perceived ease of use. The issue of dependability had a direct and indirect statistically significant effect on perceived ease of use and usefulness, and intention to use a handheld computer, respectively. Overall, three studies demonstrated that handheld computers can be effectively used in the direct observation of behavior in a school environment, without requirements of any settings.

[1]  C. Parr,et al.  CyberTracker in BioKIDS: Customization of a PDA-based scientific data collection application for inquiry learning , 2003 .

[2]  A LOW‐TECH METHOD FOR CONDUCTING REAL‐TIME RECORDING , 1999 .

[3]  P. Donát,et al.  Measuring behaviour: The tools and the strategies , 1991, Neuroscience & Biobehavioral Reviews.

[4]  Arup Dutta,et al.  Development and use of behavior and social interaction software installed on Palm handheld for observation of a child’s social interactions with the environment , 2006, Behavior research methods.

[5]  D. Dillman.  Mail and internet surveys: The tailored design method, 2nd ed. , 2007 .

[6]  M Berg,et al.  Completeness of medical records in emergency trauma care and an IT-based strategy for improvement , 2007, Medical informatics and the Internet in medicine.

[7]  Erika S. Trapl,et al.  Use of audio-enhanced personal digital assistants for school-based data collection. , 2005, The Journal of adolescent health : official publication of the Society for Adolescent Medicine.

[8]  Sarah M. Davis,et al.  Research to industry: four years of observations in classrooms using a network of handheld devices , 2002, Proceedings. IEEE International Workshop on Wireless and Mobile Technologies in Education.

[9]  James M. Johnston,et al.  Readings for Strategies and tactics of behavioral research , 1993 .

[10]  Neil Selwyn,et al.  Students' attitudes toward computers: Validation of a computer attitude scale for 16-19 education , 1997, Comput. Educ..

[11]  Louis H. Berry.  Cognitive Effects of Web Page Design , 2000 .

[12]  Christopher J. McCarthy,et al.  Paper-and-Pencil Or Online? , 2002, Assessment.

[13]  Robert L. Schalock,et al.  Outcome-Based Evaluation , 1995, Springer US.

[14]  Paul D. Greene.  Handheld Computers as Tools for Writing and Managing Field Data , 2001 .

[15]  Fred D. Davis,et al.  User Acceptance of Computer Technology: A Comparison of Two Theoretical Models , 1989 .

[16]  Elliot Soloway,et al.  Pocket PiCoMap: a case study in designing and assessing a handheld concept mapping tool for learners , 2003, CHI '03.

[17]  S. Mumtaz.  Factors affecting teachers' use of information and communications technology: a review of the literature , 2000 .

[18]  Olivia R. Liu Sheng,et al.  Examining the Technology Acceptance Model Using Physician Acceptance of Telemedicine Technology , 1999, J. Manag. Inf. Syst..

[19]  Lynn Batten,et al.  The Major Drawbacks To The Use Of Wireless Communication Products In Education , 2003 .

[20]  K. Sahl,et al.  Tracing Teachers’ Use of Technology in a Laptop Computer School: The Interplay of Teacher Beliefs, Social Dynamics, and Institutional Culture , 2002 .

[21]  Richard G. Lomax,et al.  A Beginner's Guide to Structural Equation Modeling , 2022 .

[22]  Jean Claude Marquié,et al.  Do older adults underestimate their actual computer knowledge? , 2002, Behav. Inf. Technol..

[23]  Wynne W. Chin,et al.  On the use, usefulness, and ease of use of structural equation modeling in MIS research: a note of caution , 1995 .

[24]  S. Kahng,et al.  COMPUTERIZED SYSTEMS FOR COLLECTING REAL‐TIME OBSERVATIONAL DATA , 1998 .

[25]  Stefan E. Schulenberg,et al.  Ethical issues in the use of computerized assessment , 2004, Comput. Hum. Behav..

[26]  Austin Henderson,et al.  Interaction design: beyond human-computer interaction , 2002, UBIQ.

[27]  Michael J. Albers,et al.  User web browsing characteristics using palm handhelds for information retrieval , 2000, SIGDOC.

[28]  Nancy L. Leech,et al.  Linking Research Questions to Mixed Methods Data Analysis Procedures 1 , 2006 .

[29]  Angus S. McDonald,et al.  The impact of individual differences on the equivalence of computer-based and paper-and-pencil educational assessments , 2002, Comput. Educ..

[30]  Peter M. Bentler,et al.  Estimates and tests in structural equation modeling. , 1995 .

[31]  Amel Bouzeghoub,et al.  A Model of Reusable Educational Components for the Generation of Adaptative Courses , 2003, CAiSE Workshops.

[32]  Ralph B. D'Agostino,et al.  Goodness-of-Fit-Techniques , 2020 .

[33]  Demetrios G. Sampson,et al.  Integrating Adaptive Educational Content into Different Courses and Curricula , 2001, J. Educ. Technol. Soc..

[34]  Jaeyool Boo,et al.  Computerized and Paper-and-Pencil Versions of the Rosenberg Self-Esteem Scale: A Comparison of Psychometric Features and Respondent Preferences , 2001 .

[35]  Laku Chidambaram,et al.  A test of the technology acceptance model: the case of cellular telephone adoption , 2000, Proceedings of the 33rd Annual Hawaii International Conference on System Sciences.

[36]  Robert E. Ployhart,et al.  WEB‐BASED AND PAPER‐AND‐PENCIL TESTING OF APPLICANTS IN A PROCTORED SETTING: ARE PERSONALITY, BIODATA, AND SITUATIONAL JUDGMENT TESTS COMPARABLE? , 2003 .

[37]  Martyn Wild.  Technology refusal: Rationalising the failure of student and beginning teachers to use computers , 1996, Br. J. Educ. Technol..

[38]  Mark van 't Hooft,et al.  Technology and the Prospective Teacher: Exploring the Use of the TI-83 Handheld Devices in Social Studies Education , 2006 .

[39]  D. Dillman,et al.  How to conduct your own survey , 1994 .

[40]  B A Weber,et al.  Data collection using handheld computers. , 2000, Nursing research.

[41]  N Heddle,et al.  Comparing hand‐held computers and paper diaries for haemophilia home therapy: a randomized trial , 2004, Haemophilia : the official journal of the World Federation of Hemophilia.

[42]  Gillian H. Ice.  Technological Advances in Observational Data Collection: The Advantages and LImitations of Computer-Assisted Data Collection , 2004 .

[43]  Kenneth W. Merrell,et al.  Behavioral, Social, and Emotional Assessment of Children and Adolescents , 2002 .

[44]  Fred D. Davis,et al.  A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies , 2000, Management Science.

[45]  R. Hoyle.  The structural equation modeling approach: Basic concepts and fundamental issues. , 1995 .

[46]  E. Soloway,et al.  Creating a Framework for Research on Systemic Technology Innovations , 2004 .

[47]  Kimberly J. Vannest,et al.  Research in brief: Web‐based Formative Assessment as Evidence Based Practice in Science Instruction , 2008 .

[48]  A. Yuen,et al.  Gender Differences in Teacher Computer Acceptance. , 2002 .

[49]  Kieran Mathieson,et al.  Predicting User Intentions: Comparing the Technology Acceptance Model with the Theory of Planned Behavior , 1991, Inf. Syst. Res..

[50]  Brian R. Bryant,et al.  Assistive Technology , 1998 .

[51]  M. Schofield,et al.  Surveys and questionnaires in health research , 2010 .

[52]  S. Davis.  The Past Is Not The Future , 2008 .

[53]  Dave L. Edyburn.  Critical issues in special education technology research: What do we know? What do we need to know? , 2001 .

[54]  Clay Spinuzzi.  Using a handheld PC to collect and analyze observational data , 2003, SIGDOC '03.

[55]  I. Ajzen.  Attitudes, Personality and Behavior , 1988 .

[56]  Eric Emerson,et al.  Palm-top computer technologies for behavioural observation research. , 2000 .

[57]  P. Emmelkamp.  Technological Innovations in Clinical Assessment and Psychotherapy , 2005, Psychotherapy and Psychosomatics.

[58]  Paul Jen-Hwa Hu,et al.  Examining technology acceptance by school teachers: a longitudinal study , 2003, Inf. Manag..

[59]  Albert H. Segars,et al.  Re-examining perceived ease of use and usefulness , 1993 .

[60]  Robert Maribe Branch,et al.  Educational Media and Technology Yearbook, 2000. Volume 25. , 2000 .

[61]  Albert C. Oosterhof.  Classroom applications of educational measurement , 1994 .

[62]  Fred D. Davis.  A technology acceptance model for empirically testing new end-user information systems : theory and results , 1985 .

[63]  Laura M. O’Dwyer,et al.  Examining Teacher Technology Use , 2003 .

[64]  Clarence C. Gravlee,et al.  Mobile Computer-Assisted Personal Interviewing with Handheld Computers: The Entryware System 3.0 , 2002 .

[65]  J. Stanton.  AN EMPIRICAL ASSESSMENT OF DATA COLLECTION USING THE INTERNET , 1998 .

[66]  Alex Koohang,et al.  A study of attitudes toward computers: anxiety, confidence, liking, and perception of usefulness , 1987 .

[67]  Gary Hackbarth,et al.  Computer playfulness and anxiety: positive and negative mediators of the system experience effect on perceived ease of use , 2003, Inf. Manag..

[68]  Gustav Mikkelsen,et al.  Consequences of impaired data quality on information retrieval in electronic patient records , 2005, Int. J. Medical Informatics.

[69]  E J Hammond,et al.  Electronic data collection by trainee anaesthetists using palm top computers. , 2000, European journal of anaesthesiology.

[70]  Wilson D. Pace,et al.  Electronic Data Collection Options for Practice-Based Research Networks , 2005, The Annals of Family Medicine.

[71]  Rex B. Kline,et al.  Principles and Practice of Structural Equation Modeling , 1998 .

[72]  Mary Cobol,et al.  Imprecision and incompleteness in relational databases : survey , 2005 .

[73]  Clarence C. Gravlee,et al.  Handheld Computers for Direct Observation of the Social and Physical Environment , 2006 .

[74]  F. A. Pettit.  Exploring the use of the World Wide Web as a psychology data collection tool , 1999 .

[75]  J. Epstein,et al.  Insuring sample equivalence across internet and paper-and-pencil assessments , 2001, Comput. Hum. Behav..

[76]  Israel Spiegler,et al.  Technology and knowledge: bridging a "generating" gap , 2003, Inf. Manag..

[77]  John Ingham,et al.  Why do people use information technology? A critical review of the technology acceptance model , 2003, Inf. Manag..

[78]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[79]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[80]  Heinz Ulrich Hoppe,et al.  Guest editorial: Wireless and Mobile Technologies in Education , 2003, J. Comput. Assist. Learn..

[81]  Darin J Erickson,et al.  Handheld Computers , 2003, Evaluation review.

[82]  Donald R. Bell.  Observing and recording children's behavior , 1977 .

[83]  D. Reschly.  The Present and Future Status of School Psychology in the United States , 2000 .

[84]  Constantine Stephanidis,et al.  User Interface Design for PDAs: Lessons and Experience with the WARD-IN-HAND Prototype , 2002, User Interfaces for All.

[85]  G. F. Hayhoe.  From desktop to palmtop: creating usable online documents for wireless and handheld devices , 2001, IPCC 2001. Communication Dimensions. Proceedings IEEE International Professional Communication Conference (Cat. No.01CH37271).

[86]  V Hampshire.  Handheld digital equipment for weight composite distress paradigms: new considerations and for rapid documentation and intervention of rodent populations. , 2001, Contemporary topics in laboratory animal science.

[87]  Mark R Dixon.  Creating a portable data-collection system with Microsoft Embedded Visual Tools for the Pocket PC. , 2003, Journal of applied behavior analysis.

[88]  Charles R. Greenwood,et al.  Development and validation of standard classroom observation systems for school practitioners: Ecobehavioral Assessment Systems Software (EBASS) , 1994 .

[89]  Dorian G. W. Smith,et al.  Palm computer demonstrates a fast and accurate means of burn data collection. , 2000, The Journal of burn care & rehabilitation.

[90]  Jun Gong,et al.  GUIDELINES FOR HANDHELD MOBILE DEVICE INTERFACE DESIGN , 2004 .

[91]  Kevin B. Johnson,et al.  An Electronic Menstrual Cycle Calendar: Comparison of Data Quality With a Paper Version , 2000, Menopause.

[92]  Peter W. Airasian.  Classroom Assessment: Concepts and Applications , 1991 .

[93]  Improvement,et al.  In Teachers' Hands: Investigating the Practices of Classroom Assessment , 1992 .

[94]  S. Lapinsky,et al.  Review Paper: Handheld Computing in Medicine , 2003, J. Am. Medical Informatics Assoc..

[95]  A. Wolfe,et al.  A Framework for Analyzing Dialogues over the Acceptability of Controversial Technologies , 2002 .

[96]  James R. Sanders,et al.  The changing face of educational evaluation , 1991 .

[97]  Edward S. Shapiro,et al.  Academic Skills Problems: Direct Assessment and Intervention , 2010 .

[98]  Paul P Stork,et al.  A randomized trial of electronic versus paper pain diaries in children: impact on compliance, accuracy, and acceptability , 2004, Pain.

[99]  J. Lawton,et al.  A Review of the Literature on Attitudes Towards Computers and Computerized Instruction. , 1982 .

[100]  Tufan Adiguzel,et al.  Improving School Children's Mathematical Word Problem Solving Skills through Computer-Based Multiple Representations , 2004 .

[101]  G. Cziko,et al.  Teacher Adoption of Technology: A Perceptual Control Theory Perspective , 2001 .

[102]  V. Anfara,et al.  Qualitative Analysis on Stage: Making the Research Process More Public , 2002 .

[103]  Edith de Leeuw,et al.  Technological Innovations in Data Collection: Acceptance, Data Quality and Costs , 1996 .

[104]  C L Berthelsen,et al.  Automated personal health inventory for dentistry: a pilot study. , 2000, Journal of the American Dental Association.

[105]  Stanley E. Kaufman,et al.  Electronic collection of health-related quality of life data: Validity, time benefits, and patient preference , 2004, Quality of Life Research.

[106]  Ann Bell.  Handheld Computers in Schools and Media Centers , 2007 .

[107]  A. Onwuegbuzie,et al.  Mixed Methods Research: A Research Paradigm Whose Time Has Come , 2004 .

[108]  Brian Randell,et al.  Fundamental Concepts of Dependability , 2000 .

[109]  Damian W. Betebenner,et al.  Accountability Systems: Implications of Requirements of the No Child Left Behind Act of 2001 , 2002 .

[110]  Elliot Soloway,et al.  Handheld use in K-12: a descriptive account , 2002, Proceedings. IEEE International Workshop on Wireless and Mobile Technologies in Education.

[111]  Edwin H. Blake,et al.  A eld computer for animal trackers , 2002 .

[112]  David R. Morse,et al.  Using while moving: HCI issues in fieldwork environments , 2000, TCHI.