Technology-Enabled and Universally Designed Assessment: Considering Access in Measuring the Achievement of Students with Disabilities--A Foundation for Research

This paper represents one outcome from the Invitational Research Symposium on Technology-Enabled and Universally Designed Assessments, which examined technology-enabled assessments (TEA) and universal design (UD) as they relate to students with disabilities (SWD). It was developed to stimulate research into TEAs designed to make tests appropriate for the full range of the student population through enhanced accessibility. Four themes are explored: (a) a construct-centered approach to developing accessible assessments; (b) how technology and UD can provide access to targeted knowledge, skills, and abilities by embedding access and interactive features directly into systems that deliver TEAs; (c) the possibility of incorporating scaffolding directly into innovative assessment items; and (d) the importance of investigating the validity of inferences from TEAs that incorporate accessibility features designed to maximize validity. Through the paper, symposium participants and contributing authors share their understanding of issues and offer insights to researchers who conduct studies on the design, development, and validation of technology-enabled and universally designed assessments that include SWD. The paper proposes a focused research agenda and makes it clear that a principled program of research is needed to properly develop and use technology-enabled and universally designed educational assessments that encourage the inclusion of SWD. As research progresses, TEAs need to improve how they assess students’ understanding of complex academic content and how they provide equitable access to all students including SWD.

[1]  G. Solano-Flores,et al.  The Use of Generalizability (G) Theory in the Testing of Linguistic Minorities. , 2006 .

[2]  Michael Russell,et al.  NimbleTools: A Universally Designed Test Delivery System , 2009 .

[3]  Valerie J. Shute,et al.  Adaptive Technologies. Research Report. ETS RR-07-05. , 2007 .

[4]  Martha Thurlow,et al.  Universal Design Applied to Large Scale Assessments (NCEO Synthesis Report) , 2002 .

[5]  Robert J. Mislevy,et al.  Implications of Evidence‐Centered Design for Educational Testing , 2007 .

[6]  Martha Thurlow,et al.  The Changing Landscape of Alternate Assessments Based on Modified Academic Achievement Standards: An Analysis of Early Adopters of AA-MASs , 2009 .

[7]  Joseph Krajcik,et al.  A Scaffolding Design Framework for Software to Support Science Inquiry , 2004, The Journal of the Learning Sciences.

[8]  Leanne R. Ketterlin-Geller,et al.  Testing Students with Special Needs: A Model for Understanding the Interaction between Assessment and Student Characteristics in a Universally Designed Environment. , 2008 .

[9]  Martha Thurlow,et al.  Considerations for the Administration of Tests to Special Needs Students: Accommodations, Modifications, and More , 2006 .

[10]  Michael Russell,et al.  Computer-Based Testing and Validity : A Look Back and Into the Future , 2002 .

[11]  Jody Clarke-Midura Immersive Interfaces for Assessing Student Learning , 2009 .

[12]  Ryan J. Kettler,et al.  Assessment accommodations for children with special needs , 2010 .

[13]  Charles W. Stansfield,et al.  The Effects of Linguistic Simplification of Science Test Items on Performance of Limited English Proficient and Monolingual English-Speaking Students. , 2001 .

[14]  Beverley Park Woolf,et al.  Building Intelligent Interactive Tutors , 2008 .

[15]  David H. Rose,et al.  Teaching Every Student in the Digital Age: Universal Design for Learning , 2002 .

[16]  Tracey E. Hall,et al.  Universal Design for Learning : Implications for Large-Scale Assessment , 2004 .

[17]  Alija Kulenović,et al.  Standards for Educational and Psychological Testing , 1999 .

[18]  R. Glaser,et al.  Knowing What Students Know: The Science and Design of Educational Assessment , 2001 .

[19]  Stanley Rabinowitz,et al.  Computer-Based Assessment: Can It Deliver on Its Promise? , 2012 .

[20]  Robert Helwig,et al.  Reading as an Access to Mathematics Problem Solving on Multiple-Choice Tests for Sixth-Grade Students , 1999 .

[21]  Martha Thurlow,et al.  Improving Test Performance of Students With Disabilities...On District and State Assessments , 2000 .

[22]  Lynn S. Fuchs,et al.  Supplementing Teacher Judgments of Mathematics Test Accommodations with Objective Data Sources , 2000 .

[23]  Gerald Tindal,et al.  Developing a New Paradigm for Conducting Research on Accommodations in Mathematics Testing , 2007 .

[24]  Antonio Pflüger,et al.  Executive Summary. , 2012, Journal of the ICRU.

[25]  Stephen G. Sireci,et al.  Test Accommodations for Students With Disabilities: An Analysis of the Interaction Hypothesis , 2005 .

[26]  Robert J. Mislevy,et al.  Psychometric Principles in Student Assessment , 2003 .

[27]  Thomas R. Kratochwill,et al.  Experimental Analysis of the Effects of Testing Accommodations on the Scores of Students with and without Disabilities , 2001 .

[28]  Charles W. Stansfield,et al.  Test translation and adaptation in public education in the USA , 2003 .

[29]  Jamal Abedi,et al.  Research-Supported Accommodation for English Language Learners in NAEP. CSE Technical Report. , 2003 .

[30]  Neal M. Kingston,et al.  Measuring Cognition of Students with Disabilities Using Technology-Enabled Assessments: Recommendations for a National Research Agenda , 2010 .

[31]  Laurene L. Christensen,et al.  A Longitudinal Analysis of State Accommodations Policies , 2009 .

[32]  Laura M. Kuti,et al.  Accommodations for English Language Learners on Statewide English Language Proficiency Assessment , 2012, The Journal of Special Education Apprenticeship.

[33]  Jamal Abedi,et al.  Language Accommodations for English Language Learners in Large-Scale Assessments: Bilingual Dictionaries and Linguistic Modification. CSE Report 666. , 2005 .

[34]  Robert P. Dolan,et al.  Applying Principles of Universal Design to Test Delivery: The Effect of Computer-Based Read-Aloud on Test Performance of High School Students with Learning Disabilities , 2005 .

[35]  P. David Pearson,et al.  Improving the Information Value of Performance Items in Large Scale Assessments: NAEP Validity Studies. Working Paper Series. , 1997 .

[36]  Christine Chin,et al.  Using Concept Cartoons in Formative Assessment: Scaffolding students’ argumentation , 2009 .

[37]  David H. Rose Universal Design for Learning , 2000 .

[38]  J. Bruner,et al.  The role of tutoring in problem solving. , 1976, Journal of child psychology and psychiatry, and allied disciplines.

[39]  Jamie L. Schissel Improving testing for English language learners , 2008 .

[40]  Susy Macqueen,et al.  Validity , 1973, Just Algorithms.

[41]  Samuel Messick,et al.  Alternative Modes of Assessment, Uniform Standards of Validity. Research Report. , 1994 .

[42]  Gerald Tindal,et al.  Research on Mathematics Test Accommodations Relevant to NAEP Testing. , 2004 .

[43]  Brent Bridgeman,et al.  The Effect of Computer-Based Tests on Racial-Ethnic and Gender Groups , 2002 .

[44]  R. Almond,et al.  Focus Article: On the Structure of Educational Assessments , 2003 .

[45]  Dylan Wiliam,et al.  Integrating Assessment with Learning: What Will It Take to Make It Work? , 2007 .

[46]  James E. Ysseldyke,et al.  Varied Opinions on How to Report Accommodated Test Scores: Findings Based on CTB/McGraw-Hill's Framework for Classifying Accommodations (NCEO Synthesis Report) , 2003 .

[47]  Neil T. Heffernan,et al.  Addressing the assessment challenge with an online system that tutors as it assesses , 2009, User Modeling and User-Adapted Interaction.

[48]  H. Wainer,et al.  Are Tests Comprising Both Multiple‐Choice and Free‐Response Items Necessarily Less Unidimensional Than Multiple‐Choice Tests?An Analysis of Two Tests , 1994 .

[49]  J. Pellegrino,et al.  Large-Scale Assessments That Support Learning: What Will It Take? , 2003 .

[50]  Lauren B. Resnick,et al.  Benchmarking and Alignment of Standards and Testing. CSE Technical Report. , 2002 .

[51]  James W. Pellegrino,et al.  A Validity Framework for Evaluating the Technical Quality of Alternate Assessments. , 2007 .

[52]  Martha Thurlow,et al.  Non-Approved Accommodations: Recommendations for Use and Reporting. NCEO Policy Directions, Number 11. , 2000 .

[53]  Sandra J. Thompson,et al.  Using the Think Aloud Method (Cognitive Labs) to Evaluate Test Design for Students with Disabilities and English Language Learners. Technical Report 44. , 2006 .

[54]  Julie Alonzo,et al.  Recommendations for Accommodations , 2007 .

[55]  Joseph Krajcik,et al.  Scaffolding Design Guidelines for Learner-Centered Software Environments , 2002 .

[56]  Leanne R. Ketterlin-Geller,et al.  Knowing What All Students Know: Procedures for Developing Universal Design for Assessment. , 2005 .

[57]  C. Hofstetter,et al.  Assessment Accommodations for English Language Learners: Implications for Policy-Based Empirical Research , 2004 .

[58]  L. S. Vygotksy Mind in society: the development of higher psychological processes , 1978 .

[59]  Lauren B. Resnick,et al.  Benchmarking and Alignment of Standards and Testing , 2004 .

[60]  Meagan Karvonen,et al.  Links for Academic Learning (LAL): A Conceptual Model for Investigating Alignment of Alternate Assessments Based on Alternate Achievement Standards , 2009 .

[61]  R. Almond,et al.  A BRIEF INTRODUCTION TO EVIDENCE-CENTERED DESIGN , 2003 .

[62]  Bill Tucker Beyond the Bubble: Technology and the Future of Student Assessment. , 2009 .

[63]  Lorrie A. Shepard,et al.  Linking Formative Assessment to Scaffolding , 2005 .

[64]  Phoebe C. Winter,et al.  Exploring individual and item factors that affect assessment validity for diverse learners: Results from a large-scale cognitive lab ☆ , 2006 .

[65]  Martha Thurlow,et al.  A Summary of Research on the Effects of Test Accommodations: 1999 through 2001 (NCEO Technical Report) , 2002 .

[66]  Gerald Tindal,et al.  An Oral Versus a Standard Administration of a Large-Scale Mathematics Test , 2002 .

[67]  Michael Russell,et al.  The Paradoxes of High Stakes Testing: How They Affect Students, Their Parents, Teachers, Principals, Schools, and Society , 2009 .

[68]  Martha Thurlow,et al.  Access to Computer-Based Testing for Students with Disabilities (NCEO Synthesis Report) , 2002 .

[69]  Lyle F. Bachman,et al.  Keeping score for all : the effects of inclusion and accommodation policies on large-scale educational assessments , 2004 .

[70]  Robert J. Mislevy,et al.  DESIGN PATTERNS FOR IMPROVING ACCESSIBILITY FOR TEST TAKERS WITH DISABILITIES , 2008 .