Examining accommodation effects for equity by overcoming a methodological challenge of sparse data.

BACKGROUND As accommodation itself is an equity issue in varied contexts in and beyond education (e.g., the provision of assistive technology, extended time, or read-aloud), it is critical to examine the equitability of accommodation policies and practices by examining their effects on student performance. AIMS This study sought to assess the effectiveness of thirty-one bundled accommodations for students with learning disabilities, emotional or behavioral disorders, or multiple exceptionalities writing a provincial literacy test in Ontario, Canada. METHODS AND PROCEDURES We employed quantitative methods of log-linear analysis and odds ratio to examine the data. To analyze sparse data, we compared three different adjustment methods to meet this methodological challenge. OUTCOMES AND RESULTS Our findings suggest that the problems with sparse data can be overcome by an adjustment method. We also found that the likelihood of achieving the provincial standards may differ among students with special needs depending on whether they did or did not receive certain combinations of accommodations for the literacy test. CONCLUSIONS AND IMPLICATIONS We recommend that education stakeholders review the accommodations that produced significant differential effects to address the concerns regarding whether the test results were interpreted validly and fairly for students with special needs.

[1]  J. Deeks When can odds ratios mislead? , 1998 .

[2]  F. Lord Applications of Item Response Theory To Practical Testing Problems , 1980 .

[3]  Benjamin J. Lovett,et al.  Extended Time as a Testing Accommodation for Students With Reading Disabilities , 2008 .

[4]  Pei-Ying Lin,et al.  Examining Student Factors in Sources of Setting Accommodation DIF , 2014 .

[5]  Lyle F. Bachman,et al.  Keeping score for all : the effects of inclusion and accommodation policies on large-scale educational assessments , 2004 .

[6]  D. Klinger The Evolving Culture of Large-Scale Assessments in Canadian Education. , 2008 .

[7]  Martha Thurlow,et al.  A Summary of Research on the Effects of Test Accommodations: 1999 through 2001 (NCEO Technical Report) , 2002 .

[8]  J. Vandenbroucke,et al.  Prophylactic corticosteroids for cardiopulmonary bypass in adults. , 2011, The Cochrane database of systematic reviews.

[9]  James E. Ysseldyke,et al.  Varied Opinions on How to Report Accommodated Test Scores: Findings Based on CTB/McGraw-Hill's Framework for Classifying Accommodations (NCEO Synthesis Report) , 2003 .

[10]  Hongli Li The Effects of Read-Aloud Accommodations for Students With and Without Disabilities: A Meta-Analysis , 2014 .

[11]  Sandra J. Thompson,et al.  State Policies on Assessment Participation and Accommodations for Students with Disabilities , 2005 .

[12]  T. Einarson,et al.  The safety of anti-tumour necrosis factor treatments in rheumatoid arthritis: meta and exposure-adjusted pooled analyses of serious adverse events , 2008, Annals of the rheumatic diseases.

[13]  Lynn S. Fuchs,et al.  Supplementing Teacher Judgments of Mathematics Test Accommodations with Objective Data Sources , 2000 .

[14]  Stephen N. Elliott,et al.  Handbook of accessible achievement tests for all students : bridging the gaps between research, practice, and policy , 2011 .

[15]  H. Davies,et al.  When can odds ratios mislead? , 1998, BMJ.

[16]  Ryan J. Kettler Testing Accommodations: Theory and research to inform practice , 2012 .

[17]  Stephen G. Sireci,et al.  Test Accommodations for Students With Disabilities: An Analysis of the Interaction Hypothesis , 2005 .

[18]  Vitaliy Shyyan,et al.  A Summary of the Research on the Effects of Test Accommodations: 2007-2008. Technical Report 56. , 2010 .

[19]  Kenneth F Schulz,et al.  Making sense of odds and odds ratios. , 2008, Obstetrics and gynecology.

[20]  Thomas R. Kratochwill,et al.  The effects and perceived consequences of testing accommodations on math and science performance assessments. , 2009 .

[21]  Laura S. Hamilton,et al.  Assessing Students with Disabilities in Kentucky: The Effects of Accommodations, Format, and Subject. CSE Technical Report. , 1999 .

[22]  J. Abedi Computer Testing as a Form of Accommodation for English Language Learners , 2009 .

[23]  B. Tabachnick,et al.  Using Multivariate Statistics , 1983 .

[24]  Laura S. Hamilton,et al.  Assessment of Students With Disabilities in Kentucky: Inclusion, Student Performance, and Validity , 2000 .

[25]  Lynn S. Fuchs,et al.  Using Objective Data Sources to Enhance Teacher Judgments about Test Accommodations , 2000 .

[26]  Alija Kulenović,et al.  Standards for Educational and Psychological Testing , 1999 .

[27]  S. Bolt,et al.  Comparing DIF Across Math and Reading/Language Arts Tests for Students Receiving a Read-Aloud Accommodation , 2006 .

[28]  J. Leahy Uncertain Effects of Rosiglitazone on the Risk for Myocardial Infarction and Cardiovascular Death , 2008 .

[29]  G. Tindal,et al.  Validity Evidence for Making Decisions About Accommodated and Modified Large-Scale Tests , 2011 .

[30]  R. Silvestri,et al.  Assistive Technology Use by Students With LD in Postsecondary Education , 2012 .

[31]  A Summary of the Research on the Effects of Test Accommodations: 2009-2010 (NCEO Technical Report) , 2012 .

[32]  Stephen N. Elliott,et al.  Extended Time as a Testing Accommodation: Its Effects and Perceived Consequences , 2004 .

[33]  Shameem Nyla NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION , 2004 .

[34]  Lynn S. Fuchs,et al.  Fair and Unfair Testing Accommodations. , 1999 .

[35]  Magdalena Szumilas Explaining odds ratios. , 2010, Journal of the Canadian Academy of Child and Adolescent Psychiatry = Journal de l'Academie canadienne de psychiatrie de l'enfant et de l'adolescent.

[36]  Andy P. Field,et al.  Discovering Statistics Using SPSS , 2000 .

[37]  A. Agresti,et al.  Categorical Data Analysis , 1991, International Encyclopedia of Statistical Science.

[38]  A. Cohen,et al.  The Role of Extended Time and Item Content on a High–Stakes Mathematics Test , 2005 .

[39]  B. Pennington,et al.  Psychiatric comorbidity in children and adolescents with reading disability. , 2000, Journal of child psychology and psychiatry, and allied disciplines.

[40]  Maria E. Hernández Finch,et al.  Investigation of Specific Learning Disability and Testing Accommodations Based Differential Item Functioning Using a Multilevel Multidimensional Mixture Item Response Theory Model , 2013 .

[41]  Z. Stoneman,et al.  Situational Influences on Assessment Performance , 1978, Exceptional children.

[42]  S. E. Phillips High-Stakes Testing Accommodations: Validity Versus Disabled Rights , 1994 .

[43]  R. Warner Applied Statistics: From Bivariate through Multivariate Techniques [with CD-ROM]. , 2007 .

[44]  Charles A. MacArthur,et al.  Dictation and Speech Recognition Technology as Test Accommodations , 2004 .

[45]  R. Childs,et al.  Untangling complex effects of disabilities and accommodations within a multilevel IRT framework , 2016 .

[46]  S. Vaughn,et al.  Effects of a Bundled Accommodations Package on High-Stakes Testing for Middle School Students with Reading Disabilities , 2009 .

[47]  Alexander J Sutton,et al.  What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. , 2004, Statistics in medicine.

[48]  J. Irala,et al.  Abnormally wide confidence intervals in logistic regression: interpretation of statistical program results , 1997 .

[49]  Mark D. Reckase,et al.  TECHNICAL GUIDELINES FOR ASSESSING COMPUTERIZED ADAPTIVE TESTS , 1984 .

[50]  U.S. Legal Issues in Educational Testing of Special Populations , 2011 .

[51]  Charles A. MacArthur,et al.  Learning Disabled Students' Composing Under Three Methods of Text Production , 1987 .

[52]  B. Fingleton Models of Category Counts , 1984 .

[53]  Philip Nagy The Three Roles of Assessment: Gatekeeping, Accountability, and Instructional Diagnosis , 2000 .

[54]  S. Dubrava,et al.  Clinical and Statistical Assessment of Suicidal Ideation and Behavior in Pharmaceutical Trials , 2011, Clinical pharmacology and therapeutics.

[55]  H. Buzick,et al.  A Meta‐Analysis of Research on the Read Aloud Accommodation , 2014 .

[56]  W. Rogers,et al.  Principles for Fair Student Assessment Practices for Education in Canada , 1993 .

[57]  R. Crundwell Alternative Strategies for Large Scale Student Assessment in Canada: Is Value-Added Assessment One Possible Answer , 2005 .