The Influence of Measurement Error Associated With Oral Reading Progress Monitoring Measures on the Consistency and Accuracy of Nonparametric Single-Case Design Effect Size Outcomes

Researchers and practitioners frequently use curriculum-based measures of reading (CBM-R) within single-case design (SCD) frameworks to evaluate the effects of reading interventions with individual students. Effect sizes (ESs) developed specifically for SCDs are often used as a supplement to visual analysis to gauge treatment effects. The degree to which measurement error associated with academic measures like CBM-R influences said ESs has not been fully explored. We used simulation methodology to evaluate how common magnitudes of error influenced the consistency and accuracy of outcomes from two nonparametric SCD ESs, percentage of data exceeding baseline trend and TauU. After accounting for other data characteristics, measurement error accounted for a statistically and practically significant amount of variance in the consistency and accuracy of outcomes from both ESs. This article suggests that the psychometric properties of academic measures are important to consider when interpreting ESs from SCDs.

[1]  D. Barnett,et al.  Response to Intervention , 2004 .

[2]  Joseph Betts,et al.  An Investigation of the Psychometric Evidence of CBM-R Passage Equivalence: Utility of Readability Statistics and Equating for Alternate Forms. , 2009 .

[3]  Jack M Fletcher,et al.  Form effects on the estimation of students' oral reading fluency using DIBELS. , 2008, Journal of school psychology.

[4]  J. Begeny,et al.  Single-case effect size calculation: comparing regression and non-parametric approaches across previously published reading intervention data sets. , 2014, Journal of school psychology.

[5]  Jeffrey D Long,et al.  Curriculum-based measurement oral reading as an indicator of reading achievement: a meta-analysis of the correlational evidence. , 2009, Journal of school psychology.

[6]  Philip K. Axtell,et al.  An Investigation of the Reliability and Standard Error of Measurement of Words Read Correctly Per Minute Using Curriculum-Based Measurement , 2005 .

[7]  Theodore J. Christ,et al.  Curriculum-based measurement of oral reading: Passage equivalence and probe-set development , 2009 .

[8]  J. Hendrickson,et al.  Effects of Combined Reading and Question Generation on Reading Fluency and Comprehension of Three Young Adults With Autism and Intellectual Disability , 2012 .

[9]  M. Stoolmiller,et al.  Using school-level student achievement to engage in formative evaluation: comparative school-level rates of oral reading fluency growth conditioned by initial skill for second grade students , 2015 .

[10]  John M Ferron,et al.  From a single-level analysis to a multilevel analysis of single-case experimental designs. , 2014, Journal of school psychology.

[11]  Joel R. Levin,et al.  Meta- and statistical analysis of single-case intervention research data: quantitative gifts and a wish list. , 2014, Journal of school psychology.

[12]  Theodore J. Christ,et al.  Curriculum-based measurement of oral reading: multi-study evaluation of schedule, duration, and dataset quality on progress monitoring outcomes. , 2013, Journal of school psychology.

[13]  Theodore J. Christ,et al.  Foundations of fluency-based assessments in behavioral and psychometric paradigms , 2016 .

[14]  Kimberly J. Vannest,et al.  Effect Size in Single-Case Research: A Review of Nine Nonoverlap Techniques , 2011, Behavior modification.

[15]  Stanley L. Deno,et al.  Individual Differences and Individual Difference , 1990 .

[16]  Hsen-Hsing Ma.  An Alternative Method for Quantitative Synthesis of Single-Subject Researches , 2006, Behavior modification.

[17]  Kimberly J. Vannest,et al.  Combining nonoverlap and trend for single-case research: Tau-U. , 2011, Behavior therapy.

[18]  L. Fuchs,et al.  Treatment Validity as a Unifying Construct for Identifying Learning Disabilities , 2002 .

[19]  Daniel M. Maggin,et al.  Curriculum-based measurement of oral reading (R-CBM): a diagnostic test accuracy meta-analysis of evidence supporting use in universal screening. , 2014, Journal of school psychology.

[20]  Michelle K. Hosp,et al.  Oral Reading Fluency as an Indicator of Reading Competence: A Theoretical, Empirical, and Historical Analysis , 2001 .

[21]  William R. Shadish,et al.  Statistical Analyses of Single-Case Designs , 2014 .

[22]  L. Fuchs,et al.  The “Blurring” of Special Education in a New Continuum of General Education Placements and Services , 2010 .

[23]  Claire M. Roof,et al.  Evaluating Curriculum-Based Measurement from a Behavioral Assessment Perspective. , 2008 .

[24]  Hariharan Swaminathan,et al.  A generalized least squares regression approach for computing effect sizes in single-case research: application examples. , 2011, Journal of school psychology.

[25]  Kelli D. Cummings,et al.  Form Effects on DIBELS Next Oral Reading Fluency Progress- Monitoring Passages , 2013 .

[26]  Theodore J. Christ.  Short-Term Estimates of Growth Using Curriculum-Based Measurement of Oral Reading Fluency: Estimating Standard Error of the Slope to Construct Confidence Intervals , 2006 .

[27]  Deborah K. Reed,et al.  Examiner error in curriculum-based measurement of oral reading. , 2014, Journal of school psychology.

[28]  Erin E. Barton,et al.  Comparison of Overlap Methods for Quantitatively Synthesizing Single-Subject Data , 2010 .

[29]  D. Carnine.  Trustworthiness, useability, and accessibility of educational research , 1995 .

[30]  Lynn S. Fuchs,et al.  Using Curriculum-based Measurement to Establish Growth Standards for Students with Learning Disabilities , 2001 .

[31]  Brian Reichow,et al.  Visual analysis of multiple baseline across participants graphs when change is delayed. , 2010 .

[32]  N. Cooke,et al.  Using a Repeated Reading Program to Improve Generalization of Oral Reading Fluency , 2011 .

[33]  Stanley L. Deno,et al.  Developments in Curriculum-Based Measurement , 2003 .

[34]  John D. Cone,et al.  The relevance of reliability and validity for behavioral assessment , 1977 .

[35]  Eleazar Vasquez,et al.  Evaluation of Synchronous Online Tutoring for Students at Risk of Reading Failure , 2012 .

[36]  J. Kranzler,et al.  Effect of Instructions on Curriculum-Based Measurement of Reading , 2006 .

[38]  M. Bottai,et al.  Logistic quantile regression for bounded outcomes , 2010, Statistics in medicine.

[39]  Justin D. Smith,et al.  Single-case experimental designs: a systematic review of published research and current standards. , 2012, Psychological methods.

[40]  Theodore J. Christ,et al.  A systematic review and summarization of the recommendations and research surrounding Curriculum-Based Measurement of oral reading fluency (CBM-R) decision rules. , 2013, Journal of school psychology.

[41]  Theodore J. Christ,et al.  Curriculum-Based Measurement of Oral Reading: Standard Errors Associated With Progress Monitoring Outcomes From DIBELS, AIMSweb, and an Experimental Passage Set , 2009 .

[42]  Kimberly J. Vannest,et al.  Effect Sizes in Single Case Research: How Large is Large? , 2005 .

[43]  Kimberly J. Vannest,et al.  Interrater Agreement Between Visual Analysts of Single-Case Data , 2015, Behavior modification.

[44]  S. Deno,et al.  Curriculum-Based Measurement: The Emerging Alternative , 1985, Exceptional children.

[45]  David L. Gast,et al.  General Factors in Measurement and Evaluation , 2009 .

[46]  Theodore J. Christ,et al.  Curriculum Based Measurement of Reading: Consistency and Validity Across Best, Fastest, and Question Reading Conditions , 2013 .

[47]  S. Musti-Rao,et al.  Using an iPad® App to Improve Sight Word Reading Fluency for At-Risk First Graders , 2015 .

[48]  Mark Wolery,et al.  A Commentary , 2013 .

[49]  Breda V. O’Keeffe,et al.  Variability in DIBELS Next Progress Monitoring Measures for Students at Risk for Reading Difficulties , 2017 .

[50]  Theodore J. Christ,et al.  An Examination of Variability as a Function of Passage Variance in CBM Progress Monitoring , 2004 .

[51]  Lee J. Cronbach,et al.  The Signal/Noise Ratio in the Comparison of Reliability Coefficients , 1964 .

[52]  Robert H. Hariharan George Keith Horner,et al.  Considerations for the Systematic Analysis and Use of Single-Case Research , 2012 .

[53]  B. Gorman,et al.  Calculating effect sizes for meta-analysis: the case of the single case. , 1993, Behaviour research and therapy.

[54]  Theodore J. Christ,et al.  Estimates of the Standard Error of Measurement for Curriculum-Based Measures of Oral Reading Fluency , 2007 .

[55]  Craig A Albers,et al.  Using Flashcard Drill Methods and Self-Graphing Procedures to Improve the Reading Performance of English Language Learners , 2012 .

[56]  C. Walcott,et al.  Effectiveness of a Computer-Assisted Intervention for Young Children with Attention and Reading Problems , 2014 .