Interrater Agreement Between Visual Analysts of Single-Case Data

Visual analysis is the most widely applied method of data interpretation for single-case research as it encompasses multifaceted considerations relevant to evaluating behavior change. However, a previous research synthesis found low levels of interrater agreement between visually analyzed ratings of graphed data across all variables under analysis. The purpose of this meta-analysis was to evaluate the peer-reviewed literature to date for potential moderators affecting the proportion of interrater agreement between visual analysts. Nineteen articles with 32 effects were assembled. Potential moderators evaluated included (a) design families, (b) rater expertise, (c) the provision of contextual information for graphs, (d) the use of visual aids, (e) the provision of an operational definition of the construct being rated, and (f) rating scale ranges. Results yielded an overall weighted interrater agreement proportion of .76. Moderator variables identified produced low to adequate levels of interrater agreement. Practical recommendations for future research are discussed.

[1]  K J Ottenbacher,et al.  Reliability and accuracy of visually analyzing graphed data from single-subject designs. , 1986, The American journal of occupational therapy : official publication of the American Occupational Therapy Association.

[2]  Michel Hersen,et al.  Comprehensive handbook of psychological assessment , 2004 .

[3]  D. Barnett,et al.  Intervention-based school psychology services: Training for child-level accountability; preparing for program-level accountability , 2004 .

[4]  Craig H. Kennedy,et al.  Single-Case Designs for Educational Research , 2004 .

[5]  Richard I. Parker,et al.  The Relationship Between Visual Analysis and Five Statistical Analyses in a Simple AB Single-Case Research Design , 2006, Behavior modification.

[6]  Rumen Manolov,et al.  Factors affecting visual analysis in single-case designs , 2009 .

[7]  Rumen Manolov,et al.  Factors Affecting Visual Inference in Single-Case Designs , 2009, The Spanish journal of psychology.

[8]  Matthew P. Normand,et al.  The Effects of Celeration Lines on Visual Data Analysis , 2006, Behavior modification.

[9]  Mark Carter,et al.  Reconsidering Overlap-Based Measures for Quantitative Synthesis of Single-Subject Data , 2013, Behavior modification.

[10]  Michael E Kelley,et al.  Using modified visual-inspection criteria to interpret functional analysis outcomes. , 2013, Journal of applied behavior analysis.

[11]  T. Matyas,et al.  Visual analysis of single-case time series: Effects of variability, serial dependence, and magnitude of intervention effects. , 1990, Journal of applied behavior analysis.

[12]  Kimberly J. Vannest,et al.  Meta-analysis of Single-Case Research on Teaching Functional Living Skills to Individuals with ASD , 2015 .

[13]  Kenneth J. Ottenbacher,et al.  COMPARISON OF VISUAL INSPECTION AND STATISTICAL ANALYSIS OF SINGLE‐SUBJECT DATA IN REHABILITATION RESEARCH , 1998, American journal of physical medicine & rehabilitation.

[14]  S. Thompson,et al.  Quantifying heterogeneity in a meta‐analysis , 2002, Statistics in medicine.

[15]  K J Ottenbacher,et al.  Interrater reliability of therapists' judgements of graphed data. , 1991, Physical therapy.

[16]  Gene S Fisch,et al.  Evaluating data from behavioral analysis: visual inspection or statistical models? , 2001, Behavioural Processes.

[17]  Justin D. Smith,et al.  Single-case experimental designs: a systematic review of published research and current standards. , 2012, Psychological methods.

[18]  A. Deprospero,et al.  Inconsistent visual analyses of intrasubject data. , 1979, Journal of applied behavior analysis.

[19]  Timothy J. Lewis,et al.  Analysis of Effect for Single-Case Design Research , 2013 .

[20]  B. Gorman,et al.  Design and Analysis of Single-Case Research , 1996 .

[21]  C. L. Spirrison,et al.  Acceptability bias: The effects of treatment acceptability on visual analysis of graphed data , 1994 .

[22]  R. Skiba,et al.  Influence Of Trend Estimation And Subject Familiarity On Practitioners' Judgments Of Intervention Effectiveness , 1989 .

[23]  Robert H. Horner,et al.  Single-Case Intervention Research Design Standards , 2013 .

[24]  Kenneth J. Ottenbacher.  When Is A Picture Worth A Thousand $Rh Values? A Comparison Of Visual And Quantitative Methods To Analyze Single Subject Data , 1990 .

[25]  Paul S. Smith,et al.  Teaching Visual Analysis of Time Series Data , 1996, Behavioural and Cognitive Psychotherapy.

[26]  A. Kazdin,et al.  Artifact, bias, and complexity of assessment: the ABCs of reliability. , 1977, Journal of applied behavior analysis.

[27]  M Perone,et al.  Statistical inference in behavior analysis: Experimental control is better , 1999, The Behavior analyst.

[28]  Amy M. Briesch,et al.  An Application of the What Works Clearinghouse Standards for Evaluating Single-Subject Research , 2013 .

[29]  Kenneth J. Ottenbacher,et al.  Characteristics Influencing the Visual Analysis of Single-Subject Data: An Empirical Analysis , 1988 .

[30]  Kyong-Mee Chung,et al.  Consistent visual analyses of intrasubject data. , 2010, Journal of applied behavior analysis.

[31]  V. Sierra,et al.  Assessing Functional Relations in Single-Case Designs , 2014, Behavior modification.

[32]  B. Wampold,et al.  Intervention effects and relative variation as dimensions in experts' use of visual inference. , 1982, Journal of applied behavior analysis.

[33]  Kimberly J. Vannest,et al.  Chapter 5 Synthesizing Single-Case Research to Identify Evidence-Based Treatments , 2013 .

[34]  A. House.  Measures of interobserver agreement: Calculation formulas and distribution effects , 1981 .

[35]  Sterett H. Mercer,et al.  The impact of baseline trend control on visual analysis of single-case data. , 2012, Journal of school psychology.

[36]  B. Iwata,et al.  Toward the development of structured criteria for interpretation of functional analysis data. , 1997, Journal of applied behavior analysis.

[37]  B. Loyd,et al.  Visual Analysis of Student Evaluation Data: A Qualitative Analysis of Teacher Decision Making , 1989 .

[38]  Donald M. Baer,et al.  The Visual Analysis of Data, and Current Research into the Stimuli Controlling It , 2015 .

[39]  Joel R. Levin,et al.  Meta- and statistical analysis of single-case intervention research data: quantitative gifts and a wish list. , 2014, Journal of school psychology.

[40]  T. Matyas,et al.  The Effect of Serial Dependence on Visual Judgment of Single-Case Charts: An Addendum , 1990 .

[41]  Bruce E. Wampold,et al.  The heuristics of visual inference , 1981 .

[42]  K. Ottenbacher,et al.  Empirical investigation of visual-inspection versus trend-line analysis of single-subject data. , 1988, Physical therapy.

[43]  R. Senf.  Tactics of Scientific Research: Evaluating Experimental Data in Psychology , 1962 .

[44]  K J Ottenbacher,et al.  Visual inspection of single-subject data: an empirical analysis. , 1990, Mental retardation.

[45]  D. Altman,et al.  Measuring inconsistency in meta-analyses , 2003, BMJ : British Medical Journal.

[46]  Matthew K. Burns,et al.  Meta-Analysis of Single-Case Design Research: Introduction to the Special Issue , 2012 .

[47]  R. R. Jones,et al.  Effects of serial dependency on the agreement between visual and statistical inference. , 1978, Journal of applied behavior analysis.

[48]  K J Ottenbacher,et al.  Trend line influence on visual analysis of single-subject data in rehabilitation research. , 1991, International disability studies.

[49]  D. Bailey,et al.  Effects of lines of progress and semilogarithmic charts on ratings of charted data. , 1984, Journal of applied behavior analysis.

[50]  D M Baer,et al.  "Perhaps it would be better not to know everything.". , 1977, Journal of applied behavior analysis.

[51]  G. Franklin.  Graphical Display and Visual Analysis , 2014 .

[52]  F. Symons,et al.  A Survey Evaluation of the Reliability of Visual Inspection and Functional Analysis Graphs , 2008, Behavior modification.

[53]  David L. Gast,et al.  Single subject research methodology in behavioral sciences , 2010 .

[54]  Kimberly J. Vannest,et al.  Bottom-Up Analysis of Single-Case Research Designs , 2012 .

[55]  J. Rojahn,et al.  The linear regression line as a judgmental aid in visual analysis of serially dependent A-B time-series data , 1985 .

[56]  Robert Gaylord-Ross,et al.  Visual Inspection and Statistical Analysis in Single-Case Designs , 1990 .

[57]  Brian Reichow,et al.  Visual analysis of multiple baseline across participants graphs when change is delayed. , 2010 .

[58]  Robert H. Horner,et al.  Single-Case Designs Technical Documentation. , 2010 .

[59]  K. Ottenbacher,et al.  An Empirical Investigation of Interrater Agreement for Single-Subject Data Using Graphs with and without Trend Lines , 1991 .

[60]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[61]  K J Ottenbacher,et al.  Interrater agreement of visual analysis in single-subject decisions: quantitative review and analysis. , 1993, American journal of mental retardation : AJMR.

[62]  Robert H. Hariharan George Keith Horner,et al.  Considerations for the Systematic Analysis and Use of Single-Case Research , 2012 .