Dependability of Measurement in Counseling Psychology

Classical approaches to the assessment of reliability neglect to take into account multiple sources of error and to consider diverse measurement contexts. Generalizability theory (GT) offers a flexible framework for assessing dependability of measurement. With GT, investigators can estimate the total proportion of variance in ratings that is due to error rather than focusing on one source of error at a time. Simultaneous consideration of multiple sources of error allows investigators to assess the overall impact of measurement error in terms of attenuation of study findings and reduction of statistical power. Estimation of variance components allows for flexible application of findings to a variety of possible future research designs. Illustrative analyses demonstrate the special advantages of GT for planning studies in which observer ratings will be used.

[1]  R. Downey,et al.  Rating the ratings: Assessing the psychometric quality of rating data , 1980 .

[2]  S. K. Mitchell.  Interobserver agreement, reliability, and generalizability of data collected in observational studies. , 1979 .

[3]  W. J. Dixon,et al.  BMDP-79, biomedical computer programs, P-series , 1981 .

[4]  Michael T. Kane,et al.  AN INDEX OF DEPENDABILITY FOR MASTERY TESTS , 1977 .

[5]  C. Hoyt.  Test reliability estimated by analysis of variance , 1941 .

[6]  J. Tukey,et al.  AVERAGE VALUES OF MEAN SQUARES IN FACTORIALS , 1956 .

[7]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[8]  Michael S. Robbins,et al.  Observational coding in family therapy process research. , 1995 .

[9]  D. Blustein,et al.  Developing and Using Educational and Psychological Tests and Measures: The Unificationist Perspective. , 1991 .

[10]  D. Weiss,et al.  Interrater reliability and agreement of subjective judgments , 1975 .

[11]  Joseph L. Fleiss,et al.  Balanced Incomplete Block Designs for Inter-Rater Reliability Studies , 1981 .

[12]  C. Spearman.  CORRELATION CALCULATED FROM FAULTY DATA , 1910 .

[13]  R. Conger,et al.  Predicting Marital Instability from Spouse and Observer Reports of Marital Interaction. , 1996 .

[14]  S. Maxwell.  Dependent Variable Reliability and Determination of Sample Size , 1980 .

[15]  I. Elkin,et al.  NIMH Treatment of Depression Collaborative Research Program. Background and research plan. , 1985, Archives of general psychiatry.

[16]  E. Thorndike.  A constant error in psychological ratings. , 1920 .

[17]  W. Hoyt.  Antecedents and Effects of Perceived Therapist Credibility: A Meta-Analysis. , 1996 .

[18]  M. Kendall.  Statistical Methods for Research Workers , 1937, Nature.

[19]  William T. Hoyt,et al.  Magnitude and moderators of bias in observer ratings: A meta-analysis. , 1999 .

[20]  L. Cronbach,et al.  Generalizability of scores influenced by multiple sources of variance , 1965, Psychometrika.

[21]  P. Shrout.  Analyzing consensus in personality judgments: a variance components approach. , 1993, Journal of personality.

[22]  G. Patterson,et al.  Social interactional processes within the family: The study of the moment-by-moment family transactions in which human social development is imbedded , 1984 .

[23]  I. Elkin.  The NIMH Treatment of Depression Collaborative Research Program: Where we began and where we are. , 1994 .

[24]  Eva Nick,et al.  The dependability of behavioral measurements: theory of generalizability for scores and profiles , 1973 .

[25]  J. Hawkins.  Spouse Differences in Communication Style: Preference, Perception, Behavior. , 1980 .

[26]  Noreen M. Webb,et al.  Using Generalizability Theory in Counseling and Development. , 1988 .

[27]  C. Spearman.  CORRELATIONS OF SUMS OR DIFFERENCES , 1913 .

[28]  James P. Guinee,et al.  Generalizability of Interpersonal Communications Rating Scale Ratings across Presentation Modes. , 1990 .

[29]  K. O’grady,et al.  A Method for Investigating Sources of Rater Bias. , 1988 .

[30]  John M. Gottman,et al.  Temporal Form: Toward a New Language for Describing Relationships , 1982 .

[31]  Noreen M. Webb,et al.  Generalizability theory : 1973-1980 , 1991 .

[32]  S. Maxwell,et al.  Implications of Recent Developments in Structural Equation Modeling for Counseling Psychology , 1999 .

[33]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[34]  M. Kane.  A Sampling Model for Validity , 1982 .

[35]  Richard J. Shavelson,et al.  Generalizability Theory: A Primer , 1991 .

[36]  Philip L. Smith.  Sampling Errors of Variance Components in Small Sample Multifacet Generalizability Studies , 1978 .

[37]  C. Hill,et al.  Assessing psychotherapy outcomes and processes. , 1994 .

[38]  H. Markman,et al.  Coding Marital and Family Interaction , 1987 .

[39]  Susy Macqueen,et al.  Validity , 1973, Just Algorithms.

[40]  S. Strong.  Counseling: An interpersonal influence process. , 1968 .

[41]  F. Schmidt,et al.  Measurement Error in Psychological Research: Lessons From 26 Research Scenarios , 1996 .

[42]  C. Hill,et al.  Trends in psychotherapy process research: samples, measures, researchers, and classic publications , 1994 .

[43]  E. Hetherington.  Siblings, family relationships, and child development: Introduction. , 1994 .

[44]  L. Cronbach,et al.  THEORY OF GENERALIZABILITY: A LIBERALIZATION OF RELIABILITY THEORY† , 1963 .

[45]  N. Gonzales,et al.  Interobserver agreement in the assessment of parental behavior and parent-adolescent conflict: African American mothers, daughters, and independent observers. , 1996, Child development.

[46]  A. J. Conger,et al.  Self-ratings and judges' ratings of heterosexual social anxiety and skill: a generalizability study. , 1979, Journal of consulting and clinical psychology.