On the Relationship Between Maximal Reliability and Maximal Validity of Linear Composites

A linear combination of a set of measures is often sought as an overall score summarizing subject performance. The weights in this composite can be selected to maximize its reliability or to maximize its validity, and the optimal choice of weights is in general not the same for these two optimality criteria. We explore several relationships between the resulting reliability and validity estimates in different situations. Only in the case of congeneric tests are maximal reliability and maximal validity attained with the same weights, and a precise relationship between these two maximality concepts can be derived. A widely and readily applicable procedure for point and interval estimation of maximal validity is also outlined. Several inequalities are established for the case when the measures are not congeneric.

[1]  K. Jöreskog.  Statistical analysis of sets of congeneric tests , 1971 .

[2]  Heng Li,et al.  The Resolution of Some Paradoxes Related to Reliability and Validity , 2003 .

[3]  Peter M. Bentler,et al.  EQS : structural equations program manual , 1989 .

[4]  B. Green.  THE ORTHOGONAL APPROXIMATION OF AN OBLIQUE STRUCTURE IN FACTOR ANALYSIS , 1952 .

[5]  Heng Li.  A unifying expression for the maximal reliability of a linear composite , 1997 .

[6]  Lawrence M. Rudner.  Informed Test Component Weighting , 2005 .

[7]  T Raykov,et al.  Estimation of congeneric scale reliability using covariance structure analysis with nonlinear constraints. , 2001, The British journal of mathematical and statistical psychology.

[8]  Maximally Reliable Composites for Unidimensional Measures , 1980 .

[9]  Tenko Raykov,et al.  Estimation of maximal reliability: a note on a covariance structure modelling approach. , 2004, The British journal of mathematical and statistical psychology.

[10]  L. Cronbach,et al.  Time-limit tests: Estimating their reliability and degree of speeding , 1951, Psychometrika.

[11]  David J. Bartholomew,et al.  The Statistical Approach to Social Measurement , 1996 .

[12]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[13]  Donald B. Rubin,et al.  Reliability of measurement in psychology: From Spearman-Brown to maximal reliability. , 1996 .

[14]  J. Magnus,et al.  Matrix Differential Calculus with Applications in Statistics and Econometrics , 1991 .

[15]  Calyampudi R. Rao,et al.  Linear Statistical Inference and Its Applications. , 1975 .

[16]  Karl G. Jöreskog,et al.  Lisrel 8: User's Reference Guide , 1997 .

[17]  Ke-Hai Yuan,et al.  On robusiness of the normal-theory based asymptotic distributions of three reliability coefficient estimates , 2002 .

[18]  A note on the calculation of weights for maximum battery reliability , 1950 .

[19]  A Direct Method for Obtaining Approximate Standard Error and Confidence Interval of Maximal Reliability for Composites With Congeneric Measures , 2006, Multivariate behavioral research.

[20]  G. Thomson.  WEIGHTING FOR BATTERY RELIABILITY AND PREDICTION , 1940 .

[21]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[22]  Jiabo Li,et al.  Efficient evaluation of the algebrants of VB wave functions using the successive expansion method. I. SpinS = 0, , 1997 .

[23]  R. P. McDonald,et al.  Test Theory: A Unified Treatment , 1999 .

[24]  N. L. Johnson,et al.  Linear Statistical Inference and Its Applications , 1966 .