A Comparative Study of Test Data Dimensionality Assessment Procedures Under Nonparametric IRT Models

In this article, an overview of nonparametric item response theory methods for determining the dimensionality of item response data is provided. Four methods were considered: MSP, DETECT, HCA/CCPROX, and DIMTEST. First, the methods were compared theoretically. Second, a simulation study was done to compare the effectiveness of MSP, DETECT, and HCA/CCPROX using the default settings of each program in finding a simulated dimensional structure of a matrix of item response data. In several design cells, the methods that use covariances conditional on the latent trait (DETECT and HCA/CCPROX) were superior in finding the simulated structure to the method that used normed unconditional covariances (MSP). Third, the correctness of the decision of accepting or rejecting unidimensionality based on the statistics used in DETECT and DIMTEST was considered. This decision did not always reflect the true dimensionality of the item pool. Index terms: DETECT software and method, dimensionality of item response data, DIMTEST software and method, HCA/CCPROX software and method, MSP software and method, multidimensional item response data, nonparametric item response theory, unidimensional item response data.

[1]  David Andrich,et al.  9 – A Latent-Trait Model for Items with Response Dependencies: Implications for Test Construction and Analysis* , 1985 .

[2]  B. Junker.  Conditional association, essential independence and monotone unidimensional Item response models , 1993 .

[3]  L. A. Pervin.  Handbook of Personality: Theory and Research , 1992 .

[4]  Roger W. Johnson,et al.  An Introduction to the Bootstrap , 2001 .

[5]  Thomas J. Reynolds,et al.  CDAscal: An algorithm for assessing the correspondence of one or more vectors to a symmetric matrix using ordinal regression , 1987 .

[6]  Klaus R. Scherer,et al.  On the Sequential Nature of Appraisal Processes: Indirect Evidence from a Recognition Task , 1999 .

[7]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[8]  P. Costa,et al.  Facet Scales for Agreeableness and Conscientiousness: A Revision of the NEO Personality Inventory☆ , 1991 .

[9]  D. Thissen,et al.  Local Dependence Indexes for Item Pairs Using Item Response Theory , 1997 .

[10]  Karen Caplovitz Barrett,et al.  A functionalist approach to shame and guilt. , 1995 .

[11]  R. D. Bock,et al.  Marginal maximum likelihood estimation of item parameters , 1982 .

[12]  Mark D. Reckase,et al.  A Linear Logistic Multidimensional Model for Dichotomous Item Response Data , 1997 .

[13]  Daniel O. Segall,et al.  General ability measurement: An application of multidimensional item response theory , 2001 .

[14]  P. Diggle,et al.  Analysis of Longitudinal Data , 2003 .

[15]  Ivo W. Molenaar,et al.  Some improved diagnostics for failure of the Rasch model , 1983 .

[16]  William Stout,et al.  The theoretical detect index of dimensionality and its application to approximate simple structure , 1999 .

[17]  Furong Gao,et al.  Using Resampling Methods to Produce an Improved DIMTEST Procedure , 2001 .

[18]  Susan E. Whitely,et al.  Measuring Aptitude Processes with Multicomponent Latent Trait Models. , 1981 .

[19]  R. Baumeister,et al.  Guilt: an interpersonal approach. , 1994, Psychological bulletin.

[20]  Garth J. O. Fletcher,et al.  Love, Hate, Anger, and Jealousy in Close Relationships: A Prototype and Cognitive Appraisal Analysis , 1993 .

[21]  Craig A. Smith,et al.  Appraisal theory: Overview, assumptions, varieties, controversies. , 2001 .

[22]  P. Costa,et al.  Validation of the five-factor model of personality across instruments and observers. , 1987, Journal of personality and social psychology.

[23]  Francis Tuerlinckx,et al.  A nonlinear mixed model framework for item response theory. , 2003, Psychological methods.

[24]  Melvin R. Novick,et al.  Some latent train models and their use in inferring an examinee's ability , 1966 .

[25]  Craig A. Smith,et al.  From appraisal to emotion: Differences among unpleasant feelings , 1988 .

[26]  Susan E. Whitely,et al.  Multicomponent latent trait models for ability tests , 1980 .

[27]  B. Everitt,et al.  Applied Multivariate Data Analysis. , 1993 .

[28]  William Stout,et al.  A nonparametric approach for assessing latent trait unidimensionality , 1987 .

[29]  The Impact of Conditional Scores on the Performance of DETECT. , 2003 .

[30]  Robert J. Mokken,et al.  A Theory and Procedure of Scale Analysis. , 1973 .

[31]  F. Tuerlinckx,et al.  Distinguishing Constant and Dimension-Dependent Interaction: A Simulation Study , 1999 .

[32]  N. Frijda.  The place of appraisal in emotion , 1993 .

[33]  Discriminating emotions from appraisal-relevant situational information: Baseline data for structural models of cognitive appraisals , 1993 .

[34]  L. Fahrmeir,et al.  Multivariate statistical modelling based on generalized linear models , 1994 .

[35]  P. Costa,et al.  Domains and facets: hierarchical personality assessment using the revised NEO personality inventory. , 1995, Journal of personality assessment.

[36]  J. Loevinger,et al.  The technic of homogeneous tests compared with some aspects of scale analysis and factor analysis. , 1948, Psychological bulletin.

[37]  C. Barbaranelli,et al.  Facing guilt: Role of negative affectivity, need for reparation, and fear of punishment in leading to prosocial behaviour and aggression , 2001 .

[38]  J. Tangney,et al.  Shame and guilt in interpersonal relationships. , 1995 .

[39]  S. Embretson.  Test design : developments in psychology and psychometrics , 1985 .

[40]  W. Mischel,et al.  A cognitive-affective system theory of personality: reconceptualizing situations, dispositions, dynamics, and invariance in personality structure. , 1995, Psychological review.

[41]  D. Grayson,et al.  Two-group classification in latent trait theory: Scores with monotone likelihood ratio , 1988 .

[42]  Donald Hedeker,et al.  Full-information item bi-factor analysis , 1992 .

[43]  Hirotugu Akaike,et al.  On entropy maximization principle , 1977 .

[44]  Charles Lewis,et al.  A Nonparametric Approach to the Analysis of Dichotomous Item Responses , 1982 .

[45]  W. Kempf.  Dynamic Models for the Measurement of "Traits" in Social Behavior , 1977 .

[46]  F. Baker,et al.  Item response theory : parameter estimation techniques , 1993 .

[47]  Fumiko Samejima,et al.  Acceleration model in the heterogeneous case of the general graded response model , 1995 .

[48]  P. Rosenbaum,et al.  Conditional Association and Unidimensionality in Monotone Latent Variable Models , 1985 .

[49]  H. Swaminathan,et al.  An Assessment of Stout's Index of Essential Unidimensionality , 1996 .

[50]  Andrew Ortony,et al.  The Cognitive Structure of Emotions , 1988 .

[51]  Terry Ackerman,et al.  Graphical Representation of Multidimensional Item Response Theory Analyses , 1996 .

[52]  A. Ortony,et al.  What's basic about basic emotions? , 1990, Psychological review.

[53]  Craig A. Smith,et al.  Appraisal components, core relational themes, and the emotions , 1993 .

[54]  Hae-Rim Kim.  New techniques for the dimensionality assessment of standardized test data , 1994 .

[55]  P. Costa,et al.  Personality trait structure as a human universal. , 1997, The American psychologist.

[56]  Kurt W. Fischer,et al.  Self-conscious emotions: The psychology of shame, guilt, embarrassment, and pride. , 1995 .

[57]  N. Frijda,et al.  Appraisal: What is the dependent? , 2001 .

[58]  S. Embretson.  A general latent trait model for response processes , 1984 .

[59]  P. Costa,et al.  Rotation to Maximize the Construct Validity of Factors in the NEO Personality Inventory. , 1989, Multivariate behavioral research.

[60]  K. Scherer.  Profiles of Emotion-antecedent Appraisal: Testing Theoretical Predictions across Cultures , 1997 .

[61]  Cees A. W. Glas,et al.  Testing the Rasch Model , 1995 .

[62]  William Stout,et al.  Conditional covariance structure of generalized compensatory multidimensional items , 1999 .

[63]  B. Junker,et al.  Factor composition of the Suicide Intent Scale. , 1993, Suicide & life-threatening behavior.

[64]  William F. Strout.  A new item response theory modeling approach with applications to unidimensionality assessment and ability estimation , 1990 .

[65]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[66]  Klaas Sijtsma,et al.  Rejoinder to "The Mokken Scale: A Critical Discussion" , 1986 .

[67]  Robert J. Jannarone,et al.  Conjunctive item response theory kernels , 1986 .

[68]  B. L. Omdahl,et al.  Cognitive appraisal, emotion, and empathy , 1995 .

[69]  Eric T. Bradlow,et al.  A Bayesian random effects model for testlets , 1999 .

[70]  Jinming Zhang.  Some fundamental issues in item response theory with applications , 1996 .

[71]  Ratna Nandakumar,et al.  Refinements of Stout’s Procedure for Assessing Latent Trait Unidimensionality , 1993 .

[72]  P. Gilbert,et al.  The phenomenology of shame and guilt: an empirical investigation. , 1994, The British journal of medical psychology.

[73]  William Stout,et al.  Using New Proximity Measures With Hierarchical Cluster Analysis to Detect Multidimensionality , 1998 .

[74]  Raymond J. Adams,et al.  Rasch models for item bundles , 1995 .

[75]  Todd F. Heatherton,et al.  Interpersonal aspects of guilt: Evidence from narrative studies. , 1995 .

[76]  K. Scherer,et al.  Appraisal processes in emotion: Theory, methods, research. , 2001 .

[77]  Francis Tuerlinckx,et al.  Measuring needs with the thematic apperception test: a psychometric study. , 2002, Journal of personality and social psychology.

[78]  C. Izard,et al.  Four systems for emotion activation: cognitive and noncognitive processes. , 1993, Psychological review.

[79]  Stanley Wasserman,et al.  Mathematical Models for Social Psychology. , 1979 .

[80]  M. Petersen,et al.  Introduction to Nonparametric Item Response Theory , 2005, Quality of Life Research.

[81]  Carl P. M. Rijkes,et al.  Loglinear multidimensional IRT models for polytomously scored items , 1988 .

[82]  Klaas Sijtsma,et al.  Selection of Unidimensional Scales From a Multidimensional Item Bank in the Polytomous Mokken I RT Model , 1995 .

[83]  Roderick P. McDonald,et al.  Linear Versus Models in Item Response Theory , 1982 .

[84]  J. Rivera,et al.  Differentiating guilt and shame and their effects on motivation. , 1995 .

[85]  F. W. Wicker,et al.  Participant descriptions of guilt and shame , 1983 .

[86]  Wendy M. Yen,et al.  Scaling Performance Assessments: Strategies for Managing Local Item Dependence , 1993 .

[87]  J. Ware,et al.  Applications of Statistics , 1978 .

[88]  N. Frijda,et al.  Relations among emotion, appraisal, and emotional action readiness , 1989 .

[89]  Brian W. Junker,et al.  Stochastic ordering using the latent trait and the sum score in polytomous IRT models , 1997 .

[90]  J. Long,et al.  Covariance structure models , 1983 .

[91]  Paul De Boeck,et al.  A parametric model for local dependence among test items. , 1997 .

[92]  J. Douglas,et al.  LSAT Dimensionality Analysis for the December 1991, June 1992, and October 1992 Administrations. Statistical Report. LSAC Research Report Series. , 1999 .

[93]  R. D. Bock,et al.  Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm , 1981 .

[94]  Hua-Hua Chang,et al.  DIMTEST: A Fortran Program for Assessing Dimensionality of Binary Item Responses , 1992 .

[95]  David Thissen,et al.  Data analysis using item response theory. , 1988 .

[96]  A. Béguin,et al.  MCMC estimation and some model-fit analysis of multidimensional IRT models , 2001 .

[97]  Douglas M. Hawkins,et al.  Interactive LISREL : user's guide , 2001 .

[98]  Ira J. Roseman.  Appraisal Determinants of Emotions: Constructing a More Accurate and Comprehensive Theory , 1996 .

[99]  Herbert W. Marsh,et al.  Self Description Questionnaire III: The construct validity of multidimensional self-concept ratings by late adolescents. , 1984 .

[100]  Brian Habing,et al.  Conditional Covariance-Based Nonparametric Multidimensionality Assessment , 1996 .

[101]  Mark D. Reckase,et al.  The Discriminating Power of Items That Measure More Than One Dimension , 1991 .

[102]  Georg Rasch,et al.  Probabilistic Models for Some Intelligence and Attainment Tests , 1981, The SAGE Encyclopedia of Research Design.

[103]  Fumiko Samejima,et al.  Logistic positive exponent family of models: Virtue of asymmetric item characteristic curves , 2000 .

[104]  David Thissen,et al.  Trace Lines for Testlets: A Use of Multiple-Categorical-Response Models. , 1989 .

[105]  D. Knol,et al.  Empirical Comparison Between Factor Analysis and Multidimensional Item Response Models. , 1991, Multivariate behavioral research.