Detection of Aberrant Responding on a Personality Scale in a Military Sample: an Application of Evaluating Person Fit with Two-level Logistic Regression Two-level Logistic Regression for Person Fit the Person Response Function

Person-fit assessment is used to identify persons who respond aberrantly to a test or questionnaire. In this study, S. P. Reise's (2000) method for evaluating person fit using 2-level logistic regression was applied to 13 personality scales of the Schedule for Nonadaptive and Adaptive Personality (SNAP; L. Clark, 1996) that had been administered to military recruits (N = 2,026). Results revealed significant person-fit heterogeneity and indicated that for 5 SNAP scales (Disinhibition, Entitlement, Exhibitionism, Negative Temperament, and Workaholism), the scale was more discriminating for some people than for others. Possible causes of aberrant responding were explored with several covariates. On all 5 scales, severe pathology emerged as a key influence on responses, and there was evidence of differential test functioning with respect to gender, ethnicity, or both. Other potential sources of aberrancy were carelessness, haphazard responding, or uncooperativeness. Social desirability was not as influential as expected.

[1]  P. Fayers.  Item Response Theory for Psychologists , 2004, Quality of Life Research.

[2]  A. Agresti.  An introduction to categorical data analysis , 1997 .

[3]  K. Ryan.  Methods for identifying biased test items , 1997 .

[4]  Steven P. Reise,et al.  Scoring Method and the Detection of Person Misfit in a Personality Assessment Context , 1995 .

[5]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[6]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[7]  Wim Jansen,et al.  Multilog: Multiple, Categorical Item Analysis and Test Scoring Using Item Response Theory , 1994 .

[8]  Klaas Sijtsma,et al.  The person response function as a tool in person-fit research , 2001 .

[9]  James Lumsden.  Tests are perfectly reliable , 1978 .

[10]  Rob R. Meijer,et al.  Person-Fit Research: An Introduction , 1996 .

[11]  S. Reise,et al.  How many IRT parameters does it take to model psychopathology items? , 2003, Psychological methods.

[12]  D. Dougherty,et al.  Personality judgment and personality pathology: self-other agreement in adolescents with conduct disorder. , 2007, Journal of personality.

[13]  David J. Weiss,et al.  The Person Response Curve: Fit of Individuals to Item Response Theory Models , 1983 .

[14]  Roel Bosker,et al.  Multilevel analysis : an introduction to basic and advanced multilevel modeling , 1999 .

[15]  Klaas Sijtsma,et al.  Methodology Review: Evaluating Person Fit , 2001 .

[16]  William Stout,et al.  A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF , 1993 .

[17]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[18]  Carol M Woods,et al.  Ramsay-curve item response theory (RC-IRT) to detect and correct for nonnormal latent variables. , 2006, Psychological methods.

[19]  R. Reitan.  Trail Making Test: Manual for Administration and Scoring , 1992 .

[20]  R. McCrae,et al.  On the invalidity of validity scales: evidence from self-reports and observer ratings in volunteer samples. , 2000, Journal of personality and social psychology.

[21]  Auke Tellegen,et al.  The Analysis of Consistency in Personality Assessment , 1988 .

[22]  Rob R. Meijer,et al.  A Comparison of the Person Response Function and the lz Person-Fit Statistic , 1998 .

[23]  Eric Turkheimer,et al.  Perceptions of Self and Others Regarding Pathological Personality Traits. , 2006 .

[24]  D. Watson,et al.  Journal of Personality and Social Psychology Self-other Agreement in Personality and Affectivity: the Role of Acquaintanceship, Trait Visibility, and Assumed Similarity , 2022 .

[25]  Dc Washington.  Diagnostic and Statistical Manual of Mental Disorders, 4th Ed. , 1994 .

[26]  S. Reise,et al.  Using Multilevel Logistic Regression to Evaluate Person-Fit in IRT Models , 2000, Multivariate behavioral research.

[27]  S. Stouffer,et al.  Measurement and Prediction , 1954 .

[28]  H. Wainer,et al.  Differential Item Functioning. , 1994 .

[29]  Eric Turkheimer,et al.  Factorial structure of pathological personality as evaluated by peers. , 2003, Journal of abnormal psychology.

[30]  B. Zumbo,et al.  An Empirical Test of Roskam's Conjecture about the Interpretation of an ICC Parameter in Personality Inventories , 1997 .

[31]  L. Shepard,et al.  Methods for Identifying Biased Test Items , 1994 .

[32]  Steven P. Reise,et al.  Assessing Person-Fit on Measures of Typical Performance , 1996 .

[33]  Klaas Sijtsma,et al.  Detection of Aberrant Item Score Patterns: A Review of Recent Developments. Research Report 94-8. , 1994 .

[34]  David Thissen,et al.  Item Response Theory with Estimation of the Latent Population Distribution Using Spline-Based Densities , 2006, Psychometrika.

[35]  Steven P. Reise,et al.  Traitedness and the assessment of response pattern scalability , 1993 .

[36]  R. Maruyama,et al.  On Test Scoring , 1927 .

[37]  David J. Weiss,et al.  Book Review : New Horizons in Testing: Latent Trait Test Theory and Computerized Adaptive Testing David J. Weiss (Ed.) New York: Academic Press, 1983, 345 pp., $35.00 , 1984 .

[38]  Lee Anna Clark,et al.  Assessment and diagnosis of personality disorder: perennial issues and an emerging reconceptualization. , 2007, Annual review of psychology.

[39]  Carol M. Woods.  Ramsay Curve IRT for Likert-Type Data , 2007 .

[40]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[41]  D. Collett,et al.  Modeling Binary Data. , 1993 .

[42]  Stephen G West,et al.  What do you learn about someone over time? The relationship between length of acquaintance and consensus and self-other agreement in judgments of personality. , 2007, Journal of personality and social psychology.

[43]  David W. Hosmer,et al.  Applied Logistic Regression , 1991 .

[44]  Howard T. Everson,et al.  Methodology Review: Statistical Approaches for Assessing Measurement Bias , 1993 .

[45]  D. Klein,et al.  Patients' versus informants' reports of personality disorders in predicting 7 1/2-year outcome in outpatients with depressive disorders. , 2003, Psychological assessment.

[46]  Eric Turkheimer,et al.  Traits associated with personality disorders and adjustment to military life: predictive validity of self and peer reports. , 2004, Military medicine.

[47]  C. Morris.  Parametric Empirical Bayes Inference: Theory and Applications , 1983 .

[48]  P. Pilkonis,et al.  Self- and other-reports of traits from the five-factor model: relations to personality disorder. , 2005, Journal of personality disorders.

[49]  Carol M. Woods.  Consequences of Ignoring Guessing When Estimating the Latent Density in Item Response Theory , 2008 .

[50]  Carol M Woods,et al.  Monte Carlo Evaluation of Two-Level Logistic Regression for Assessing Person Fit , 2008, Multivariate behavioral research.

[51]  Klaas Sijtsma,et al.  Detection and Validation of Unscalable Item Score Patterns Using Item Response Theory: An Illustration with Harter's Self-Perception Profile for Children , 2008, Journal of personality assessment.