Using G-Theory to Enhance Evidence of Reliability and Validity for Common Uses of the Paulhus Deception Scales

We applied a new approach to Generalizability theory (G-theory) involving parallel splits and repeated measures to evaluate common uses of the Paulhus Deception Scales based on polytomous and four types of dichotomous scoring. G-theory indices of reliability and validity accounting for specific-factor, transient, and random-response measurement error supported use of polytomous over dichotomous scores as contamination checks; as control, explanatory, and outcome variables; as aspects of construct validation; and as indexes of environmental effects on socially desirable responding. Polytomous scoring also provided results for flagging faking as dependable as those when using dichotomous scoring methods. These findings argue strongly against the nearly exclusive use of dichotomous scoring for the Paulhus Deception Scales in practice and underscore the value of G-theory in demonstrating this. We provide guidelines for applying our G-theory techniques to other objectively scored clinical assessments, for using G-theory to estimate how changes to a measure might improve reliability, and for obtaining software to conduct G-theory analyses free of charge.

[1]  Delroy L. Paulhus,et al.  Self-presentation of personality: An agency-communion framework. , 2008 .

[2]  Christopher Mushquash,et al.  SPSS and SAS programs for generalizability theory analyses , 2006, Behavior research methods.

[3]  D. Paulhus.  Measurement and control of response bias. , 1991 .

[4]  T. Skilling,et al.  Moderators of informant agreement in the assessment of adolescent psychopathology: extension to a forensic sample. , 2012, Psychological assessment.

[5]  Samuel B. Green,et al.  Commentary on Coefficient Alpha: A Cautionary Tale , 2009 .

[6]  Richard K. Burdick,et al.  Confidence intervals on linear combinations of variance components that are unrestricted in sign , 1990 .

[7]  Eva Nick,et al.  The dependability of behavioral measurements: theory of generalizability for scores and profiles , 1973 .

[8]  Walter P. Vispoel,et al.  Psychometric properties for the Balanced Inventory of Desirable Responding: dichotomous versus polytomous conventional and IRT scoring. , 2014, Psychological assessment.

[9]  C. Melville,et al.  The Impact of Social Desirability Biases on Self-Report Among College Student and Problem Gamblers , 2008, Journal of Gambling Studies.

[10]  Chiungjung Huang,et al.  Relation between self-esteem and socially desirable responding and the role of socially desirable responding in the relation between self-esteem and performance , 2013 .

[11]  C. Spearman.  The proof and measurement of association between two things. , 2015, International journal of epidemiology.

[12]  Julie A. Schumacher,et al.  Social Desirability and Partner Agreement of Men’s Reporting of Intimate Partner Violence in Substance Abuse Treatment Settings , 2015, Journal of interpersonal violence.

[13]  Walter P. Vispoel,et al.  Response Biases and Their Relation to Sex Differences in Multiple Domains of Self-Concept , 2000 .

[14]  Knut-Andreas Christophersen,et al.  A generalizability study of the Norwegian version of KINDLR in a sample of healthy adolescents , 2008, Quality of Life Research.

[15]  Walter P. Vispoel,et al.  A generalizability analysis of score consistency for the Balanced Inventory of Desirable Responding. , 2013, Psychological assessment.

[16]  Thomas J. Dunn,et al.  From alpha to omega: a practical solution to the pervasive problem of internal consistency estimation. , 2014, British journal of psychology.

[17]  B. Byrne.  Measuring Self-Concept Across the Life Span: Issues and Instrumentation , 1996 .

[18]  David E. Wiley,et al.  Socially Desirable Responding: The Evolution of a Construct , 2001 .

[19]  C. Reeve,et al.  Estimation of Transient Error in Cognitive Ability Scales , 2005 .

[20]  R. Linn.  Educational measurement, 3rd ed. , 1989 .

[21]  Donald B. Rubin,et al.  The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. , 1974 .

[22]  Klaas Sijtsma,et al.  On the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s Alpha , 2008, Psychometrika.

[23]  Nigel O'Brian,et al.  Generalizability Theory I , 2003 .

[24]  R. Holden,et al.  A model of personality test item response dissimulation. , 1992 .

[25]  N. Schmitt.  Uses and abuses of coefficient alpha. , 1996 .

[26]  Jochen Musch,et al.  Comparing Continuous and Dichotomous Scoring of the Balanced Inventory of Desirable Responding , 2002, Journal of personality assessment.

[27]  Noreen M. Webb,et al.  Using Generalizability Theory in Counseling and Development. , 1988 .

[28]  L. Crocker,et al.  Introduction to Classical and Modern Test Theory , 1986 .

[29]  Marise Ph. Born,et al.  The Role of Constructs in Psychological and Educational Measurement , 2002 .

[30]  Delroy L. Paulhus,et al.  Balanced Inventory of Desirable Responding , 2012 .

[31]  G. Bonanno,et al.  Rasch Modeling of the Self-Deception Scale of the Balanced Inventory of Desirable Responding , 2009 .

[32]  A. L. Edwards.  The social desirability variable in personality assessment and research , 1958 .

[33]  Simine Vazire,et al.  The self-report method. , 2007 .

[34]  G Becker,et al.  How important is transient error in estimating reliability? Going beyond simulation studies. , 2000, Psychological methods.

[35]  Huy Le,et al.  Beyond alpha: an empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs. , 2003, Psychological methods.

[36]  Noreen M. Webb,et al.  The generalizability of test scores. , 2013 .

[37]  Richard J. Shavelson,et al.  Generalizability Theory: A Primer , 1991 .

[38]  Caoilte Ó Ciardha,et al.  Community Males Show Multiple-Perpetrator Rape Proclivity , 2014, Sexual abuse : a journal of research and treatment.

[39]  D. Schretlen,et al.  A psychological test battery to detect prison inmates who fake insanity or mental retardation , 1990 .

[40]  W. Nugent.  A Psychometric Study of the MPSI Suicidal Thoughts Subscale , 2006 .

[41]  P. Gendreau,et al.  Evaluating response set styles on the MMPI with prisoners: Faking good adjustment and maladjustment. , 1973 .

[42]  A. Hess,et al.  The fakability of subtle and obvious measures of aggression by male prisoners. , 1984, Journal of personality assessment.

[43]  Philip L. Smith.  Sampling Errors of Variance Components in Small Sample Multifacet Generalizability Studies , 1978 .

[44]  M. Greenberg,et al.  Validation of a needs‐based biopsychosocial distress instrument for cancer patients , 2012, Psycho-oncology.

[45]  Kristopher J Preacher,et al.  On the practice of dichotomization of quantitative variables. , 2002, Psychological methods.