Construct Validation in Social and Personality Research

The verity of results about a psychological construct hinges on the validity of its measurement, making construct validation a fundamental methodology to the scientific process. We reviewed a representative sample of articles published in the Journal of Personality and Social Psychology for construct validity evidence. We report that latent variable measurement, in which responses to items are used to represent a construct, is pervasive in social and personality research. However, the field does not appear to be engaged in best practices for ongoing construct validation. We found that validity evidence of existing and author-developed scales was lacking, with coefficient α often being the only psychometric evidence reported. We provide a discussion of why the construct validation framework is important for social and personality researchers and recommendations for improving practice.

[1]  R. Larsen,et al.  The Satisfaction with Life Scale , 1985, Journal of personality assessment.

[2]  Leland Wilkinson,et al.  Statistical Methods in Psychology Journals Guidelines and Explanations , 2005 .

[3]  S. Green,et al.  Coefficient Alpha: A Reliability Coefficient for the 21st Century? , 2011 .

[4]  Joey T. Cheng,et al.  Toward a Unified Science of Hierarchy: Dominance and Prestige are Two Fundamental Pathways to Human Social Rank , 2014 .

[5]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[6]  N. Schmitt.  Uses and abuses of coefficient alpha. , 1996 .

[7]  Brian A. Nosek,et al.  Promoting an open research culture , 2015, Science.

[8]  E. McColl.  Cognitive Interviewing. A Tool for Improving Questionnaire Design , 2006, Quality of Life Research.

[9]  Chris S. Hulleman,et al.  A meta-analytic review of achievement goal measures: different labels for the same constructs or different constructs with similar labels? , 2010, Psychological bulletin.

[10]  S. Sireci.  The Construct of Content Validity , 1998 .

[11]  M. Appelbaum,et al.  Psychometric methods. , 1989, Annual review of psychology.

[12]  L. Cronbach,et al.  Construct validity in psychological tests. , 1955, Psychological bulletin.

[13]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[14]  Jay J Van Bavel,et al.  Contextual sensitivity in scientific reproducibility , 2016, Proceedings of the National Academy of Sciences.

[15]  Stephen G West,et al.  Doctoral training in statistics, measurement, and methodology in psychology: replication and extension of Aiken, West, Sechrest, and Reno's (1990) survey of PhD programs in North America. , 2008, The American psychologist.

[16]  J. E. Kurtz,et al.  Internal Consistency, Retest Reliability, and Their Implications for Personality Scale Validity , 2011, Personality and social psychology review : an official journal of the Society for Personality and Social Psychology, Inc.

[17]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[18]  D. Borsboom,et al.  The concept of validity. , 2004, Psychological review.

[19]  Austin S Baldwin,et al.  Examining the construct validity of affective judgments of physical activity measures. , 2016, Psychological assessment.

[20]  Klaas Sijtsma,et al.  Playing with Data—Or How to Discourage Questionable Research Practices and Stimulate Researchers to Do Things Right , 2016, Psychometrika.

[21]  G. Cumming.  The New Statistics: Why and How , 2013 .

[22]  D. Eignor.  The standards for educational and psychological testing. , 2013 .

[23]  D. Watson,et al.  Constructing validity: Basic issues in objective scale development , 1995 .

[24]  K. Petrides.  Introduction to Psychometric Theory , 2011 .

[25]  Klaas Sijtsma,et al.  On the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s Alpha , 2008, Psychometrika.

[26]  R. Eisinga,et al.  The reliability of a two-item scale: Pearson, Cronbach, or Spearman-Brown? , 2013, International Journal of Public Health.

[27]  J. Loevinger.  Objective Tests as Instruments of Psychological Theory , 1957 .

[28]  Jessica L. Tracy,et al.  The Jingle and Jangle of Emotion Assessment: Imprecise Measurement, Casual Scale Usage, and Conceptual Fuzziness in Emotion Research , 2017, Emotion.

[29]  Leif D. Nelson,et al.  Data from Paper “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant” , 2014 .

[30]  L. Crocker,et al.  Introduction to Classical and Modern Test Theory , 1986 .

[31]  Jeri Benson,et al.  Developing a Strong Program of Construct Validation: A Test Anxiety Example , 2005 .

[32]  Don A. Moore,et al.  Preregister if you want to. , 2016, The American psychologist.

[33]  J. Graham.  Congeneric and (Essentially) Tau-Equivalent Estimates of Score Reliability , 2006 .

[34]  S. Messick.  Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[35]  M. Strauss,et al.  Construct validity: advances in theory and methodology. , 2009, Annual review of clinical psychology.

[36]  M. Kane.  Validating the Interpretations and Uses of Test Scores , 2013 .

[37]  Hunter Gehlbach,et al.  Measure Twice, Cut down Error: A Process for Enhancing the Validity of Survey Scales , 2011 .

[38]  J. M. Cortina,et al.  What Is Coefficient Alpha? An Examination of Theory and Applications , 1993 .

[39]  Duane T. Wegener,et al.  Conceptualizing and evaluating the replication of research results , 2016 .

[40]  D. Watson,et al.  What is being assessed and why it matters: the impact of transient error on trait research. , 2009, Journal of personality and social psychology.

[41]  K. Trzesniewski,et al.  Measuring Global Self-Esteem: Construct Validation of a Single-Item Measure and the Rosenberg Self-Esteem Scale , 2001 .

[42]  A. Diamantopoulos,et al.  Guidelines for choosing between multi-item and single-item scales for construct measurement: a predictive validity perspective , 2012 .