Tackling the Problem of Construct Proliferation

Construct proliferation—the accumulation of ostensibly different but potentially identical constructs representing organizational phenomena—is a salient problem in contemporary research. While a number of construct validation procedures exist, relatively few validation studies conduct comprehensive assessments of the discriminant validity of theoretically distinct constructs. In this article, we outline the key considerations a researcher must take into account when attempting to establish the empirical distinctness of new or existing constructs and provide a step-by-step guide on how to assess the discriminant validity of constructs while accounting for three major sources of measurement error: random error, specific factor error, and transient error. Using a number of popular measures from the leadership literature, we provide an illustrative example of how to conduct a study of discriminant validity. We include several analytic strategies in our study and discuss the similarities and differences between the results they yield. We also discuss several additional issues related to this type of research and make recommendations for conducting discriminant validity analyses.

[1]  M. Sandy Hershcovis,et al.  “Incivility, social undermining, bullying…oh my!”: A call to reconcile constructs within workplace aggression research , 2011 .

[2]  F. Al-Shamali,et al.  Author Biographies. , 2015, Journal of social work in disability & rehabilitation.

[3]  Nila A Sathe,et al.  Searching for studies: a guide to information retrieval for Campbell systematic reviews , 2017 .

[4]  David J. Henderson,et al.  Servant leadership: Development of a multidimensional measure and multi-level assessment ☆ , 2008 .

[5]  C. Fornell,et al.  Evaluating structural equation models with unobservable variables and measurement error. , 1981 .

[6]  S. Carless,et al.  Assessing the discriminant validity of transformational leader behaviour as measured by the MLQ , 1998 .

[7]  B. Tepper,et al.  Consequences of Abusive Supervision , 2000 .

[8]  R. N. Kanungo,et al.  Charismatic leadership and follower effects , 2000 .

[9]  Chester A. Schriesheim,et al.  A theoretical and empirical examination of the transactional and non-leadership dimensions of the Multifactor Leadership Questionnaire (MLQ) , 2008 .

[10]  Phillip W. Braddy,et al.  Power and sensitivity of alternative fit indices in tests of measurement invariance. , 2008, The Journal of applied psychology.

[11]  Eric P. Charles,et al.  The correction for attenuation due to measurement error: clarifying concepts and creating confidence sets. , 2005, Psychological methods.

[12]  Robert P. Althauser,et al.  Validity and the Multitrait-Multimethod Matrix , 1970 .

[13]  A. Sias.  Interpretation of educational measurements. , 1928 .

[14]  Russell V. Lenth,et al.  Java Applets for Power and Sample Size , 2015 .

[15]  James K. Harter,et al.  Causal Impact of Employee Work Perceptions on the Bottom Line of Organizations , 2010, Perspectives on psychological science : a journal of the Association for Psychological Science.

[16]  Mark G Ehrhart,et al.  Leadership and Procedural Justice Climate as Antecedents of Unit-Level Organizational Citizenship Behavior. , 2004 .

[17]  Barry M. Staw,et al.  The dispositional approach to job attitudes: A lifetime longitudinal test. , 1986 .

[18]  Kevin E. Voss,et al.  A proposed procedure for construct definition in marketing , 2013 .

[19]  F. Schmidt,et al.  Detecting and Correcting the Lies That Data Tell , 2010, Perspectives on psychological science : a journal of the Association for Psychological Science.

[20]  Jay A. Conger,et al.  Charismatic leadership in organizations: Perceived behavioral attributes and their measurement , 1994 .

[21]  F. Schmidt,et al.  Are True Scores and Construct Scores the Same? A Critical Examination of Their Substitutability and the Implications for Research Results , 2013 .

[22]  Jeffrey Pfeffer,et al.  The Ambiguity of Leadership , 1977 .

[23]  G Becker,et al.  How important is transient error in estimating reliability? Going beyond simulation studies. , 2000, Psychological methods.

[24]  Jay A. Conger,et al.  Measuring Charisma: Dimensionality and Validity of the Conger-Kanungo Scale of Charismatic Leadership , 2009 .

[25]  Huy Le,et al.  Beyond alpha: an empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs. , 2003, Psychological methods.

[26]  Steven D. Brown,et al.  Handbook of applied multivariate statistics and mathematical modeling , 2000 .

[27]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[28]  R. Rosenthal,et al.  Statistical Procedures and the Justification of Knowledge in Psychological Science , 1989 .

[29]  C. A. Higgins,et al.  THE BIG FIVE PERSONALITY TRAITS, GENERAL MENTAL ABILITY, AND CAREER SUCCESS ACROSS THE LIFE SPAN , 1999 .

[30]  Huy Le,et al.  The problem of empirical redundancy of constructs in organizational research: An empirical investigation , 2010 .

[31]  E. A. Fleishman,et al.  Taxonomic efforts in the description of leader behavior: A synthesis and functional interpretation , 1991 .

[32]  G. Graen,et al.  Relationship-based approach to leadership: Development of leader-member exchange (LMX) theory of leadership over 25 years: Applying a multi-level multi-domain perspective , 1995 .

[33]  John E. Hunter,et al.  Theory Testing and Measurement Error. , 1999 .

[34]  W. W. Rozeboom.  The fallacy of the null-hypothesis significance test. , 1960, Psychological bulletin.

[35]  R. Hackett,et al.  Leader-Member Exchange as a Mediator of the Relationship Between Transformational Leadership and Followers' Performance and Organizational Citizenship Behavior , 2005 .

[36]  B. Bass.  From transactional to transformational leadership: Learning to share the vision. , 1990 .

[37]  L. Mann,et al.  A Short Measure of Transformational Leadership , 2000 .

[38]  J. M. Cortina,et al.  Determining relative importance of predictors with the observational design. , 1998 .

[39]  Terri A. Scandura,et al.  The MLQ revisited: psychometric properties and recommendations , 2001 .

[40]  T. Judge,et al.  Five-factor model of personality and transformational leadership. , 2000, The Journal of applied psychology.

[41]  Angela P. Wetzel.  Internet, mail, and mixed‐mode surveys: The tailored design method , 2010 .

[42]  Arthur G. Bedeian,et al.  Job Burnout and Employee Engagement , 2012 .

[43]  Herbert W. Marsh,et al.  Confirmatory Factor Analysis of Multitrait Multimethod Matrices. , 1983 .

[44]  C. Schriesheim.  Causal Analysis: Assumptions, Models, and Data , 1982 .

[45]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[46]  R. MacCallum,et al.  Power analysis and determination of sample size for covariance structure modeling. , 1996 .

[47]  D. Whetten.  What Constitutes a Theoretical Contribution , 1989 .

[48]  Michael Donnelly,et al.  Systematically Retrieving Research: A Case Study Evaluating Seven Databases , 2007 .

[49]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[50]  C. Reeve,et al.  Estimation of Transient Error in Cognitive Ability Scales , 2005 .

[51]  A. Tsui.  A role set analysis of managerial reputation , 1984 .

[52]  Timothy R. Hinkin,et al.  A Brief Tutorial on the Development of Measures for Use in Survey Questionnaires , 1998 .

[53]  Thomas D. Taber,et al.  A Hierarchical Taxonomy of Leadership Behavior: Integrating a Half Century of Behavior Research , 2002 .

[54]  J. Swales.  CITATION ANALYSIS AND DISCOURSE ANALYSIS , 1986 .

[55]  E. A. Fleishman,et al.  The description of supervisory behavior. , 1953 .

[56]  Oliver P. John,et al.  Measurement: Reliability, construct validation, and scale construction. , 2000 .

[57]  J. Pfeffer.  Barriers to the Advance of Organizational Science: Paradigm Development as a Dependent Variable , 1993 .

[58]  Robert E. Ployhart,et al.  Longitudinal Research: The Theory, Design, and Analysis of Change , 2010 .

[59]  J. Rowold,et al.  Transformational and charismatic leadership: Assessing the convergent, divergent and criterion validity of the MLQ and the CKS , 2007 .

[60]  Rik Crutzen,et al.  Time is a jailer: What do alpha and its alternatives tell us about reliability? , 2007 .

[61]  Rick H. Hoyle,et al.  Confirmatory Factor Analysis , 1983 .

[62]  A Reaction to Greenwald , Pratkanis , Leippe , and Baumgardner ( 1986 ) : Under What Conditions Does Research Obstruct Theory Progress ? , 2004 .

[63]  S. Bonaccio,et al.  On the myth and the reality of the temporal validity degradation of general mental ability test scores , 2011 .

[64]  B. Bass,et al.  Re‐examining the components of transformational and transactional leadership using the Multifactor Leadership , 1999 .

[65]  Claudia C. Cogliser,et al.  Construct Validation Using Computer-Aided Text Analysis (CATA) , 2010 .

[66]  Martha Kneale,et al.  The development of logic , 1963 .

[67]  Michael E. Brown,et al.  Ethical leadership: A social learning perspective for construct development and testing , 2005 .

[68]  L. Gottfredson.  Why g matters: The complexity of everyday life , 1997 .

[69]  Christine A. Henle,et al.  A case for recognizing distinctions among constructs that capture interpersonal mistreatment in work organizations , 2011 .

[70]  Edgar Erdfelder,et al.  G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences , 2007, Behavior research methods.

[71]  Janice Tripney,et al.  Systematic literature searching in policy relevant, inter-disciplinary reviews: an example from culture and sport. , 2011, Research synthesis methods.

[72]  Howard D. White,et al.  SCIENTIFIC COMMUNICATION AND LITERATURE RETRIEVAL , 2019, The Handbook of Research Synthesis and Meta-Analysis.

[73]  S. Noranee,et al.  Influence of impression management on interpersonal citizenship behavior: Leader-member exchange as a mediator , 2012, 2012 International Conference on Innovation Management and Technology Research.

[74]  B. Avolio,et al.  . , 2005, The Leadership Quarterly.

[75]  N. Ambady,et al.  Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis. , 1992 .

[76]  Denise M. Rousseau,et al.  A STICKY, LEVERAGING, AND SCALABLE STRATEGY FOR HIGH-QUALITY CONNECTIONS BETWEEN ORGANIZATIONAL PRACTICE AND SCIENCE , 2007 .

[77]  J. Felfe,et al.  Examining the Factor Structure of the MLQ: Recommendation for a Reduced Set of Factors. , 2005 .

[78]  Jeroen K. Vermunt,et al.  Composing Group-Level Constructs From Individual-Level Survey Data , 2009 .

[79]  F. Schmidt,et al.  Measurement Error in Psychological Research: Lessons From 26 Research Scenarios , 1996 .

[80]  Greg L Stewart,et al.  Initial Evaluations in the Interview: Relationships with Subsequent Interviewer Evaluations and Employment Offers , 2010, The Journal of applied psychology.

[81]  R. Nickerson,et al.  Null hypothesis significance testing: a review of an old and continuing controversy. , 2000, Psychological methods.

[82]  R. MacCallum,et al.  Testing differences between nested covariance structure models: Power analysis and null hypotheses. , 2006, Psychological methods.

[83]  Dan J. Putka,et al.  The Multifaceted Nature of Measurement Artifacts and Its Implications for Estimating Construct-Level Relationships , 2009 .

[84]  Ralph M. Stogdill,et al.  Manual for the leader behavior description questionnaire - Form XII , 1963 .

[85]  Phil Wood.  Confirmatory Factor Analysis for Applied Research , 2008 .

[86]  James K. Harter,et al.  Conceptual Versus Empirical Distinctions Among Constructs: Implications for Discriminant Validity , 2008, Industrial and Organizational Psychology.

[87]  A. Halpin.  Manual for the leader behavior description questionnaire , 1957 .

[88]  J. Conway,et al.  Estimation and Uses of the Proportion of Method Variance for Multitrait-Multimethod Data , 1998 .

[89]  Richard P. Bagozzi,et al.  Assessing Construct Validity in Organizational Research , 1991 .

[90]  D. Watson,et al.  Personality development in emerging adulthood: integrating evidence from self-ratings and spouse ratings. , 2006, Journal of personality and social psychology.

[91]  Stephen E. Humphrey,et al.  TRAIT AND BEHAVIORAL THEORIES OF LEADERSHIP: AN INTEGRATION AND META-ANALYTIC TEST OF THEIR RELATIVE VALIDITY , 2011 .