Construct Validation Using Computer-Aided Text Analysis (CATA)

Construct validity continues to pose challenges in the organizational sciences. To capture difficult-to-measure constructs of interest, researchers have often relied on content analysis. One content analysis technique, computer-aided text analysis (CATA), is particularly attractive because of the ability to process large samples with high speeds and reliabilities. Unfortunately, inconsistent guidance exists to guide researchers through the use of this tool in a manner compatible with accepted methods used to validate constructs in a rigorous manner. The authors review research using content analysis to examine the extent to which such studies integrate methods for assessing content, external, discriminant, and predictive validity. To provide direction for organizational researchers interested in using CATA to measure theoretically based constructs relevant to the management field, they suggest a number of possible procedures to enhance construct validity. They illustrate these procedures using the construct of entrepreneurial orientation.

[1]  Hamid Mehran,et al.  Executive compensation structure, ownership, and firm performance , 1995 .

[2]  R. Duane Ireland,et al.  Entrepreneurship Research in AMJ: What Has Been Published, and What Might the Future Hold?* , 2005 .

[3]  R. Hyman Quasi-Experimentation: Design and Analysis Issues for Field Settings (Book) , 1982 .

[4]  R. Morgan,et al.  Deconstructing the relationship between entrepreneurial orientation and business performance at the embryonic stage of firm growth , 2007 .

[5]  D. Hambrick THE FIELD OF MANAGEMENT'S DEVOTION TO THEORY: TOO MUCH OF A GOOD THING? , 2007 .

[6]  David J. Wright,et al.  Alternative Small-Cap Stock Benchmarks , 2002 .

[7]  John,et al.  Reclaiming Qualitative Methods for Organizational Research: A Preface. , 1979 .

[8]  Jeffrey C. Kohles,et al.  Charisma under Crisis: Presidential Leadership, Rhetoric, and Media Responses Before and After the September 11th Terrorist Attacks , 2004 .

[9]  J. Edwards Multidimensional Constructs in Organizational Behavior Research: An Integrative Analytical Framework , 2001 .

[10]  David J. Miller,et al.  The Correlates of Entrepreneurship in Three Types of Firms , 1983 .

[11]  L. Ellis Research Methods in the Social Sciences , 1994 .

[12]  Johan Wiklund,et al.  Entrepreneurial Orientation, Risk Taking, and Performance in Family Firms , 2007 .

[13]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[14]  P. Barr,et al.  Cognitive change, strategic action, and organizational renewal , 1992 .

[15]  S. Zahra,et al.  Contextual influences on the corporate entrepreneurship-performance relationship: A longitudinal analysis , 1995 .

[16]  Joseph F. Porac,et al.  Worth, words, and the justification of executive pay , 1997 .

[17]  D. Slevin,et al.  THE INFLUENCE OF ORGANIZATION STRUCTURE ON THE UTILITY OF AN ENTREPRENEURIAL TOP MANAGEMENT STYLE , 1988 .

[18]  A. Meyer What Is Strategy's Distinctive Competence? , 1991 .

[19]  S. Rogelberg,et al.  Introduction Understanding and Dealing With Organizational Survey Nonresponse , 2007 .

[20]  J. Courtis,et al.  PRIVATE SHAREHOLDER RESPONSE TO CORPORATE ANNUAL REPORTS , 1982 .

[21]  D. McAdams,et al.  Construct validity and content analysis. , 1993, Journal of personality assessment.

[22]  Jeremy C. Short,et al.  The Application of DICTION to Content Analysis Research in Strategic Management , 2008 .

[23]  Jeremy C. Short,et al.  Mission Statements in U.S. Colleges of Business: An Empirical Examination of Their Content With Linkages to Configurations and Performance , 2008 .

[24]  Melissa Woodard Barringer,et al.  A Review and Analysis of the Policy-Capturing Methodology in Organizational Research: Guidelines for Research and Practice , 2002 .

[25]  M. Hitt,et al.  The new competitive landscape , 1995 .

[26]  Yehuda Baruch,et al.  Response Rate in Academic Studies — A Comparative Analysis , 1999 .

[27]  Wladimir M. Sachs,et al.  Grounded Theory Method in Management Research , 2008 .

[28]  Timothy G. Pollock,et al.  Industry Categories and the Politics of the Comparable Firm in CEO Compensation , 1999 .

[29]  T. Scandura,et al.  Research Methodology In Management: Current Practices, Trends, And Implications For Future Research , 2000 .

[30]  Gregory G. Dess,et al.  Clarifying the Entrepreneurial Orientation Construct and Linking It To Performance , 1996 .

[31]  Xianggui Qu,et al.  Multivariate Data Analysis , 2007, Technometrics.

[32]  Dan R. Dalton,et al.  Investment bankers and IPO pricing: does prospectus information matter? , 2005 .

[33]  Cynthia G. Emrich,et al.  Images in Words: Presidential Rhetoric, Charisma, and Greatness , 2001 .

[34]  J. Pennebaker,et al.  Psychological aspects of natural language. use: our words, our selves. , 2003, Annual review of psychology.

[35]  Douglas W. Lyon,et al.  Enhancing Entrepreneurial Orientation Research: Operationalizing and Measuring a Key Strategic Decision Making Process , 2000 .

[36]  R. Weber Basic content analysis, 2nd ed. , 1990 .

[37]  M. Hitt,et al.  Construct measurement in strategic management research: illusion or reality? , 2005 .

[38]  Russell Craig,et al.  The transformational leader as pedagogue, physician, architect, commander, and saint: Five root metaphors in Jack Welch's letters to stockholders of General Electric , 2007 .

[39]  Jeremy C. Short,et al.  Organizational performance referents: An empirical examination of their content and influences , 2003 .

[40]  David L. Deephouse,et al.  Media Reputation as a Strategic Resource: An Integration of Mass Communication and Resource-Based Theories , 1999 .

[41]  Shaker A. Zahra,et al.  Ownership and the Internationalization of Small Firms , 2005 .

[42]  J. Pearce,et al.  Corporate Mission Statements: The Bottom Line , 1987 .

[43]  Theresa M. Welbourne,et al.  The People/Performance Balance in IPO Firms: The Effect of the Chief Executive Officer's Financial Orientation , 2000 .

[44]  L. Cronbach,et al.  Construct validity in psychological tests. , 1955, Psychological bulletin.

[45]  S. Parker Content Analysis for the Social Sciences and Humanities , 1970 .

[46]  Alison M. Konrad,et al.  The impact of work‐life programs on firm productivity , 2000 .

[47]  M. Sarkar,et al.  Alliance entrepreneurship and firm market performance , 2001 .

[48]  Jeffrey S. Conant,et al.  Entrepreneurial Orientation, Distinctive Marketing Competencies And Organizational Performance , 2011 .

[49]  Louis D. Marino,et al.  Exploring an Inverted U–Shape Relationship between Entrepreneurial Orientation and Performance in Chinese Ventures , 2007 .

[50]  David L. Deephouse,et al.  Does Isomorphism Legitimate? , 1996 .

[51]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[52]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[53]  Johan Wiklund,et al.  Entrepreneurial orientation and business performance: Cumulative empirical evidence , 2004 .

[54]  Karen Golden-Biddle,et al.  Using Content Analysis to Elaborate Adolescent Theories of Organization , 2008 .

[55]  R. Morgan,et al.  Business performance and dimensions of strategic orientation , 2003 .

[56]  Scott Shane,et al.  Encouraging university entrepreneurship? The effect of the Bayh-Dole Act on university patenting in the United States , 2004 .

[57]  Kathleen M. Carley Extracting team mental models through textual analysis , 1997 .

[58]  Eric W. K. Tsang,et al.  Replication and Theory Development in Organizational Science: A Critical Realist Perspective , 1999 .

[59]  Yingmei Cheng,et al.  Buy-Side Analysts, Sell-Side Analysts, and Investment Decisions of Money Managers , 2005, Journal of Financial and Quantitative Analysis.

[60]  Justin W. Webb,et al.  Entrepreneurship Research in AMJ : What Has Been Published , and What Might the Future Hold ? , 2022 .

[61]  Lawrence R. Jauch,et al.  Structured Content Analysis of Cases: A Complementary Method for Organizational Research , 1980 .

[62]  C. C. Snow,et al.  The Role of Entrepreneurship in Building Cultural Competitiveness in Different Organizational Types , 2003 .

[63]  Robert K. Kazanjian Relation of Dominant Problems to Stages of Growth in Technology-Based New Ventures , 1988 .

[64]  M. Frese,et al.  Entrepreneurial Orientation and Business Performance: An Assessment of past Research and Suggestions for the Future , 2009 .

[65]  K. Atuahene–Gima,et al.  An Empirical Investigation of the Effect of Market Orientation and Entrepreneurship Orientation Alignment on Product Innovation , 2001 .

[66]  P. Schnurr,et al.  Content analysis: a comparison of manual and computerized systems. , 1990, Journal of personality assessment.

[67]  C. Lance,et al.  The Sources of Four Commonly Reported Cutoff Criteria , 2006 .

[68]  R. Duane Ireland,et al.  Theory and Methodology in Entrepreneurship Research , 2005 .

[69]  D. Slevin,et al.  Strategic management of small firms in hostile and benign environments , 1989 .

[70]  Kimberly A. Neuendorf,et al.  The Content Analysis Guidebook , 2001 .

[71]  W. James Potter,et al.  Rethinking validity and reliability in content analysis , 1999 .

[72]  M. Porter,et al.  HOW MUCH DOES INDUSTRY MATTER , REALLY ? , 1997 .

[73]  P P Schnurr,et al.  A methodological note on content analysis: estimates of reliability. , 1986, Journal of personality assessment.

[74]  J. I. Rodale The Synonym Finder , 1958 .

[75]  Anne Smith,et al.  Improving Survey Response Rates from Chief Executive Officers in Small Firms: The Importance of Social Networks , 2006 .

[76]  Boris Kabanoff,et al.  Computer-aided content analysis: What do 240 advertising slogans have in common? , 1996 .

[77]  Rhonda K. Reger,et al.  A Content Analysis of the Content Analysis Literature in Organization Studies: Research Themes, Data Sources, and Methodological Refinements , 2007 .

[78]  Paul C. Godfrey,et al.  THE PROBLEM OF UNOBSERVABLES IN STRATEGIC MANAGEMENT RESEARCH , 1995 .

[79]  Shaker A. Zahra,et al.  Goverance, Ownership, and Corporate Entrepreneurship: The Moderating Impact of Industry Technological Opportunities , 1996 .

[80]  Douglas W. Lyon,et al.  Issues of Research Design and Construct Measurement in Entrepreneurship Research: The past Decade , 2001 .

[81]  David J. Ketchen,et al.  The Role of Sampling in Strategic Management Research on Performance: A Two-Study Analysis , 2002 .

[82]  Patrick M. Kreiser,et al.  Assessing the Psychometric Properties of the Entrepreneurial Orientation Scale: A Multi-Country Analysis , 2002 .

[83]  J. Hair Multivariate data analysis , 1972 .

[84]  Jeffrey C. Kohles,et al.  Charting the language of leadership: a methodological investigation of President Bush and the crisis of 9/11. , 2004, The Journal of applied psychology.

[85]  Richard P. Bagozzi,et al.  Assessing Construct Validity in Organizational Research , 1991 .

[86]  Achilles A. Armenakis,et al.  Measuring Emotion in Open-ended Survey Responses: An Application of Textual Data Analysis , 1995 .

[87]  Robert Waldersee,et al.  Espoused Values and Organizational Change Themes , 1995 .

[88]  Eric Abrahamson,et al.  Concealment of Negative Organizational Outcomes: An Agency Theory Perspective , 1994 .

[89]  Dennis A. Gioia,et al.  Sensemaking and sensegiving in strategic change initiation , 1991 .

[90]  John C. Narver,et al.  The Positive Effect of a Market Orientation on Business Profitability: A Balanced Replication , 2000 .

[91]  Kathleen M. Carley Extracting team mental models through textual analysis , 1997 .

[92]  S. Rogelberg,et al.  Understanding and Dealing With Organizational Survey Nonresponse , 2010 .

[93]  Gregory G. Dess,et al.  Linking Two Dimensions of Entrepreneurial Orientation to Firm Performance: The Moderating Role of Environment and Industry Life Cycle , 2001 .

[94]  S. Maitlis,et al.  Triggers And Enablers Of Sensegiving In Organizations , 2007 .

[95]  Carole Elliott,et al.  Knowledge for theory and practice: balancing commitments in collaborative research , 2008 .

[96]  P. Barr,et al.  Cognitive change, strategic action and organizational renewal , 1993 .

[97]  Siah Hwee Ang,et al.  Increasing Replication for Knowledge Accumulation in Strategy Research , 2003 .

[98]  A. Oliveira,et al.  A new competitive landscape , 2007 .

[99]  Thomas Ritter,et al.  The impact of network capabilities and entrepreneurial orientation on university spin-off performance , 2006 .

[100]  Karen A. Jehn,et al.  Analyzing harsh words in a sensitive setting: American expatriates in communist China , 1997 .

[101]  Norman N. Markel A semiotic contribution to libel action law , 1998 .

[102]  M. Porter,et al.  HOW MUCH DOES INDUSTRY MATTER, REALLY? , 1997 .

[103]  Warren D. Miller Firm, Strategic Group, and Industry Influences on Performance , 2007 .

[104]  David A. Whetten,et al.  Family Firms and Social Responsibility: Preliminary Evidence from the S&P 500 , 2006 .

[105]  Walter J. Ferrier Navigating the Competitive Landscape: The Drivers and Consequences of Competitive Aggressiveness , 2001 .

[106]  Michael D. Michalisin Validity of annual report assertions about innovativeness: an empirical investigation , 2001 .

[107]  Marc J. Dollinger,et al.  Content Analysis in Entrepreneurship Research: The Case of Initial Public Offerings , 1989 .

[108]  D. Hambrick,et al.  Speed, Stealth, and Selective Attack: How Small Firms Differ From Large Firms in Competitive Behavior , 1995 .

[109]  R. Rumelt How much does industry matter , 1991 .

[110]  Johan Wiklund,et al.  Entrepreneurial orientation and business performance , 2009 .

[111]  S. Zahra Predictors and Financial Outcomes of Corporate Entrepreneurship: An Exploratory Study , 1991 .

[112]  Richard A. D'Aveni,et al.  Crisis and the Content of Managerial Communications: A Study of the Focus of Attention of Top Managers in Surviving and Failing Firms. , 1990 .

[113]  Rebecca J. Morris,et al.  Computerized Content Analysis in Management Research: A Demonstration of Advantages & Limitations , 1994 .

[114]  Marc Bonnet,et al.  The Emergence of Implicit Criteria Actually Used by Reviewers of Qualitative Research Articles , 2008 .

[115]  T. Elfring,et al.  Entrepreneurial Orientation and New Venture Performance: The Moderating Role of Intra- And Extraindustry Social Capital , 2008 .