Understanding the Role of Theory on Instrument Development: An Examination of Strengths and Weaknesses of Discriminant Validity Analysis Techniques

Abstract:Numerous researchers have called attention to many important issues in instrument development throughout the relatively short history of the information systems (IS) academic research discipline (e.g., Petter, Straub, & Rai 2007; Straub, Boudreau, & Gefen 2004; Straub 1989). With the accumulation of knowledge related to the process of instrument development, it has now become necessary to take a closer look at specific aspects of this process. This paper focuses on construct validity, specifically discriminant validity, and examines some popular methods of supporting this type of validity when using cross-sectional data. We examine strengths and weaknesses of these analysis techniques with a focus on the role of theory and informed interpretation. We highlight the applicability of these techniques by analyzing a sample dataset where we theorize two constructs to be highly correlated. With this paper, we provide both researchers and reviewers a greater understanding of the highlighted discriminant validity analysis techniques.Keywords: Construct Validity, Quantitative Analysis, Discriminant Validity, Average Variance Extracted, χ2 Analysis.1 IntroductionThroughout the years, the information systems (IS) discipline has raised issues regarding survey instrument development (e.g., Gefen, Straub, & Boudreau, 2000; Lewis, Templeton, & Byrd, 2005; MacKenzie, Podsakoff, & Podsakoff, 2011; Petter, Straub, & Rai, 2007; Straub, Boudreau, & Gefen, 2004; Straub, 1989). IS research commonly uses cross-sectional data; as such, numerous techniques have been developed to validate instruments. Previous culminating works in this line of research include Straub et al.'s (2004) and Lewis et al.'s (2005) papers that propose validation guidelines for measurement instruments. A paper in a recent special issue of MIS Quarterly has also stressed the need to integrate and disseminate advancements in these areas to accumulate knowledge (MacKenzie et al., 2011). Researchers have also focused on specific issues regarding instrument development and validation (e.g., formative vs. reflective construct measurement, and common method bias) (Bagozzi, 2011; Bollen 2011; Diamantopoulos 2011; Petter et al., 2007; Straub & Burton-Jones, 2007). Following this stream of research, we focus on the area of discriminant validity analysis-specifically on the role of theory and the strengths and weaknesses of some common analysis techniques.We examine some analysis techniques that are confirmatory (i.e., correlation-based, average variance extracted, χ2 difference tests) as opposed to content-driven or item-selection methods along with their inherent strengths and weaknesses. With this understanding, we provide researchers the tools to use these techniques faithfully and interpret their results appropriately. We do not focus on invalidating any of the analysis techniques but compare and contrast different types of discriminant validity analysis. In doing so, we highlight the importance of theoretically informed interpretations of results and minimize the strict adherence to blind statistical procedure and subjective rules-of-thumb.To accomplish these goals, we first discuss the role of discriminant validity analysis under the larger context of instrument development in Section 2. In Section 2.1, we discuss some of the difficulties in establishing discriminant validity. In Sections 3 and 4, we use a sample dataset to illustrate a process of instrument development. The research study used in this example is theoretically justified and includes cross-sectional data containing two highly correlated constructs. Although the situation is not typical, the dataset does provide key illustrations of how IS researchers can effectively use construct validation procedures and guidelines. During this process, the question of discriminant validity arises and we feature the strengths and weaknesses of some discriminant validity analysis techniques. …