Illusions of Significance in a Rugged Landscape

Introduction Suppose you had collected, or had been given, a set of data summarizing the characteristics of a great many informing systems, including information relating to the sender, the client, the delivery system, and the associated task for each system. Further suppose that you also had, for each system, one or more measures of system success, such as user satisfaction, degree of use, and contribution to profitability. Under such circumstances, it would be perfectly natural to want to explore the degree to which different informing system characteristics contribute to success. Furthermore, if you had the proper training--such as a doctorate in MIS research--you would have at your disposal a variety of statistical tools, such as multiple regression, that could be employed to perform such analysis. The use of such tools involves an implicit assumption. Specifically, they assume that the underlying process that generated the data is decomposable. What this means in the case of a basic multiple regression model (i.e., a model with a single term for each independent variable--often called a first-order model) is that each variable in the model impacts the success measure in a manner that is independent of the values of the remaining variables. Where such decomposability is not predicted to be present, it is possible to incorporate specific interaction terms, thereby modeling the effects of combinations of more than one independent variable value. The potential number of such terms, however, is extraordinarily large for all but the most trivial set of characteristics. Thus, decomposability tends to be assumed unless strong evidence to the contrary exists. In a companion paper (Gill, 2008), a conceptual argument is presented that the assumption of decomposability is unlikely to be valid where the characteristics that drive the success of an informing system are being studied. Instead, it proposes that the relationship between system characteristics and system success is likely to be better described as a rugged fitness landscape, a conceptual framework developed for evolutionary biology (Kauffman, 1993). In rugged landscapes, a particular characteristic's impact on fitness cannot be determined without knowing the values of several more attributes. For example, an informing system that is useful, easy to use, and impacts a minor aspect of its user's job may be greeted with enthusiasm, signifying a high value on the "user acceptance" fitness scale. On the other hand, if a nearly identical system impacts a significant part of the user's job, the user's reaction could be resistance stemming from concerns about job loss. Moreover, the level of that resistance might further depend on the strength of various user motivational drives (e.g., Lawrence & Nohria, 2002). Users with a strong drive to bond might well place organizational needs on a par with their personal needs and thus welcome the system; users with a strong defensive drive--which manifests itself as the individual's need to protect possessions and status--might go so far as resorting to sabotage. A compelling argument that can be advanced against the rugged fitness landscape model of the companion paper is the large body of research that has found significant relationships between system characteristics and fitness using statistical techniques that assume decomposability. Indeed, one of the authors has advanced such models himself in the past (e.g., Gill, 1996). Multiple regression, for example, provides an estimate of the significance for each model coefficient that it estimates. As researchers, we typically ignore any relationship that could happen by chance more than one time in twenty (p>=0.05). Frequently, our tests tell us that what we observe could be explained as the result of random variations in the data less than one time in a hundred (p