Organisational Size Measurement in IS Research: Implications and Consequences of Using Different Metrics

Organisational size is an important research variable in information systems research. Prior research has identified a large number of organisational size metrics in use in the published information systems research literature. However, many researchers offer little supportive discussion for their choice of size metric. This paper documents a pilot study to determine the effects of using different organisational size metrics. The study takes a dataset from an existing study and analyses a group of research hypotheses in terms of different organisational size metrics in an operational research context. It is the thesis of this paper that if researchers do not pay attention to the size metric in use, they risk delivering erroneous results. Additionally, due to the high amount of extant literature available, researchers are easily able to unwittingly justify these results. The paper makes several important findings. First, the outcome of hypothesis testing depends on the metric used. Second, this effect does not appear to be systematic. Third, regardless of the outcome of this hypothesis testing, the results can be justified. Introduction There has been recent argument in the literature concerning IS as a reference discipline. Authors such as Baskerville and Myers (2002) have argued that whereas IS has drawn on older disciplines for formation and foundation, IS should now seek to stand on its own as a reference discipline. These authors appear to argue not for isolation, but rather co-operation and consistency in building a model discipline that other scholars would look to for guidance and innovation. The advantages of such an approach are clear. Cooperation with academics from other disciplines should introduce other researchers to the fresh approaches prevalent in IS research. It would also serve to reinforce the research methods prevalent in IS. Such cohesion would demonstrate disciplinary maturity and agreement among researchers. To some extent, this is attractive but remains elusive. Examination of the behaviour of organisations (and, macroscopically, groups of organisations) remains a pivotal component of many research domains. This scholarly Working Paper – Do Cite Without Permission attention is echoed in the IS research literature (Baroudi and Lucas 1994). This organisational research comes in a variety of forms, including social organisations, governmental organisations, managerial groups and commercial organisations, with particular attention given to the analysis of social organisations (Holsapple and Luo 1996). Amid the many experimental variables that researchers employ in their scholarly endeavour, organisational size is of particular importance. Many have observed it as important in the analysis of organisations and technology. Some studies have argued that organisational size has been a poor indicator of behaviour. For instance, Grover and Teng (1992) observed similar technology adoption behaviour between larger and smaller organisations. Sampler and Short (1994) and Ewusi-Mensah (1997) delivered similar findings with regard to project and system development failure respectively. Brynjolfsson et al. (1994) provided inconclusive results with respect to size and technology use: Ettlie et al. (1984) argued that, ultimately, only extremely large organisational size is a useful predictor of technology adoption or organisational behaviour. Gifford (1992:295) wrote, “if firm size matters at all, it matters only in industries with low technological opportunity”. While researchers persist in using size as a component of their research, its application continues to deliver inconclusive results. Clearly, organisational size deserves sober reassessment. This at once provides a dual opportunity. First, it allows us to colour the construct and hence reinforce/improve IS research. Second, it contributes to the reputation of IS as a reference discipline in and of itself. These arguments lead to the following research questions. Do different organisational size metrics affect research outcomes? What are the research implications of using different organisational size metrics? It is the purpose of this chapter to illustrate some of the inconsistency regarding the application and use of the organisational size construct. Evidence of the inconsistency of organisational size will be observed in terms of primary evidence. This paper comes in the tradition of Hitt and Brynjolfsson who examined the concept of IT value using different measures. They obtained similar results for that construct. The rest of this paper is structured as follows. The paper first establishes organisational size as an important construct in the IS discipline. The paper then discusses the research method used in the study. The results are then detailed. Based on these results, the paper offers two alternative but exclusive courses of discussion. The paper then attempts to illustrate how easily researchers can justify these results based on extant argument from the literature by Working Paper – Do Cite Without Permission offering discussion from two separate points of view as if only one metric had been used. Conclusions and areas for further research are offered. An Empirical Study of IT Adoption IT adoption is one phenomenon that has been frequently examined in the IS literature. Researchers are keen to determine the differences in adoption behaviour between “larger” and “smaller” firms. Despite the competing theories, as discussed above, the topic remains popular in the literature. Unsurprisingly, given this intense literature coverage, a number of technologies have been examined with regard to organisational adoption. These include database management systems (Grover and Teng 1992), expert systems (Shao 1999), microcomputers (Robey 1981, Delone 1988, Lind et al. 1989), and Electronic Data Interchange (Iacovou and Benbasat 1995). Business technology adoption can be analysed according to the business’ physical characteristics. Given certain business characteristics, inferences can be made about how the business will adopt technology. Numerous business characteristics receive attention in the Information Systems literature (Yap 1990). Ein-Dor and Segev (1978), identify 22 characteristics of businesses which have particular bearing on the success of information systems. Ginzberg (1980) recognised 12 factors which affected the implementation of an information system, while Lind et al. (1989) focused on just two elements. Other characteristics in addition to those outlined above are identified, however their relationship to technology adoption has been inconsistent. Such discrepant characteristics include the presence of a systems analyst (Yap et al. 1992), management support (Cale and Eriksen 1994), management structure (Sanders and Courtney 1985), remoteness of business location (Raymond 1985), and customer requirements (Yap 1990). The size of the business is held to be the most important characteristic in the analysis of technology adoption (Lind et al. 1989). However, despite the arguments of authors such as Goode (2001), many researchers rely on univariate measures of organisational size and several metrics are available in the IS literature. However, regardless of this apparent popularity, there appears to be very little supportive discussion of organisational size metrics. Whereas authors typically offer sound reasons for why their use of size is relevant to the experiment at hand, they are reluctant to apply the same rigour to their measurement of the construct. Furthermore, these analyses appear to be undertaken with scant regard for their effects on experimental rigour, validity and reliability. Working Paper – Do Cite Without Permission Authors may pursue a number of avenues when selecting a metric for their studies involving organisational size. First, they may adopt a metric already in use in the literature (as in Karimi et al. 1996). Alternatively, instead of basing measurement on theory, authors may base measurement on their understanding of the size construct or metric availability (Kimberly 1976). They may eventually choose a metric for which data are easiest to obtain. Yankelovich (1972) described this as McNamara’s fallacy: “The first step is to measure whatever can be easily measured. This is OK as far as it goes. The second step is to disregard that which can’t be easily measured or give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume what can’t be measured easily really isn’t important. This is blindness. The fourth step is to say that what can’t be measured really doesn’t exist. This is suicide.” The apparent assumption is that the choice of organisational size metric does not affect the outcome of the research testing. Approach It is important to note that the research approach was not aimed at testing or validating theory on technology adoption per se, but rather to test the effects of using different organisational size metrics: in essence, the study was geared towards determining sensitivity towards experimental error through mis-measurement. The study aims to show that if theory development is based primarily on hypothesis testing then researchers risk making errors if their understanding and measurement of size is poorly founded. In the interests of exploratory analysis, this study takes six size metrics that have received some empirical use in the research literature. The goals of the study placed several requirements on the research method, so care was taken to choose a sound path of research program development. The first requirement was for empirical organisational data. The second requirement was that these data had to be amenable to satisfying analysis of organisational size as an explanatory experimental variable. Data were taken from an existing study, aimed at examining the different reasons for IT adoption. The study surveyed 588 publicly listed Australian